BigData DevOps Engineer

About the role

Experian is looking for a BigData DevOps Engineer to support and administer its Hadoop and AWS EMR environments. You will work closely with the IT Infrastructure and Development teams to ensure the reliability, performance, and security of the data platform while supporting product development road‑maps.

Key responsibilities

• Implement, support and administer Hadoop clusters and AWS EMR workloads.
• Maintain Terraform and other IaC scripts for provisioning AWS resources.
• Provide CI/CD pipelines and apply SDLC best practices.
• Automate deployment and operations of Big Data technologies using DevOps tools.
• Onboard Hadoop users and manage Kerberos, HDFS, Hive, HBase and YARN access.
• Apply security best practices across HBase, HDFS, Kafka, Hive and related components.
• Tune Hadoop, MapReduce and Spark jobs and optimise EMR for cost and performance.
• Monitor cluster health, logs and storage; perform capacity planning.
• Collaborate with infrastructure, network, DB, application and BI teams to ensure platform reliability.
• Support upgrades, patching and troubleshooting of EMR, HBase, Spark and other data services.
• Deploy and manage Hadoop clusters, configure HA, schedule jobs and perform backups.
• Configure and maintain tools such as Sentry, Spark, Kafka, Oozie, Solr, MongoDB, DocumentDB and ELK.
• Integrate AD/LDAP with Cloudera and manage Sentry policies.

Required profile

• Bachelor’s degree in Computer Science or a related field, or equivalent experience.
• 3+ years of Hadoop infrastructure administration (AWS EMR or Cloudera).
• 2+ years of experience with cloud platforms, preferably AWS.
• Proven ability to work in a 24x7 support environment.

Required skills

• Hadoop, HDFS, Hive, HBase, YARN, Kerberos.
• AWS EMR, EC2, S3, IAM.
• Terraform, Infrastructure as Code.
• CI/CD tools (e.g., Jenkins, GitLab CI).
• Spark, MapReduce, Kafka, Oozie, Solr.
• Monitoring and logging tools (ELK, Sentry).
• AD/LDAP integration, Cloudera management.