Senior Big Data Engineer – Scalable Data Pipelines

About the role

We are looking for a senior Big Data Engineer to design and deliver scalable, high‑quality data solutions for our Housing (RentBureau and Mortgage) products. You will build large‑scale pipelines, implement modern architectures such as Medallion, and leverage AI tools to accelerate development.

Key responsibilities

• Design and implement robust, scalable solutions for processing massive data volumes using Spark, AWS Glue, Airflow, SageMaker and related technologies.
• Develop, maintain and optimise batch processing systems and aggregate tables across multiple product lines and geographies.
• Build CI/CD pipelines for data applications and ensure data quality, observability and operational runbooks.
• Research emerging data tools, adopt generative AI (GitHub Copilot, Claude Code) for code generation and documentation, and support cross‑functional data consumption.

Required profile

• 5+ years of experience in data engineering, requirement gathering, estimation and delivery of large‑scale projects.
• Strong communication skills to translate ambiguous business requirements into clear technical designs.
• Hands‑on experience with production data pipelines and cloud services, especially AWS.

Required skills

• Python, SQL, PySpark
• Apache Spark, AWS Glue, Athena, Redshift, S3, EMR, Glue Catalog
• Airflow, AWS Step Functions, CI/CD pipelines
• Open table formats (Iceberg), Databricks, Snowflake
• Relational databases (PostgreSQL, MySQL)
• Generative AI tools (GitHub Copilot, Claude Code)