Data Engineering Specialist (Databricks)

About the role

Lead a delivery workstream within a client data engineering team, migrating data from legacy warehouses to an Azure Databricks lakehouse. Own technical design, production‑grade pipeline delivery, engineering standards, and coordination of a distributed engineering team while translating business requirements into technical solutions.

Key responsibilities

• Design and build scalable batch and streaming pipelines using PySpark, Databricks SQL, Lakeflow Jobs and Declarative Pipelines, including data cleansing, validation, monitoring and logging.
• Implement Delta Lake medallion architecture (bronze, silver, gold) and dimensional models with fact and dimension tables.
• Optimize Databricks workloads for performance and cost through compute sizing, partitioning, liquid clustering and shuffle optimization.
• Plan and execute migrations from legacy warehouses and ETL tools, handling data reconciliation and cut‑over validation.
• Guide engineers across India, Canada and Costa Rica, manage backlogs, estimate effort, track sprint delivery and mitigate risks.

Required profile

• Bachelor's degree in Computer Science, Engineering or related field, or equivalent professional experience.
• 5+ years of data engineering experience, including 3+ years of hands‑on production work with Databricks and Apache Spark.
• Proven experience leading technical workstreams, reviewing code and coordinating delivery with minimal supervision.

Required skills

• PySpark (DataFrame API, performance tuning)
• Databricks SQL, Delta Lake, Unity Catalog, Lakeflow Jobs & Pipelines
• Azure Data Factory, Azure Data Lake Storage Gen2
• Python for data processing and automation
• Git, CI/CD practices for data pipelines
• Data warehousing concepts, dimensional modeling, data governance

What we offer

• Health and Life Insurance
• Accenture days, 3 additional vacation days
• On‑site doctor
• Birthday leave
• Internet reimbursement