Senior Data Engineer - Streaming Platform

About the role

Deel is the all-in-one payroll and HR platform for global teams, and its Data Platform team is rebuilding how operational data becomes data products. It is moving from a batch, trigger-based system to a streaming architecture built on change data capture (CDC), Spark Structured Streaming and Apache Iceberg, aiming to deliver business-critical data models in under 5 minutes. The Senior Data Engineer owns the platform end to end.

Key responsibilities

• Design, build and harden Spark Structured Streaming applications that turn CDC events into near-real-time data models, including stateful processing, slowly changing dimensions, late and partial records and safe restarts.
• Run and scale AWS EMR clusters, owning instance sizing, managed scaling, multi-cluster deployment, memory tuning and consolidation of low-traffic workloads.
• Maintain Apache Iceberg tables: partition specs, compaction, snapshot expiry, delete-file cleanup and fast-lane maintenance for high-churn tables.
• Onboard new business domains: configure CDC sources and destinations in Estuary Flow, set up Kafka-compatible topics, verify backfills in Snowflake and coordinate with database owners.
• Debug production issues such as connection drops, offset regressions, token expiry, schema mismatches and write conflicts.
• Build observability and tooling, such as sending Spark logs to Grafana Loki, watchdog and recovery steps, and AI-assisted runbooks.
• Improve deployment and developer experience through CI/CD pipelines, configuration structure and guardrails.
• Handle ad hoc ingestion and backfill requests from partner teams.

Required profile

• 7+ years of data engineering experience, with at least 3 years running streaming pipelines in production.
• Methodical debugging style: reading logs, forming hypotheses and proving fixes with data.

Required skills

• Apache Spark, Structured Streaming, state stores (RocksDB), checkpointing and performance tuning.
• A lakehouse format, ideally Apache Iceberg (Delta Lake or Hudi also relevant).
• Spark on AWS EMR, EKS or similar, with cluster capacity planning and autoscaling.
• CDC and Kafka semantics, including offsets and delivery guarantees.
• Python and SQL; Scala or Java is a plus.
• Airflow or a similar orchestrator, CI/CD and infrastructure-as-code.
• Nice to have: Snowflake, Grafana, Loki or Alloy, K8s, and experience helping other teams self-serve on a data platform.

What we offer

• Salary range quoted in USD, with offers localized to the candidate's country.
• Stock grant opportunities depending on role, employment status and location.
• Additional perks and benefits depending on employment status and country.
• Optional flexible working office membership.