Data Engineer / Data Analyst – Real-time Databricks Lakehouse

About the role

Lumenalta is looking for an experienced Data Engineer / Data Analyst to help a high‑performance motorsport client build a real‑time Databricks Lakehouse platform. You will be responsible for profiling incoming high‑frequency telemetry, optical tracking, and timing/scoring feeds, and establishing data‑quality rules that ensure trustworthy data for engineers, analysts, and race officials.

Key responsibilities

• Profile raw sensor data in the Bronze layer, measuring distributions, null rates, cardinality, schema drift, duplicates, and out‑of‑order records.
• Collaborate with race engineers, analysts, and competition officials to translate domain knowledge into concrete data‑quality expectations.
• Define, document, and codify validity, completeness, uniqueness, timeliness, and range checks for Bronze‑to‑Silver promotion.
• Build and maintain Bronze‑to‑Silver transformation pipelines, including schema enforcement, type casting, deduplication, standardization, and quarantine of failing records.
• Implement data‑quality enforcement using Databricks‑native tools (Delta Live Tables, Lakeflow Declarative Pipelines) or frameworks such as Great Expectations or Databricks DQX.
• Establish data contracts and maintain clear documentation of expected schemas and semantics.
• Monitor data‑quality metrics over time and generate alerts for rule violations.

Required profile

• Proven experience as a Data Engineer or Data Analyst working with high‑frequency, multi‑sensor data.
• Strong ability to work closely with domain experts to define trustworthy data standards.
• Experience designing and operating data‑quality pipelines in a lakehouse environment.

Required skills

• Databricks platform
• Delta Live Tables
• Lakeflow Declarative Pipelines
• Great Expectations or Databricks DQX
• Data profiling and quality rule implementation