Senior Data Engineer

📋 Description

• • Architect and scale EDO’s petabyte-scale TV outcomes data platform that ingests, transforms, and serves billions of real-time ad impressions and consumer-behavior events across linear and streaming TV. You will own the end-to-end data lifecycle—from raw set-top-box logs, smart-TV ACR data, and programmatic bid streams to investment-grade, syndicated datasets that the world’s largest advertisers rely on for multi-million-dollar decisions.
• • Design cloud-native, event-driven pipelines that process 10 TB+ per day with sub-minute latency, leveraging Apache Kafka, Spark on Kubernetes, and serverless functions in AWS (S3, Glue, Lambda, EMR, Kinesis). Your code will directly influence how quickly clients can optimize campaigns in flight and how accurately they can predict sales lift.
• • Build and harden the semantic layer that maps disparate TV airings to deterministic and probabilistic identity graphs, enabling cross-screen attribution and reach/frequency deduplication. You will partner with data-science teams to productionize vertical AI models that score creative effectiveness and forecast incremental conversions.
• • Establish data-contracts, schema-evolution policies, and automated quality gates so that every dataset meets investment-grade SLAs (≥99.9 % completeness, ≤0.5 % error rate). You will implement real-time anomaly detection and self-healing workflows that reduce on-call incidents by 40 %.
• • Lead the migration from legacy batch ETL to a declarative, SQL-first Lakehouse architecture (Iceberg, Trino, dbt) that empowers analysts and data scientists to ship reliable transformations 3× faster. You will mentor junior engineers through rigorous code reviews and pair programming, raising the bar for Python, Scala, and SQL craftsmanship.
• • Optimize compute and storage economics to cut annual AWS spend by 25 % while doubling throughput. You will institute FinOps guardrails, auto-scaling policies, and spot-instance orchestration that balance cost, latency, and reliability.
• • Champion privacy-by-design: implement differential privacy, GDPR/CCPA deletion workflows, and secure enclaves for PII so that EDO remains the industry’s most trusted measurement provider. You will collaborate with legal and InfoSec to achieve annual SOC 2 Type II and ISO 27001 renewals with zero findings.
• • Instrument end-to-end observability (OpenTelemetry, Prometheus, Grafana) that surfaces pipeline health, data freshness, and cost KPIs in a single pane of glass. Your dashboards will become the daily cockpit for 100+ internal users and external client success teams.
• • Drive cross-functional OKRs with Product, Science, and Revenue teams to launch new streaming currency datasets and expand into international markets. Your roadmap will unlock $10 M+ in incremental ARR by enabling measurement for FAST channels and shoppable TV formats.
• • Publish technical blogs and speak at conferences (Data Council, AWS re:Invent) to position EDO as a thought leader in convergent TV analytics. You will represent the engineering brand to recruits, open-source communities, and Fortune 500 clients alike.