Senior Data Engineer (UK)

📋 Description

• • Own the end-to-end design, build, and evolution of cloud-native data pipelines that ingest, transform, and serve petabyte-scale datasets for a global client base. You will architect resilient batch and streaming workflows using dbt, Kafka, Spark, AWS Kinesis, Lambda, and Airflow, ensuring sub-second latency for soft real-time dashboards and downstream ML models.
• • Translate complex business requirements into elegant data models that balance analytical flexibility with strict governance. You will apply Kimball, Data Vault, or wide-table patterns as appropriate, document lineage in tools like dbt docs, and enforce contracts through schema registries and automated testing.
• • Instrument every stage of the data lifecycle with production-grade observability. From CloudWatch custom metrics to Datadog synthetic checks, you will define SLIs/SLOs, set up anomaly detection, and create actionable alerts that reduce MTTR to minutes—not hours—when pipelines drift or upstream sources change.
• • Champion security and compliance by embedding OAuth2, IAM fine-grained policies, and row-level ACLs into APIs and data shares. You will partner with InfoSec to run threat modelling sessions, implement encryption at rest/in transit, and maintain SOC 2 and GDPR evidence packages for auditors.
• • Scale data access patterns for a diverse set of internal and external stakeholders. Whether exposing GraphQL endpoints, signed S3 pre-signed URLs, or Snowflake secure shares, you will design multi-tenant, rate-limited gateways that abstract complexity while guaranteeing backward compatibility.
• • Coach and mentor a distributed team of mid-level engineers through pair programming, architecture reviews, and brown-bag sessions. You will craft growth plans, run inclusive retros, and raise the bar on code quality by introducing linting, type checking, and automated CI/CD gates in GitHub Actions.
• • Collaborate with product managers, data scientists, and UX designers to prioritise features that deliver measurable business impact. You will translate OKRs into technical epics, run estimation workshops, and present sprint demos to C-suite stakeholders, demonstrating how faster data insights translate to revenue.
• • Continuously evaluate emerging technologies—such as lakehouse formats, serverless Spark, or real-time OLAP engines—and run proof-of-concepts that keep the client two steps ahead of the market. You will publish internal tech notes and contribute to YLD’s open-source repos, amplifying our thought leadership.