Lead - Data Platform Engineering

📋 Description • Own the end-to-end architecture of Freshworks’ enterprise-grade Data Lake on AWS and Databricks, ensuring it scales to petabyte-level workloads while maintaining 99.9 % uptime, governance, and cost-efficiency. • Architect, develop, and continuously tune high-throughput CDC pipelines that stream every change from hundreds of MySQL shards into Kafka in near real-time (< 5 s SLA), guaranteeing zero data loss and exactly-once semantics. • Design Spark Structured Streaming jobs that consume from Kafka and land curated datasets into Delta Lake on Databricks; optimize for micro-batch latency, checkpoint reliability, and schema evolution without downtime. • Implement and maintain MapReduce workflows for heavy-duty batch transformations on multi-terabyte datasets, squeezing every ounce of performance from distributed clusters while preserving fault tolerance and replayability. • Build a metadata-driven processing framework that auto-generates ETL/ELT jobs from declarative YAML specs, enforcing data contracts, lineage, and quality rules across every pipeline in the lakehouse. • Instrument the entire stack with Prometheus, Grafana, and custom exporters to deliver real-time observability—latency, throughput, error rates, and cost—so anomalies are caught before business stakeholders notice. • Partner with Product, Legal, Privacy, and Security teams to embed GDPR, SOC-2, and ISO controls into pipelines, including encryption at rest/in transit, PII tokenization, and fine-grained IAM policies. • Champion data quality by integrating Great Expectations or Deequ into CI/CD, blocking merges when freshness, completeness, or accuracy SLAs are violated. • Automate infrastructure provisioning and deployment using Terraform, CloudFormation, and Jenkins/GitHub Actions; enforce blue-green releases, automated rollbacks, and immutable artifacts for every pipeline. • Mentor a team of senior and junior data engineers through design reviews, pair programming, and brown-bag sessions, cultivating a culture of engineering excellence and continuous learning. • Drive quarterly architecture reviews to evaluate emerging technologies—Iceberg, Flink, serverless Spark—and pilot those that reduce cost or improve performance by > 20 %. • Collaborate with Analytics, Data Science, and ML Engineering teams to expose curated datasets via high-performance SQL endpoints, feature stores, and real-time APIs that power predictive models and executive dashboards. • Establish and track SLOs for data freshness, query latency, and pipeline reliability; present weekly metrics to leadership and own the action plan for any regression. • Contribute to the open-source community by upstreaming patches, writing tech blogs, and speaking at meetups to amplify Freshworks’ engineering brand. 🎯 Requirements • 8+ years designing and operating large-scale data lakes/lakehouses on AWS (S3, EMR, Glue, Databricks) with proven success handling multi-petabyte datasets. • Expert-level proficiency in Spark (batch & streaming), Kafka, and Delta Lake; hands-on experience building exactly-once CDC pipelines from MySQL to Kafka to Databricks. • Deep understanding of distributed systems concepts—partitioning, replication, consensus—and demonstrated ability to tune MapReduce jobs for performance and fault tolerance. • Strong software engineering discipline: test-driven development, CI/CD automation (Jenkins/GitHub Actions), infrastructure as code (Terraform/CloudFormation), and code reviews. • Nice-to-have: contributions to open-source data projects, experience with Flink or Iceberg, and familiarity with data governance tools like Apache Atlas or LakeFS. 🏖️ Benefits • Competitive salary plus annual performance bonus and RSUs that let you share in Freshworks’ continued growth. • Flexible hybrid work model with state-of-the-art Chennai office, ergonomic workstations, and quarterly hackathons. • Comprehensive health insurance for you and your dependents, plus wellness stipend for gym, therapy, or mindfulness apps. • Annual learning & development budget of ₹3,00,000 for conferences, certifications, and online courses—grow as fast as the data you manage.