Specialist - Data Platform Engineering

📋 Description

• • Own the end-to-end architecture of Freshworks’ enterprise Data Lake on AWS and Databricks, ensuring petabyte-scale reliability, security, and governance for every business-critical dataset.
• • Architect and continuously evolve the real-time CDC pipeline that streams transactional data from MySQL into Kafka, then into Databricks via Spark Structured Streaming—handling millions of events per hour with sub-second latency.
• • Design and optimize distributed MapReduce workflows that transform, enrich, and aggregate multi-terabyte datasets across Spark clusters, balancing cost, throughput, and fault-tolerance for both batch and streaming workloads.
• • Build metadata-driven frameworks that provide complete data lineage, schema evolution, and automated quality checks—empowering analysts and data scientists to trust every table, column, and metric they consume.
• • Implement comprehensive observability using Prometheus, Grafana, and custom SLIs/SLOs to detect anomalies, predict capacity needs, and trigger auto-remediation before users feel any pain.
• • Champion Infrastructure-as-Code (Terraform, CloudFormation) and CI/CD best practices (Jenkins, GitHub Actions) to deliver repeatable, auditable, and rollback-safe deployments across dev, staging, and production environments.
• • Collaborate with Product, Security, and Compliance teams to embed privacy-by-design controls (PII masking, encryption at rest/in transit, fine-grained RBAC) into every pipeline, ensuring we exceed SOC 2, GDPR, and regional regulatory standards.
• • Partner with data scientists and BI engineers to translate raw events into curated, analytics-ready datasets that power executive dashboards, customer-facing features, and predictive models.
• • Continuously benchmark and tune Spark jobs, Kafka partitions, and underlying EC2/EMR clusters to cut cost per GB processed while doubling query performance quarter-over-quarter.
• • Lead incident post-mortems and root-cause analyses, turning outages into actionable runbooks and architectural improvements that raise the bar for reliability across the entire data platform.
• • Mentor junior engineers through design reviews, pair programming, and lunch-and-learn sessions, cultivating a culture of excellence, curiosity, and psychological safety.
• • Stay ahead of the curve by evaluating emerging technologies—Delta Lake, Flink, Iceberg, serverless Spark—and piloting those that can unlock new capabilities or reduce operational overhead.
• • Contribute to the open-source community by upstreaming bug fixes and enhancements, amplifying Freshworks’ reputation as a thought leader in big-data engineering.
• • Translate complex technical concepts into crisp narratives for stakeholders, ensuring that every roadmap item is tied to measurable business impact and ROI.