Senior Data Platform Engineer

📋 Description

• • Aspora is seeking a Senior Data Platform Engineer to architect, build, and maintain the critical data infrastructure that underpins our innovative borderless financial operating system. This role is pivotal in ensuring that data is processed reliably, insights are generated efficiently, and decisions across the company are data-driven. You will be instrumental in shaping the future of global banking by providing the foundational data systems that enable real-time analytics, large-scale batch computations, and seamless data movement, processing billions of events with precision and speed.
• • As a Senior Data Platform Engineer, your day-to-day responsibilities will encompass a broad range of activities focused on enhancing our data capabilities. You will be responsible for the design, development, and operation of large-scale data processing infrastructure, with a strong emphasis on Apache Spark running on Databricks, ensuring optimal reliability, performance, and cost-efficiency. This includes architecting and maintaining robust lakehouse solutions utilizing technologies like Delta Lake or Apache Iceberg, meticulously managing partitioning strategies, Z-ordering, and compaction jobs to maximize efficiency. A key aspect of your role will involve owning and optimizing cluster management, implementing sophisticated autoscaling policies, and ensuring effective resource governance across multiple Databricks workspaces. You will also be a driving force behind platform-level enhancements, focusing on query optimization, implementing advanced caching strategies, championing compute-storage separation, and fine-tuning shuffle operations to achieve peak performance.
• • Furthermore, you will design and construct resilient, idempotent, and thoroughly testable data pipelines capable of handling both batch and near-real-time workloads. This involves managing and expanding our Apache Airflow-based orchestration layer, establishing clear DAG authoring standards, ensuring robust dependency management, implementing comprehensive alerting mechanisms, and rigorously enforcing Service Level Agreements (SLAs). A significant part of your work will be implementing and maintaining Change Data Capture (CDC) pipelines using tools like Debezium, Kafka Connect, or native database replication methods, guaranteeing low-latency and high-fidelity data propagation across systems. You will also define critical data pipeline contracts, including schemas, SLAs, and quality assertions, and ensure these are strictly enforced through automated data quality frameworks.
• • In the realm of analytical storage and computation, you will model and manage analytical data stores, employing dimensional models, One Big Table (OBT) patterns, and aggregation layers meticulously optimized for Business Intelligence (BI) and self-serve analytics. You will own the strategic evolution of our analytical warehouse/lakehouse stack, conducting performance benchmarking, detailed cost modeling, and informed technology selection. Building and maintaining efficient data serving layers for dashboards, ML feature stores, and reverse ETL use cases will be a core responsibility. Additionally, you will implement and manage data retention, archival, and lifecycle management policies across various storage tiers, from hot to warm to cold.
• • Beyond data processing, you will play a crucial role in platform engineering and enhancing the developer experience for data professionals. This includes defining and enforcing best practices for data platform engineering, such as code standards, implementing CI/CD for data pipelines, establishing automated testing protocols, and ensuring robust observability. You will develop internal tooling and libraries designed to accelerate the work of data engineers, creating reusable Spark utilities, standardized pipeline templates, and streamlined local development environments. Championing data reliability engineering will be paramount, involving meticulous lineage tracking, developing incident response playbooks, monitoring pipeline SLOs, and conducting thorough root cause analyses to prevent future issues.
• • About the team and company: Aspora, founded in 2022, is rapidly building a borderless financial operating system designed for people on the move, making banking as mobile and transparent as its users. Backed by prominent venture capitalists, we are a growing team of over 75 professionals across India, the UK, the UAE, EU, and the US. Our culture is built on principles of extreme ownership, radical candor, and an unwavering obsession with customer impact. We foster an environment that celebrates builders who challenge assumptions, ship quickly, and transform regulatory complexities into elegant, user-friendly solutions. If you are passionate about redefining global banking, Aspora offers a dynamic and impactful environment to achieve this.
• • In this role, you will have the opportunity to gain deep expertise in cutting-edge big data technologies and cloud infrastructure, significantly advancing your skills in distributed systems, data modeling, and platform architecture. You will gain invaluable experience in building and operating highly scalable and reliable data systems in a fast-paced, high-growth startup environment, contributing directly to a product that is revolutionizing financial services for a global audience. This position offers the chance to own critical components of the data platform, drive architectural decisions, and mentor other engineers, fostering significant professional growth and leadership development.