Distinguished Architect, Data Platform

📋 Description

• • CloudZero is seeking a Distinguished Architect for its Data Platform team to tackle complex infrastructure challenges and shape the future of its data processing capabilities. This pivotal role is designed for an experienced architect who thrives on building and scaling sophisticated data systems, directly impacting customer value and business growth. You will be instrumental in evolving CloudZero from its current batch-oriented pipelines to a cutting-edge, streaming-first architecture, enabling near real-time cost attribution for engineers.
• • As a Distinguished Architect, your day-to-day responsibilities will be deeply rooted in hands-on platform architecture. You will lead the end-to-end technical design for CloudZero's next-generation data platform, encompassing everything from event ingestion and stream processing to hot/cold storage and the query layer, culminating in the API surface. This involves meticulously documenting architectural decisions, outlining tradeoffs, and defining robust migration strategies through a rigorous RFC-driven process. You will be responsible for shaping and driving every critical layer of this new architecture, including event ingestion mechanisms, stream processing and enrichment pipelines, real-time serving capabilities, analytical storage solutions, the query layer, and the overarching API. A significant part of your role will be to drive the development and production deployment of CloudZero's real-time data pipeline, from initial ingestion through enrichment to final serving. This includes establishing stringent Service Level Objectives (SLOs) for throughput, latency, and data correctness, and developing comprehensive operational playbooks to ensure the reliability and trustworthiness of this new system, which will ultimately replace the existing batch pipelines that are fundamental to the entire product. You will be tackling real-time streaming challenges at a massive scale, managing thousands of customers concurrently, with an unwavering focus on fault tolerance, backpressure awareness, and data correctness as absolute non-negotiables. Furthermore, you will address the critical dimension cardinality problem by redesigning CloudZero's dimensional cost model. This redesign aims to support high-cardinality, multi-dimensional cost attribution without incurring prohibitive materialization costs. You will drive incremental, delta-based materialization strategies leveraging modern open table formats, significantly reducing expensive full-rebuild jobs and unlocking substantial annual infrastructure savings. Your responsibilities will also extend to evolving the query layer. This involves assessing CloudZero's current query infrastructure, driving in-flight migrations to successful completion, and leading the future evolution of the query engine layer. You will own performance optimization across key areas such as partition pruning, predicate pushdown, and query planning, while also setting the strategic vision for how the query layer will scale effectively as data volumes increase tenfold. A core objective is to extend cost attribution capabilities to real-time, transforming CloudZero's proprietary cost attribution engine from its current batch-oriented model to one that assigns complex cost dimensions by team, feature, and customer within seconds of resource usage. This will require rethinking enrichment processes, data lineage, and correctness guarantees within a streaming context. Finally, you will play a crucial role in shaping the Data Engineering roadmap by partnering closely with product, infrastructure, and analytics engineering teams to define a multi-year vision for the data platform. You will build consensus across engineering leadership on foundational investments in critical technologies such as table formats, streaming frameworks, query engines, and schema management. You will also elevate the Data Engineering team by actively participating in architecture reviews, contributing to the development of design patterns and best practices, and mentoring senior and staff engineers through code reviews, pairing sessions, and structured feedback, ultimately aiming to enhance the collective engineering craft.
• • CloudZero is a rapidly growing SaaS company at the forefront of cloud cost management and FinOps. Our mission is to make efficient innovation a reality for every cloud-driven organization by transforming complex cloud cost data into actionable intelligence. We are solving problems of massive scale, business importance, and complexity in a market that critically needs our solutions. The Data Engineering team is a core part of this mission, responsible for building and maintaining the robust data infrastructure that powers our product and insights.
• • In this role, you will have the unparalleled opportunity to define and build the future of a large-scale data platform from the ground up, making foundational architectural decisions that will have a lasting impact. You will gain deep expertise in cutting-edge streaming technologies, modern data warehousing, and advanced data modeling techniques. This position offers significant professional growth by allowing you to tackle some of the most challenging data infrastructure problems in the industry, mentor a talented team, and directly influence the success of a high-growth SaaS company.