Senior Data Platform Engineer, Data Cloud

📋 Description

• • Join Chainalysis, a trailblazer in the cryptocurrency space, as a Senior Data Platform Engineer on our dynamic Data Cloud team. This is an exceptional opportunity to shape the future of data infrastructure in a rapidly evolving industry. Our engineering culture is defined by a passion for tackling complex technical challenges and building products that foster trust and transparency in cryptocurrencies. We are a global, collaborative team, united by a drive for excellence and a commitment to continuous learning and innovation.
• • As a Senior Data Platform Engineer, you will play a pivotal role in designing, implementing, and optimizing our petabyte-scale data storage, processing, and querying platforms. Your work will directly empower every individual at Chainalysis to discover, access, analyze, and build upon our vast datasets with speed and reliability. This position is crucial for accelerating innovation, unlocking new business lines, and enabling high-performance interactive analytics and real-time streaming data applications.
• • You will be instrumental in defining the technical vision for the Data Cloud team and the broader organization. This involves articulating how our data platform and architecture can evolve to meet future demands and maintain our competitive edge. Your expertise will guide the strategic direction of our data infrastructure, ensuring it remains scalable, robust, and adaptable.
• • A core responsibility will be the design and implementation of our high-performance, scalable data serving platform. This platform is the backbone for data querying and consumption, supporting both internal analytics and external-facing data products. You will ensure that data is accessible, performant, and reliable for a wide range of users and applications.
• • Simultaneously, you will focus on designing and optimizing our high-performance, scalable data storage and transformation platform. This involves architecting solutions that handle both batch and stream processing, accommodating an immense volume of data with potentially hundreds of millions of updates per day on datasets exceeding 100 billion rows. Your efforts will ensure the integrity and efficiency of our core data assets.
• • You will build seamless integrations between our Data Cloud environment and various relational and NoSQL OLTP databases. This cross-system connectivity is vital for a holistic view of our data and for enabling comprehensive analysis.
• • Developing robust batch and streaming data pipelines for our core blockchain datasets will be a key deliverable. These pipelines are fundamental to the company's operations and analytics capabilities, requiring meticulous design and implementation.
• • Deploying and managing cloud infrastructure at scale with enterprise-grade reliability is paramount. This includes implementing and maintaining infrastructure automation, enabling self-service capabilities for data teams, and creating sophisticated CI/CD pipelines to ensure efficient and safe deployments.
• • Establishing and maintaining comprehensive observability, security, and data governance solutions is critical. These measures will guarantee the high quality, efficiency, security, and reliability of our data pipelines and the data they manage, building trust and compliance.
• • We are seeking individuals with a strong passion for leading and contributing to the technical vision of a team. You should possess a deep sense of ownership for mission-critical systems and a dedication to continuously honing your craft while actively mentoring junior engineers. Your leadership and technical acumen will be vital in elevating the team's capabilities.
• • Your experience should encompass building and maintaining both batch and streaming data pipelines, leveraging technologies such as DBT, Databricks, Apache Spark, and Apache Flink. A profound understanding of data architecture and data modeling best practices is essential.
• • Expertise with AWS services, cloud architecture, and fault-tolerant distributed data systems is required. Proficiency with Terraform for provisioning and managing cloud infrastructure will be a significant advantage.
• • A deep understanding of modern data lakehouse architectures and their associated ecosystems, including Kafka, Flink, Spark, Databricks, Snowflake, DBT, Airflow, Debezium, Delta Iceberg, StarRocks, and Clickhouse, is expected. Proficiency in Python, Java, and SQL is also a must.
• • Experience in building tools and frameworks that accelerate the development of data pipelines will be highly valued. Familiarity with data governance, data quality, and observability best practices is also important.
• • This role offers the chance to work with cutting-edge technologies in a fast-paced, innovative environment. You will be part of a team that is making a significant impact on the future of blockchain technology and its adoption worldwide. Your contributions will directly influence how businesses and governments interact with and leverage blockchain data.