Data Engineer

📋 Description

• • Astreya Partners is seeking a highly skilled and experienced Data Engineer with 8 to 10 years of professional experience to join our dynamic team. This role is pivotal in transforming architectural blueprints into robust, automated data solutions, with a strong emphasis on reliability, automation, and data quality. As a Data Engineer, you will be instrumental in building and maintaining the critical data infrastructure that powers our organization's insights and decision-making processes. You will work closely with architects and stakeholders to ensure that data flows seamlessly, is accurate, and is readily available for analysis and reporting.
• • Your primary focus will be on the end-to-end development and management of ETL/ELT pipelines. This involves designing, building, and maintaining processes that extract data from various source systems, transform it into a usable format, and load it into our data warehouse or data lake. You will be responsible for ensuring these pipelines are efficient, scalable, and resilient, capable of handling large volumes of data with high throughput.
• • A core aspect of this role is the implementation of automated data quality testing and observability. You will establish comprehensive testing frameworks to validate the accuracy, completeness, and consistency of data at various stages of the pipeline. Furthermore, you will set up robust alerting mechanisms using tools like Amazon CloudWatch to proactively identify and address any data quality issues or pipeline failures, ensuring minimal downtime and maximum data integrity.
• • You will play a key role in optimizing data storage formats to enhance query performance. This includes leveraging modern data lake technologies such as Parquet and Delta Lake, understanding their benefits for high-speed querying and analytical workloads. By carefully selecting and implementing appropriate storage strategies, you will ensure that our data is accessible and performant for downstream analytics and reporting tools.
• • Managing CI/CD pipelines for data code deployment is another critical responsibility. You will implement and maintain continuous integration and continuous delivery practices for all data-related code, including ETL scripts, data models, and testing frameworks. This ensures that changes are deployed efficiently, reliably, and with a high degree of confidence, fostering an agile development environment for data engineering.
• • The role also involves creating interactive visualizations and actionable insights tailored to specific user personas. You will collaborate with business analysts and stakeholders to understand their data needs and translate them into compelling visual narratives using tools like Amazon QuickSight. The goal is to empower users with clear, concise, and actionable information that drives informed business decisions.
• • Key deliverables for this position include production-ready, optimized data pipelines that are both scalable and maintainable. You will also be responsible for delivering cleaned, validated datasets that meet stringent quality standards. Furthermore, you will contribute to the development of interactive and intelligent dashboards that provide real-time insights and support strategic planning.
• • This role operates within the AWS ecosystem, leveraging a comprehensive suite of services. You will utilize AWS API Gateway and AWS Lambda for event-driven data ingestion and processing, AWS Glue ETL and AWS Glue Crawler for data transformation and cataloging, and Amazon S3 for scalable object storage. For data warehousing and analytics, you will work with Amazon Redshift and Amazon Athena, enabling powerful querying capabilities.
• • Data governance and security are paramount. You will implement and manage data access controls and security policies using AWS Lake Formation, AWS IAM, AWS CloudTrail, and Amazon CloudWatch, ensuring compliance with industry regulations and internal policies. The integration of AI and Analytics services, specifically AWS Bedrock, will also be explored to enhance data processing and insight generation capabilities.
• • The successful candidate will be a proactive problem-solver, adept at working independently and collaboratively in a remote environment. You will possess a strong understanding of data engineering best practices and a passion for building high-quality data solutions. This is an exciting opportunity to make a significant impact on Astreya Partners' data strategy and contribute to our continued growth and success.