AWS Data Engineer

📋 Description

• • Qodeworld is seeking a highly skilled and experienced AWS Data Engineer to join our dynamic team. This pivotal role will be instrumental in designing, developing, and optimizing large-scale data pipelines and ETL (Extract, Transform, Load) workflows, primarily leveraging the power of Amazon Web Services (AWS) cloud-native data services. The ideal candidate will possess a deep understanding of data modeling principles, pipeline orchestration, and have a proven track record of building robust, scalable, and efficient data solutions tailored for enterprise-level environments.
• • As an AWS Data Engineer, you will be at the forefront of our data infrastructure, taking ownership of the end-to-end data lifecycle. Your primary responsibilities will involve the design, development, and meticulous maintenance of sophisticated ETL pipelines. You will be proficient in utilizing AWS services such as AWS Glue, its visual development interface Glue Studio, and the comprehensive data cataloging capabilities of Glue Catalog to build and manage these pipelines.
• • A significant part of your role will focus on the ingestion, transformation, and loading of vast datasets. These datasets will originate from a diverse array of structured and unstructured sources, and you will be responsible for efficiently moving them into our AWS data lakes and data warehouses. This involves working extensively with core AWS data services including Amazon S3 for scalable object storage, Amazon Redshift for high-performance data warehousing, Amazon Athena for interactive query services, AWS Lambda for serverless compute, and AWS Step Functions for orchestrating complex workflows.
• • You will be tasked with building and optimizing complex data transformation logic using PySpark and Scala within the AWS Glue environment. This requires a strong command of these programming languages and their application in distributed data processing frameworks. The ability to write efficient, maintainable, and scalable code is paramount.
• • Ensuring the integrity and reliability of our data is a critical aspect of this role. You will implement robust data quality checks, establish clear data lineage tracking, and set up comprehensive monitoring across all data pipelines. This proactive approach will help identify and resolve issues before they impact downstream users.
• • Collaboration is key to success in this position. You will work closely with business analysts to understand data requirements, data scientists to provide them with the data they need for advanced analytics and machine learning models, and product teams to ensure our data solutions align with business objectives and product roadmaps. Delivering reliable, high-quality data solutions that drive business value will be your ultimate goal.
• • Adherence to stringent data security, governance, and regulatory requirements is non-negotiable. While experience in the BFSI (Banking, Financial Services, and Insurance) sector is preferred, a strong understanding of data compliance principles is essential for all candidates. You will contribute to maintaining a secure and compliant data environment.
• • You will also be responsible for troubleshooting production issues that arise within the data pipelines and implementing optimizations to enhance pipeline performance, reduce latency, and minimize costs. This includes performance tuning of queries, data processing jobs, and overall data architecture.
• • This role offers a fantastic opportunity to work with cutting-edge cloud technologies and contribute to building a state-of-the-art data platform that empowers Qodeworld's data-driven initiatives. You will be part of a forward-thinking organization that values innovation and technical excellence.