AWS Data Engineer

About the Role

• Design, develop, and maintain scalable data pipelines on AWS.
• Develop ETL/ELT workflows for data ingestion, transformation, and processing.
• Build data processing solutions using PySpark and SQL.
• Perform data transformations, validations, and data quality checks.
• Optimize data pipelines and SQL queries for performance and scalability.
• Work with different data sources and integrate data into cloud-based data platforms.
• Troubleshoot data pipeline failures and resolve data-related issues.
• Collaborate with business, analytics, and engineering teams to understand data requirements.
• Ability to independently understand requirements and deliver data engineering solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines on AWS.
• Develop ETL/ELT workflows for data ingestion, transformation, and processing.
• Build data processing solutions using PySpark and SQL.
• Perform data transformations, validations, and data quality checks.
• Optimize data pipelines and SQL queries for performance and scalability.
• Work with different data sources and integrate data into cloud-based data platforms.
• Troubleshoot data pipeline failures and resolve data-related issues.
• Collaborate with business, analytics, and engineering teams to understand data requirements.

Requirements & Tech Stack

• Strong hands-on experience with AWS Data Engineering services:
• S3
• Glue
• Lambda
• EMR
• Redshift
• Athena
• IAM
• Strong programming experience in PySpark.
• Excellent knowledge of SQL, including complex queries and data transformations.
• Hands-on experience in developing ETL/ELT pipelines.
• Good understanding of data warehousing, data lakes, and data modeling concepts.
• Experience handling large-scale datasets and optimizing distributed data processing workloads.
• Good analytical and problem-solving skills.
• Strong communication and collaboration skills.

Qualifications

• Strong hands-on experience with AWS Data Engineering services such as S3, Glue, Lambda, EMR, Redshift, Athena, and IAM.

• Strong programming experience in PySpark.

• Excellent knowledge of SQL, including complex queries and data transformations.

• Hands-on experience in developing ETL/ELT pipelines.

• Good understanding of data warehousing, data lakes, and data modeling concepts.

• Experience handling large-scale datasets and optimizing distributed data processing workloads.

• Good analytical and problem-solving skills.

• Strong communication and collaboration skills.

• Ability to independently understand requirements and deliver data engineering solutions.

• Knowledge of Python is preferred.

• Familiarity with Git, CI/CD, orchestration tools, and Agile development practices will be an advantage.

• Exposure to cloud migration or modernization projects would be an added advantage.