AWS Data Engineer

Key Responsibilities

• Design and develop scalable AWS Data Engineering solutions and cloud-native data platforms.
• Build batch and real-time data pipelines using AWS Glue, PySpark, Databricks, S3, Kinesis, Redshift, and Kafka/MSK.
• Develop and optimize Databricks, Delta Lake, and Spark workloads for large-scale data processing with PySpark and Databricks transformations.
• Implement streaming solutions using Amazon Kinesis Data Streams, Firehose, and Analytics for near real-time data ingestion and processing.
• Handle high-volume (TB/PB scale) data pipelines with a focus on performance, scalability, and cost optimization.
• Develop complex SQL transformations, data models, and analytics-ready datasets.
• Implement CI/CD, Infrastructure as Code (Terraform/CloudFormation), and DevOps best practices.
• Ensure data quality, governance, security, monitoring, and production support.
• Collaborate with architects, stakeholders, and cross-functional teams in Agile delivery environments.
• Troubleshoot and optimize end-to-end AWS data platforms, streaming pipelines, and cloud infrastructure.

Requirements & Tech Stack

• Strong hands-on AWS Data Engineering experience across large-scale cloud data platforms.
• Expertise in AWS S3, AWS Glue, AWS Lambda, AWS Redshift, AWS Lake Formation, AWS CloudWatch, and AWS IAM.
• Strong experience in real-time and streaming data processing using Amazon Kinesis Data Streams, Kinesis Firehose, Kinesis Analytics, Amazon Kafka, or equivalent streaming technologies.
• Hands-on coding proficiency in Python, PySpark, and SQL with strong optimization skills.
• Experience building high-volume, low-latency data ingestion and transformation pipelines.
• Strong experience in Databricks, Delta Lake, Structured Streaming, and Spark Performance Tuning.
• Experience handling TB/PB-scale datasets, partitioning strategies, file optimization, schema evolution, and data lifecycle management.
• Strong understanding of Data Lake, Lakehouse, and Modern Data Platform architectures.
• Practical data modeling experience for analytical, operational, and reporting workloads.
• Experience implementing CI/CD pipelines using AWS CodePipeline, CodeBuild, CodeDeploy, GitHub Actions, or Jenkins.
• Strong troubleshooting skills for production issues, performance bottlenecks, and cost optimization.
• Excellent client communication, stakeholder management, and problem-solving skills.