Data Engineer-Data Platforms-AWS

Data Engineer - Data Platforms - AWS

Company: IBM

About the Role

As a Data Engineer specializing in Data Platforms on AWS, you will advise on, develop, and maintain data engineering solutions on the AWS Cloud ecosystem. You will design, build, and operate batch and real-time data pipelines using various AWS services.

Key Responsibilities

• Design, build, and operate batch and real-time data pipelines using AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis.
• Create data layers on AWS RedShift, Aurora, and DynamoDB, and migrate data using AWS DMS.
• Schedule and manage data services on the AWS Platform.
• Develop batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.
• Utilize open-source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on AWS Platform.

Requirements & Tech Stack

• Exposure to AWS services: AWS EMR, AWS Glue, Glue Catalog, Kinesis.
• Experience developing batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.
• Experience with AWS RedShift, Aurora, and DynamoDB to create data layers and migrate data using AWS DMS.
• Exposure to open-source technologies: Apache Airflow, dbt, Spark / Python, Spark / Scala on AWS Platform.
• Experience scheduling and managing data services on the AWS Platform.

Qualifications

Preferred Education: Master's Degree

• Proficiency with AWS Glue Databrew for data preparation and data quality tasks.
• Exposure to using Lambda functions with Python for data processing and transformation.
• Experience with RedShift Spectrum for data warehousing and data analytics.