Data Engineer

About the Role

Experienced Data Engineer to analyze, design, and develop enterprise-scale data pipelines across legacy and strategic cloud data platforms. A key component of the position is analyzing existing data pipelines, particularly Omnia-based Spark pipelines, understanding the underlying business and transformation logic, and translating that logic into scalable data solutions on modern cloud platforms.

Key Responsibilities

• Conduct detailed analysis of existing end-to-end data pipelines across multiple platforms, with a particular focus on Omnia - PySpark-based processing.
• Review existing source code, SQL, transformation rules, mappings, and data flows to understand the current business and technical logic.
• Derive, document, and implement new transformation logic based on the analysis of existing pipelines and business requirements.
• Design, develop, and maintain reliable, scalable, and reusable data pipelines using AWS, Snowflake, and dbt.
• Develop batch and, where required, event-driven data processing solutions using cloud-native and distributed data technologies.
• Translate existing Omnia data processing logic into strategic cloud-based data engineering solutions.
• Build ingestion, transformation, enrichment, and data integration pipelines across data lakes, data warehouses, and cloud lakehouse platforms.
• Develop complex and performance-optimized SQL for data transformation, reconciliation, validation, and analytical processing.
• Analyze end-to-end data flows from source capture through transformation, enrichment, and downstream consumption.
• Work with data designers, product owners, and subject matter experts to translate business requirements into technical data rules and durable engineering solutions.
• Implement data quality, reconciliation, and balancing controls across data pipelines.
• Troubleshoot data pipeline failures, transformation issues, data discrepancies, and performance bottlenecks across multiple environments.

Requirements & Tech Stack

• Strong hands-on experience designing, developing, and supporting scalable enterprise data pipelines.
• Strong Spark-etl experience, including the ability to analyze existing Omnia pipelines, understand their processing logic, and derive new transformation and business rules.
• Strong working knowledge of AWS services, including S3, Glue, EMR, Lambda, Athena, Step Functions, SQS, IAM, Secrets Manager, and CloudWatch, combined with hands-on experience in orchestrating data pipelines using Apache Airflow.
• Practical experience with Snowflake, covering data ingestion, data modeling, transformation, and performance optimization.
• Experience using dbt to develop data models, incremental processing, reusable macros, automated tests, documentation, and dependency management.
• Excellent SQL skills, with experience in complex data transformations, query optimization, reconciliation, and understanding existing business logic.
• Good Python programming skills for data processing, automation, and reusable framework components.
• Familiarity with Git, automated testing, CI/CD, Jenkins, and DevSecOps practices within a data engineering environment.
• Previous CBA experience, particularly with Omnia or CBA data platforms and engineering standards, is highly desirable.

Qualifications

• Strong hands-on experience with AWS, Snowflake, dbt, Airflow, PySpark, Python, and advanced SQL.