DE&A, Data Engineering Apache

About the Role

Design, develop, and implement scalable, resilient, and high-performance data pipelines and data architectures for enterprise data platforms.

Key Responsibilities

• Build and optimize ETL/ELT pipelines supporting batch, near-real-time, and streaming data processing requirements.
• Design and implement enterprise data ingestion and integration workflows using Apache NiFi, including data routing, transformation, orchestration, monitoring, and integration across heterogeneous data sources and platforms.
• Develop distributed data processing solutions using Apache Spark with Scala and/or Python (PySpark).
• Design and implement data solutions across relational database management systems (RDBMS) and modern data lakehouse technologies such as Apache Iceberg.
• Build and optimize federated and distributed SQL capabilities using Starburst and Trino query engines.
• Develop event-driven and real-time data processing solutions using Apache Kafka and Spark Streaming / Structured Streaming.
• Lead end-to-end delivery of enterprise-scale data engineering initiatives involving multiple engineering, product, architecture, and business teams.
• Strengthen platform reliability through automation, CI/CD, testing, monitoring, observability, documentation, and operational excellence.
• Improve data quality, performance, scalability, resiliency, and cost efficiency across data platforms and pipelines.
• Support enterprise data governance by maintaining metadata, technical and business documentation, data lineage, data quality controls, and secure access to data assets.
• Partner with security and governance teams to ensure data solutions meet enterprise requirements for privacy, security, access control, compliance, and data lifecycle management.
• Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable solutions to improve platform stability.
• Collaborate effectively with stakeholders across engineering, architecture, analytics, AI/ML, governance, product, and business organizations.

Requirements & Tech Stack

• Apache NiFi
• Apache Spark (Scala, Python (PySpark))
• RDBMS
• Apache Iceberg
• Starburst
• Trino
• Apache Kafka
• Spark Streaming / Structured Streaming
• SQL
• CI/CD
• AI/ML

Qualifications

• Experience designing, developing, and implementing scalable, resilient, and high-performance data pipelines and data architectures.
• Experience building and optimizing ETL/ELT pipelines.
• Experience designing and implementing data ingestion and integration workflows.
• Experience developing distributed data processing solutions.
• Experience designing and implementing data solutions across RDBMS and data lakehouse technologies.
• Experience building and optimizing federated and distributed SQL capabilities.
• Experience developing event-driven and real-time data processing solutions.
• Experience leading end-to-end delivery of data engineering initiatives.
• Experience strengthening platform reliability through automation, CI/CD, testing, monitoring, observability, documentation, and operational excellence.
• Experience improving data quality, performance, scalability, resiliency, and cost efficiency.
• Experience supporting enterprise data governance.
• Experience partnering with security and governance teams.
• Experience troubleshooting production issues and performing root-cause analysis.
• Experience collaborating with various stakeholders.