Azure Data Engineer, Con

About the Role

The Azure Data Engineer is responsible for designing and developing data solutions/pipelines using Microsoft Azure cloud services. The role involves creating scalable data pipelines, integrating data from multiple sources, and implementing strong data security and privacy controls. The Azure Data Engineer will also monitor and troubleshoot data flows, optimize data storage and processing for performance and cost efficiency, and support AI and Generative AI (GenAI) initiatives by enabling high-quality and reliable data pipelines. The role requires interacting with multiple stakeholders and managing end-to-end data project delivery from project requirements, solutioning and execution and hand off to operations.

Key Responsibilities

• Own the architecture, design, and delivery of scalable data storage and processing solutions using Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, and Azure Databricks.
• Define and enforce data engineering standards, best practices, and design patterns across teams.
• Provide technical leadership and oversight for complex ETL/ELT pipelines.
• Lead data engineering support for AI and GenAI workloads, including preparation and management of training datasets, embeddings, and inference pipelines.
• Partner with data scientists, AI engineers, and solution architects to meet AI performance and governance requirements.
• Manage, mentor, and grow a team of Azure Data Engineers.
• Plan capacity, allocate work, and ensure timely delivery of initiatives.
• Drive adherence to Agile delivery practices.
• Drive optimization of storage, compute, and pipelines for performance and cost efficiency.
• Review and approve designs with long-term sustainability in mind.

Requirements & Tech Stack

• Azure cloud services, specifically data storage and data processing solutions.
• SQL, Python, and PySpark.
• ETL (Extract, Transform, Load) concepts.
• GenAI and LLM data concepts, including training data, embeddings, and inference pipelines.
• Data security and privacy best practices.
• Monitoring and troubleshooting data pipelines.

Qualifications

• Bachelor’s degree in computer science, Engineering, Information Systems, or a related technical field, or equivalent practical experience.
• Strong knowledge of data warehousing.
• Strong problem-solving and analytical skills.
• Effective communication skills to collaborate with cross-functional teams.
• Experience with data governance is a plus.
• Proven experience leading and mentoring teams.
• Strong communication and stakeholder management skills.