Azure Data Engineer, AM

About the Role

The Azure Data Engineer is responsible for designing and developing data solutions/pipelines using Microsoft Azure cloud services. This includes creating scalable data pipelines, integrating data from multiple sources, and implementing strong data security and privacy controls. The role also involves monitoring and troubleshooting data flows, optimizing data storage and processing for performance and cost efficiency, and supporting AI and Generative AI (GenAI) initiatives by enabling high-quality and reliable data pipelines. The engineer will interact with multiple stakeholders and manage end-to-end data project delivery, from requirements and solutioning to execution and handoff to operations.

Key Responsibilities

• Own the architecture, design, and delivery of scalable data storage and processing solutions using Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, and Azure Databricks.
• Define and enforce data engineering standards, best practices, and design patterns across teams.
• Provide technical leadership and oversight for complex ETL/ELT pipelines.
• Lead data engineering support for AI and GenAI workloads, including preparation and management of training datasets, embeddings, and inference pipelines.
• Partner with data scientists, AI engineers, and solution architects to meet AI performance and governance requirements.
• Manage, mentor, and grow a team of Azure Data Engineers.
• Plan capacity, allocate work, and ensure timely delivery of initiatives.
• Drive adherence to Agile delivery practices.
• Drive optimization of storage, compute, and pipelines for performance and cost efficiency.
• Review and approve designs with long-term sustainability in mind.

Requirements & Tech Stack

• Experience with Azure cloud services, specifically data storage and data processing solutions.
• Strong knowledge of data warehousing and ETL (Extract, Transform, Load) concepts.
• Hands-on experience with SQL, Python, and PySpark.
• Understanding of GenAI and LLM data concepts, including training data, embeddings, and inference pipelines.
• Strong understanding of data security and privacy best practices.
• Experience with monitoring and troubleshooting data pipelines.
• Strong problem-solving and analytical skills.
• Effective communication skills to collaborate with cross-functional teams.
• Experience with data governance is a plus.
• Proven experience leading and mentoring teams.
• Strong communication and stakeholder management skills.

Qualifications

• Bachelor’s degree in computer science, Engineering, Information Systems, or a related technical field, or equivalent practical experience.