Data Engineer, PySpark

Key Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.

Requirements & Tech Stack

• PySpark
• Spark
• SQL
• Databricks
• Azure Data Factory (ADF)
• Python
• Azure Data Lake Storage (ADLS)
• Azure SQL
• CI/CD