PySpark Data Engineer, Databricks
Infosys
Hyderabad2–5 yrs23h agovia Naukri
Key Responsibilities
- Design, develop, and maintain data pipelines.
- Build scalable data processing solutions.
- Develop ETL/ELT workflows for data ingestion, transformation, and loading.
- Work with data storage and Azure data services.
- Optimize Spark jobs and Databricks notebooks for performance and scalability.
- Implement data quality checks, validation rules, and monitoring processes.
- Create and manage end-to-end data integration solutions across multiple data sources.
- Collaborate with business stakeholders and data teams to understand data requirements.
- Troubleshoot and resolve data pipeline and performance issues.
- Follow coding standards, version control, and CI/CD best practices.
- Support data warehousing and analytics initiatives.
- Prepare technical documentation and maintain deployment artifacts.
Requirements & Tech Stack
- PySpark
- AWS
- Databricks
- ETL
- Azure Data Factory (ADF)
- Python
- Azure Databricks
- Azure Data Lake Storage (ADLS)
- Azure SQL