Databricks
Virtusa
Hyderabad, Telangana, India6d agovia LinkedIn
Databricks Pipeline Developers
Key Responsibilities
- Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage.
- Develop PySpark pipelines for:
- API ingestion
- FTP file ingestion
- Data cleansing
- Enrichment transformations
- Implement incremental & batch loads.
- Develop Bronze → Silver → Gold transformations.
- Translate legacy SQL stored procedures into Spark jobs.
- Optimize Delta Lake tables.
- Implement validation checks.
- Support testing & reconciliation.
Requirements & Tech Stack
- Strong PySpark / Spark SQL.
- Strong SQL (read & interpret legacy procedures).
- Experience with API integration.
- Experience handling flat files (FTP ingestion).
- Experience with Delta Lake.
- Knowledge of partitioning & optimization.