Data Engineer, ETL, SQL, Databricks
Optum India
Hyderabad, Telangana, India55m agovia LinkedIn
Key Responsibilities
- Contribute hands on to the design, ETL development, and operation of cloud native data pipelines on Azure and Databricks.
- Design and optimize Spark based ETL frameworks for large scale batch and incremental processing.
- Implement complex transformations and performance tuning using Python and Spark SQL.
- Modernize on-prem ETL workloads to Databricks/snowflake, ensuring enterprise standards for scalability, reliability, and governance.
- Collaborate with cross-functional teams to understand complex data requirements and deliver tailored solutions.
- Ensure data quality and integrity by implementing robust data validation and monitoring processes.
- Develop comprehensive documentation for data engineering processes and systems.
Requirements & Tech Stack
- Databricks
- Cloud native data platforms (preferably on Azure)
- ETL tools (e.g., DataStage)
- Python
- PySpark
- Spark SQL
- SQL
- Relational and distributed database systems
- Azure / AWS / GCP
- Workflow orchestration tools such as Airflow
Qualifications
- Bachelor's degree (B.E./B.Tech) in any engineering or related discipline.
- Hands on experience with Databricks and cloud native data platforms, preferably on Azure.
- Experience with ETL tools and large scale data warehouse implementations.
- Experience migrating or modernizing on-prem ETL workloads to cloud architectures.
- Solid experience building ETL pipelines using Python, PySpark, and Spark SQL.
- Solid understanding of data warehousing concepts, including dimensional modeling, data quality, reconciliation, and performance tuning.
- Familiarity with cloud ecosystems (Azure/AWS/GCP) and workflow orchestration tools such as Airflow.
- Proficient in SQL and working with relational and distributed database systems.
- Demonstrated solid analytical and problem-solving skills with the ability to deliver scalable, reliable data solutions.
- Demonstrated ability to collaborate effectively across teams; leadership or mentoring experience is a plus.