AWS Databricks Developer / Data Engineer

Key Responsibilities

• Hands-on experience with Databricks development and large-scale data processing.
• Strong expertise in Data Engineering and ETL/ELT pipeline development.
• Experience building batch and near-real-time data ingestion pipelines.
• Ability to collaborate with architects, BI teams, and data science teams.

Requirements & Tech Stack

• Proficiency in Apache Spark, PySpark, Spark SQL, Python, Scala, and Power BI.
• Hands-on experience with Delta Lake, ACID transactions, schema evolution, and performance optimization.
• Experience with AWS cloud storage services.
• Knowledge of coding standards, documentation, and data engineering best practices.
• Strong understanding of analytical and dimensional data modelling (Star and Snowflake schemas).

Qualifications

• Exposure to ML/AI workloads and feature engineering on Databricks.
• Familiarity with MLflow or similar model-tracking tools.
• Experience with Databricks CI/CD and Git-based version control.
• Understanding of data governance, security, and compliance.
• Familiarity with Agile/Scrum methodologies.
• Insurance domain experience is an added advantage.