Data Engineer (Databricks), Software Engineer

About the Role

• Position: Software Engineer
• Category: Software Engineering
• Shift: General Shift
• Work Mode: Hybrid
• Primary Location: Bangalore
• Employment Type: Full Time

Key Responsibilities

• Databricks Architecture Design: Design scalable and reliable data analytics and machine learning architectures leveraging the Databricks platform.
• Platform Implementation: Implement and configure Databricks clusters, workspaces, notebooks, and other components.
• Data Engineering: Develop and optimize data pipelines for ingesting, processing, and transforming large volumes of data using Databricks and related technologies.
• Data Modeling: Design and implement data models, schemas, and storage structures within Databricks.
• Machine Learning: Build and deploy machine learning models using Databricks ML frameworks and libraries.
• Performance Optimization: Optimize Databricks jobs, queries, and workflows for performance, scalability, and cost efficiency.
• Security and Governance: Implement security controls, access management policies, and data governance frameworks within Databricks.
• Collaboration: Understand requirements, design solutions, and deliver actionable insights using Databricks by collaborating with data engineers, data scientists, business analysts, and stakeholders.
• Documentation and Training: Document Databricks architecture, design decisions, configurations, and best practices, and provide guidance and training to team members.
• Continuous Improvement: Stay abreast of the latest developments in Databricks and related technologies, evaluate new features, and recommend enhancements.

Requirements & Tech Stack

• Databricks
• Apache Spark
• SQL
• Python
• Scala
• AWS, Azure, or Google Cloud Platform (focus on data services and analytics)
• Apache Kafka
• Apache Airflow
• Apache Hadoop
• Knowledge of containerization technologies (e.g., Docker, Kubernetes) is a plus.
• Knowledge of infrastructure as code tools (e.g., Terraform) is a plus.

Qualifications

• Educational Qualification: BE/B.TECH/ME/M.TECH
• Proven experience designing, implementing, and managing data analytics and machine learning solutions using Databricks.
• Strong proficiency in Apache Spark, SQL, Python, and Scala, with hands-on experience developing data pipelines and machine learning models.
• Expertise in cloud platforms such as AWS, Azure, or Google Cloud Platform, with a focus on data services and analytics offerings.
• Experience with data engineering tools and frameworks such as Apache Kafka, Apache Airflow, and Apache Hadoop.
• Excellent problem-solving and analytical skills, with the ability to troubleshoot complex issues and optimize performance in distributed systems.
• Effective communication and collaboration skills, with the ability to work with cross-functional teams in a dynamic environment.
• Experience in Oil & Gas domain is preferable.
• Databricks certification (e.g., Databricks Certified Associate Developer, Databricks Certified Associate Data Scientist) is desirable but not mandatory.