Data Engineer

About the Role

The Data Engineer role supports the development, maintenance, and optimization of data solutions using the Databricks platform. This is an entry-level data engineering role. The engineer will work as part of an Agile development team and collaborate with experienced data engineers, analysts, and business partners to build reliable data pipelines, data models, and analytics-ready datasets.

Key Responsibilities

• Develop and maintain data pipelines and data transformations using Databricks, SQL, Python, and PySpark.
• Assist with the ingestion, transformation, and processing of structured and semi-structured data from multiple source systems.
• Develop and maintain Databricks notebooks, pipelines, workflows, and jobs supporting data engineering and analytics use cases.
• Build and maintain data tables and datasets using Delta Lake and lakehouse architecture principles.
• Write, test, troubleshoot, and optimize SQL queries for data processing, analysis, and reporting.
• Support the development of ETL/ELT pipelines that move and transform data across source systems, cloud storage, Databricks, and downstream applications.
• Work within an Agile Scrum team to design, develop, unit test, debug, document, and maintain data solutions.
• Collaborate with senior engineers and architects to ensure solutions follow established architecture, development standards, security requirements, and engineering best practices.
• Assist with defining data models, data structures, transformation logic, and information flows.
• Perform data validation and troubleshooting to help ensure the quality, consistency, completeness, and accuracy of data products.
• Monitor data pipelines and Databricks workloads and assist with troubleshooting pipeline failures and data-related issues.
• Participate in root-cause analysis and resolution of data and application issues with cross-functional teams.
• Use Git and established CI/CD practices to manage and deploy data engineering code and configurations.
• Create and maintain technical documentation for data pipelines, transformations, data models, and other data products.
• Collaborate with business and technical teams supporting areas such as Finance, Supply Chain, and Operations to understand data requirements and deliver appropriate data solutions.
• Support data solutions spanning ingestion, transformation, data modeling, analytics, and reporting.
• Develop knowledge of data governance, data security, access controls, and data quality management practices.
• Participate in production support activities on a rotating basis as needed.
• Can work remotely.

Requirements & Tech Stack

• Foundational knowledge of Databricks or similar cloud-based data platforms.
• Experience or coursework using SQL for querying and transforming data.
• Experience or coursework using Python for data processing or software development.
• Familiarity with Apache Spark or PySpark concepts.
• Understanding of ETL/ELT and data pipeline concepts.
• Basic understanding of relational databases and data modeling.
• Familiarity with cloud computing concepts and services, preferably AWS.
• Familiarity with Git source control and software development practices.
• Understanding of basic data quality concepts, including validation, completeness, consistency, and accuracy.
• Strong analytical and problem-solving skills.
• Ability to learn new technologies and work collaboratively within a technical team.

Preferred Knowledge Or Experience

• Databricks notebooks, workflows, jobs, or Databricks SQL.
• Delta Lake and lakehouse architecture concepts.
• PySpark DataFrames and Spark SQL.
• Databricks Unity Catalog and data governance concepts.
• AWS services such as S3, IAM, Glue, Athena, or related data services.
• Apache Airflow or other workflow orchestration technologies.
• CI/CD concepts and automated deployment practices.
• Data visualization tools such as Power BI.
• Streaming and event-driven technologies such as Apache Kafka.
• Data warehouse and data lake concepts.
• Data governance, metadata management, and access-control concepts.
• Databricks or cloud-provider certifications.

Qualifications

Minimum Education Requirement: Bachelor’s degree in Computer Science, Information Technology, Data Science, Engineering, Information Systems, or a related field of study. Equivalent education, technical training, certifications, internships, or relevant practical experience may also be considered.