Data Engineer

About the Role

The Data Engineer works as part of an agile team supporting the organization's Research and Managed Services. This is a hands-on, hybrid, technical, and operational role. The team members will actively write and maintain code to build data tools, automations, and pipelines, while also performing day-to-day operational tasks that keep Research and Managed Services running reliably. The role includes identifying and evaluating internal and external data sources to feed AI-enabled and traditional research workflows. The team member will assess source relevance, quality, coverage, accessibility, reliability, compliance considerations, and applicability to defined business and technical use cases. The role drives best-in-class standards and continuous improvement and requires an adaptable professional who is willing and able to learn and adopt new technologies as they are introduced to the organization.

Key Responsibilities

Coding & Development

• Write, review, test, and maintain code, including SQL and Python, to build data tools, automations, and ingestion and transformation workflows that support Research and Managed Services.
• Automate manual processes and develop data tools to improve efficiency, accuracy, quality, and throughput.
• Develop and promote coding standards and contribute to code reviews within the agile team.
• Build and maintain web-scrapping solutions, API integrations, and reusable data-processing components.
• Support scalable ETL/ELT pipelines for structured and unstructured data.

Source Evaluation & AI Enablement

• Identify prospective sources that can feed AI solutions and Research and Managed Services workflows.
• Define and apply source-evaluation criteria covering relevance, authority, freshness, completeness, coverage, consistency, accessibility, legal or licensing constraints, privacy, security, and technical compatibility.
• Perform source profiling, sample validation, proof-of-concept testing, and comparative assessments before recommending onboarding.
• Document source decisions, metadata, lineage, ownership, limitations, refresh expectations, and approved use cases.
• Implement and support AI-enabled workflows using LangChain or equivalent orchestration frameworks, large language models, embeddings, retrieval-augmented generation, vector databases, and prompt-engineering approaches where applicable.
• Monitor source and AI-workflow performance and recommend remediation, replacement, or additional sources when quality or coverage falls below requirements.

Operational Tasks

Data & Analytics

• Perform day-to-day operational activities supporting Research and Managed Services, including monitoring, exception handling, data maintenance, and issue resolution.
• Perform database administration activities, including performance tuning and implementation of best practices.
• Implement new data-maintenance processes and provide end-to-end process ownership.
• Ensure data integrity by validating, reconciling, and regularly cleaning data.
• Investigate and resolve production incidents, pipeline failures, data-quality issues, and operational exceptions.
• Follow applicable data governance, security, and operational standards.

Collaboration & Continuous Learning

• Evaluate and implement new technology solutions, and proactively learn and adopt new tools, platforms, and methodologies introduced by the organization.
• Communicate with stakeholders and conduct knowledge-exchange sessions for technical and non-technical audiences.
• Develop and maintain data documentation, including data dictionaries, source assessments, data-flow diagrams, data mappings, runbooks, and data lineage.
• Collaborate with cross-functional teams across Data & Analytics, Technology, Research Services, Managed Services, Product, and Data Governance.
• Additional duties as assigned.

Requirements & Tech Stack

• SQL
• Python
• API integrations
• ETL/ELT pipelines
• AI solutions
• AI-enabled workflows
• LangChain (or equivalent)
• Large language models
• Embeddings
• Retrieval-augmented generation
• Vector databases
• Prompt-engineering
• Web-scraping solutions
• Data processing components
• Data governance
• Data dictionaries
• Source assessments
• Data-flow diagrams
• Data mappings
• Runbooks
• Data lineage