Applied Scientist
Amazon
About the Role
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality. This role focuses on building industry-leading technology with Large Language Models (LLMs) and multimodal systems.
Key Responsibilities
- Collaborate closely with the core scientist team developing Amazon Nova models.
- Lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
- Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies to improve performance on benchmarks.
- Perform expert-level manual audits.
- Conduct meta-audits to evaluate auditor performance.
- Provide targeted coaching to uplift overall quality capabilities.
- Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
- Set up the configuration of data collection workflows.
- Communicate quality feedback to stakeholders.
- Enhance customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
- Support quality solution design.
- Conduct root cause analysis on data quality issues.
- Research new auditing methodologies.
- Find innovative ways of optimizing data quality.
- Set examples for the team on quality assurance best practices and standards.
- Work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
Requirements & Tech Stack
- Experience programming in Java, C++, Python or related language.
- Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.
- Experience implementing algorithms using both toolkits and self-developed code.
Qualifications
- Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
- Publications at top-tier peer-reviewed conferences or journals.