Applied Scientist
Amazon Web Services (AWS)
Bellevue, WA2h agovia LinkedIn
This job at a glance
- Who can apply
- New grads, no experience needed · Master's degree
- Must know
- Machine learningLLMsStatisticsAuditingStakeholder managementPython
- Work
- On-site in Bellevue
- Pay
- Not stated by the employer.
- Skill demand
- Statistics27%Machine learning22%LLMs8%Auditing6%of 99 similar entry-level jobs posted in the US this week
- Listed
- Oct 29, 2026 · still open
Sign in free to see which of these skills your resume already shows.
About the Role
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.
Key Responsibilities
- Collaborate closely with core scientist teams developing Amazon Nova models.
- Lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
- Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies to help Nova improve performances on benchmarks.
- Perform expert-level manual audits.
- Conduct meta-audits to evaluate auditor performance.
- Provide targeted coaching to uplift overall quality capabilities.
- Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
- Set up the configuration of data collection workflows.
- Communicate quality feedback to stakeholders.
- Enhance customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
- Support quality solution design.
- Conduct root cause analysis on data quality issues.
- Research new auditing methodologies.
- Find innovative ways of optimizing data quality.
- Set examples for the team on quality assurance best practices and standards.
- Work closely with engineers, domain experts, and vendor teams to implement quality strategies and automated judging systems.
Requirements & Tech Stack
- Experience programming in Java, C++, Python or related language.
- Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.
- Experience implementing algorithms using both toolkits and self-developed code.
- Publications at top-tier peer-reviewed conferences or journals.
Qualifications
- Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.