AI Validation and Benchmarking Engineer

About the Role

This contract role requires supporting the end-to-end validation pipeline for AI tools. You will maintain test datasets, run benchmarks, and measure agent accuracy across routing decisions, classification labels, and structured output fields. This is a part-time student worker position, committed to 40 hours per week, and is a contract position through a vendor.

Key Responsibilities

• Run and maintain the benchmark pipeline, analyzing results to identify routing errors and regressions across agent variants.
• Build and expand ground truth datasets used to evaluate agent outputs against known-correct answers.
• Identify and address gaps in benchmark validation and support building a more comprehensive evaluation infrastructure to improve validation prior to release.
• Develop new evaluation dimensions such as label accuracy and structured output correctness beyond the existing team classification benchmarks.
• Investigate failure modes in agent outputs and work with engineers to surface actionable improvements.
• Write scripts and tooling to automate data collection, result parsing, and metric reporting.
• Document findings, track benchmark trends over time, and present results to the team.

Requirements & Tech Stack

• Python
• Familiarity with evaluation concepts: precision, recall, F1 score, and confusion matrices.
• Comfortable working with structured data (e.g., CSV, JSON).
• Experience modifying or writing reproducible analysis scripts.
• Familiar with Cursor or Claude.

Qualifications

• Currently enrolled in a B.S. or M.S. in Computer Science, Data Science, Engineering, or a related field.
• Available to commit to a minimum of three months.
• Able to commit to a minimum of 20 hours per week.
• Able to work on-site at one of our office locations.
• Must adhere with Zoox confidentiality requirements.

Bonus Qualifications:

• Prior exposure to LLM-based systems, prompt engineering, or AI agent evaluation.
• Experience with Jira or Slack.