Human-Evaluation Specialist

Job Title: Human-Evaluation Specialist

Department: Data & Evaluation

Overview

Human-Evaluation Specialists oversee human-in-the-loop evaluations and annotation tasks.

Key Responsibilities

• Design evaluation protocols: Define evaluation tasks, criteria, and scoring guidelines.
• Review AI outputs: Assess AI-generated outputs for quality, usability, and correctness.
• Identify inconsistencies: Detect errors, hallucinations, and ambiguous results.
• Provide structured feedback: Document issues and provide feedback to developers for model improvements.
• Contribute to workflow improvements: Suggest improvements to evaluation workflows and training data collection.

Qualifications

• Bachelor's degree in human-computer interaction or a related field.
• Experience in annotation tasks and quality assurance processes.