ML Engineer & Research Scientist, Speech

About the Role

Voice Arena is seeking an ML Engineer & Research Scientist to define, measure, and advance the capabilities of machines that talk like humans. This role combines fundamental research with hands-on engineering, empowering you to own the end-to-end process of defining speech model behaviors, building evaluation metrics, designing data, and training models to prove their effectiveness. You will work directly with the founders in a small, talent-dense team with significant ownership and public impact.

Key Responsibilities

• Identify problems in speech: Pinpoint gaps in current speech models by analyzing failures, reviewing literature, and engaging with users and labs. Translate findings into clear problem statements.
• Define desired model behavior: Precisely articulate how models should behave for each identified problem, establishing targets for labs and developers.
• Design evaluations: Develop measurement tools, including test sets, automatic metrics, and human evaluation protocols, to assess model performance. Analyze current model performance against these evaluations.
• Define datasets: Specify data requirements for training models to acquire missing behaviors, including recording, sourcing, languages, conditions, and labeling. Oversee data quality.
• Train models: Train and fine-tune models on newly defined datasets to demonstrate effectiveness and improve performance on internal and public leaderboards.
• Publish and open-source: Document research, evaluations, data, and results, releasing models for the community to build upon.

In a typical week, you might

• Analyze model outputs to identify undiscovered failures.
• Develop specifications for new evaluations and review test data.
• Conduct training runs, analyze results, and perform ablations.
• Evaluate new model launches against existing benchmarks.
• Collaborate with researchers on their challenges.
• Prepare results for publications and leaderboard releases.

What matters most

• Building the right evals: Ensure that climbing an evaluation signifies genuine progress and that evals accurately measure a model's advancement towards human-like speech.

• Begin with desired model behavior, not just computable metrics.

• Validate scores against human preferences and use human evaluation where needed.

• Minimize opportunities for gaming evals with held-out test sets and by monitoring for contamination.

• Test across diverse conditions including languages, accents, code-mixing, noise, and natural conversation.

• Maintain rigor by reporting confidence intervals and distinguishing real effects from noise.

• Building the models: Train models on defined data to demonstrate that the chosen behaviors and designed data lead to demonstrable improvements and reproducible results.

• Train and fine-tune open ASR, TTS, and speech-to-speech models on custom datasets.

• Conduct clean ablations to attribute improvements to new data and behaviors.

• Achieve high rankings on internal and public leaderboards, such as Hugging Face's Open ASR Leaderboard.

• Train efficiently, ensuring reproducibility and optimal GPU utilization.

• Release open-source models with clear model cards for community adoption.

Requirements & Tech Stack

• Strong Python and PyTorch skills.
• Comfort working with large audio datasets.
• Statistical care: Ability to discern real performance differences from noise.
• Informed opinions about the future direction of voice AI.
• Solid engineering habits: Including version control, testing, and reproducible experiments and pipelines.

Qualifications

• An MS or PhD in speech, ML, or NLP, or an equivalent industry research record.
• Preferred: Papers at Interspeech, ICASSP, ACL, NeurIPS, or similar.
• Preferred: Experience with multilingual, low-resource, or code-mixed speech.
• Preferred: Work experience at a speech lab or voice AI company.
• Preferred: Experience developing a benchmark, dataset, or open-source tool used by the field.
• Preferred: Experience running GPU training or inference at scale.
• Preferred: Experience designing human evaluation or annotation studies.

Why join Voice Arena

• Shape the trajectory of voice AI development.
• Gain early access to and evaluate the newest speech models.
• Publish research in top venues like Interspeech and ICASSP, and gain visibility within the voice AI community.
• Contribute to measuring and advancing speech across a global scope, beyond English.
• Join an early-stage, talent-dense team and influence the product, team, and research agenda.