Agent Evaluation & Evolution Machine Learning Engineer Graduate
ByteDance
About the Role
The Applied Machine Learning Ark team combines system engineering and machine learning to develop and operate Large Language Model (LLM) service platforms that offer businesses Model-as-a-Service (MaaS) solutions. The US team drives the design, development, and operation of MaaS solutions across the US and international markets outside mainland China. This role involves building full-stack, end-to-end solutions spanning text and multimodal LLM algorithms, LLM training/fine-tuning/inference frameworks, prompt engineering, model alignment, and intelligent agent systems. The team also operates large-scale log analytics pipelines for continuous, data-driven feedback loops to inform model improvement, system optimization, and product decisions.
As a graduate, you will have opportunities to pursue bold ideas, tackle complex challenges, and unlock limitless growth. Successful candidates must commit to an onboarding date by the end of the year.
Key Responsibilities
- Design evaluation systems for LLM-based agents, covering task success, tool use, reasoning quality, and reliability.
- Build benchmarks and automated judging pipelines, combining rule-based checks, model-based judging, and human review.
- Analyze agent execution traces and user feedback to identify failure patterns and translate them into concrete system improvements.
- Support the closed loop from experience to capability and collaborate with research, platform, and product teams to bring methods into production.
Requirements & Tech Stack
- Python skills and experience with a mainstream ML or agent evaluation framework.
- Hands-on experience with LLM-based systems (e.g., agents, tool calling, retrieval, multi-agent systems) through research, internships, or projects.
- Solid foundation in machine learning and deep learning.
Qualifications
Minimum Qualifications
- Individuals completing or recently completed a Bachelor's/Master's degree in Computer Science, Artificial Intelligence, Machine Learning, Data Science, or a related field.
- Demonstrated research or engineering ability through publications, substantial projects, internships, or open-source work.
Preferred Qualifications
- Publications at top-tier ML/NLP venues (e.g., NeurIPS, ICML, ICLR, ACL etc.), especially in agent learning, self-improving/self-evolving/RSI, or agent evaluation.
- Experience with evaluation methodology: metric design, model-based judging, or annotation and statistical analysis.
- Familiarity with LLM post-training, reasoning and planning methods, or continual learning.
- Experience with feedback-driven optimization loops, or with large-scale log and trace analysis.
Job Information
- Onboarding Date: By the end of the year.
- Application Limit: Maximum of two positions globally across ByteDance and its affiliates. Applications are reviewed on a rolling basis.