Large Language Model Training System Engineer Graduate

About the Role

• This is a graduate position with an anticipated start date of 2027.
• Successful candidates must commit to an onboarding date by the end of the year.
• Candidates can apply to a maximum of two positions globally across ByteDance and its affiliates.
• Applications are reviewed on a rolling basis; early application is encouraged.

Key Responsibilities

• Participate in the development of the Volcano Ark training system, enabling internal and external users to perform large model post-training (SFT and RL) on the Volcano Ark platform in a serverless manner.
• Design elastic training solutions for complex multi-tenant training scenarios, supporting mixed multi-tenant training across multiple data centers and heterogeneous hardware, while improving training throughput and stability.
• Design reinforcement learning systems to improve training efficiency and provide user-friendly reinforcement learning training interfaces.

Requirements & Tech Stack

• Proficient in one or more programming languages such as Python, Rust, or C++.
• Ability to write clean code and possess strong framework design and abstraction skills.
• Experience in training framework design or training system optimization is preferred.
• Participation in large model training engineering or complex distributed system development is preferred.
• Strong interest in tracking and solving technical problems, studying low-level principles and performance bottlenecks, and applying scientific modeling methods.
• Solid experience in computer architecture optimization, with familiarity with heterogeneous hardware and high-performance networking is preferred.
• Strong scientific optimization skills, including the ability to independently analyze training efficiency and identify optimization directions is preferred.
• Familiarity with mainstream reinforcement learning frameworks and technical insight into training frameworks is preferred.

Qualifications

• Individuals who are completing or have recently completed a Bachelor's or Master's degree in Computer Science or a related discipline.

Job Information

• Base Salary Range: $128,000 - $256,000 annually.
• Compensation may vary based on qualifications, skills, competencies, experience, and location.
• Potential eligibility for additional discretionary bonuses/incentives and restricted stock units.
• Benefits: Day one access to medical, dental, and vision insurance, 401(k) savings plan with company match, paid parental leave, short-term and long-term disability coverage, life insurance, and wellbeing benefits.
• Paid Time Off: 10 paid holidays, 10 paid sick days, and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure).