Research Intern, Frontier Speculative Decoding

About the Role

The Speculative Decoding team at Together AI builds draft models and algorithms for faster large language model generation without altering outputs. Our work spans model training, inference algorithms, and serving systems, aiming to reduce generation cost and latency.

As a Research Intern, you will explore open questions in speculative decoding, including:

• New drafter architectures and verification schemes.
• Knowledge distillation and training recipes for aligning draft and target models.
• Drafters that adapt to changing traffic patterns.

You will train and fine-tune draft models at scale on dedicated GPU clusters, and test ideas against realistic serving workloads, such as long-context and agentic traffic. Projects will focus on enabling faster generation on frontier models, developing adaptable drafters, and robust evaluation under real traffic.

Key Responsibilities

• Research and prototype new efficient speculative decoding algorithms and draft model architectures.
• Train, fine-tune, and distill draft models, and design rigorous experiments to validate hypotheses.
• Implement optimized versions for serving engines to demonstrate efficacy in real-world workloads.
• Communicate project plans, progress, and results to the team.
• Share findings through technical write-ups and blog posts, and contribute to bringing successful ideas into production.

Requirements & Tech Stack

• Currently pursuing a Master's or Ph.D. degree in Computer Science, Electrical Engineering, or a related field.
• Strong knowledge of Machine Learning and Deep Learning fundamentals.
• Hands-on experience training or fine-tuning large models with distributed training frameworks (e.g., Megatron-LM, FSDP).
• Familiarity with modern model architectures, such as Mixture-of-Experts and hybrid attention.

Qualifications

• Prior research experience in speculative decoding, efficient inference, or efficient machine learning.
• Publications at leading conferences in machine learning or systems (e.g., NeurIPS, ICML, ICLR, MLSys).
• Experience with knowledge distillation, post-training, or reinforcement learning.
• Familiarity with LLM serving engines (e.g., vLLM, SGLang, TensorRT-LLM).
• Contributions to open-source machine learning projects.