Site Reliability Engineer

About the Role

SpaceX HPC is a shared compute platform used across the company for simulation, machine learning, AI inference, and more. This role is to implement a Site Reliability Engineer operating model for these capabilities, focusing on toil reduction, automation, observability, and a sustainable incident process. You will own the entire ecosystem, from Linux machines and Infrastructure as Code to storage and user-facing applications.

Key Responsibilities

• Participate in the team's on-call rotation; practice sustainable incident response and blameless postmortems.
• Manage node lifecycle with infrastructure as code: OS images, firmware, configuration management, kernel and driver stack.
• Build observability for both HPC administrators and end users: cluster, node, and storage health for operators, and job/workflow-level signal for users.
• Reduce toil with automation; split time between operating production systems and writing software to streamline operations.
• Sustainably manage resources, including compute and storage.
• Lead capacity planning with users across the company to understand future needs and project tomorrow's compute and storage requirements.
• Collaborate with HPC systems engineers and engineers across all disciplines on operable, maintainable infrastructure.

Requirements & Tech Stack

• Linux operating systems in production
• Operating production infrastructure (servers, services, or networks), including monitoring, debugging, and repairing.
• Monitoring and alerting (Prometheus, Grafana, Nagios, or similar)
• Configuration management or infrastructure as code (Ansible, Puppet, Terraform, or similar)
• Writing scripts/code (e.g., Python or similar languages) to automate common tasks
• Containers (Docker, Podman, Singularity/Apptainer)
• Kubernetes administration for on-premise deployment
• Distributed or high-performance storage (VAST or similar), including capacity, performance, and lifecycle management
• Version control, testing, continuous integration, build, deployment, and monitoring
• Ability to communicate clearly with users, peers, and vendors in incident and design settings.
• Comfortable working with mission-critical and sensitive systems, with a sense of urgency.
• Must be able to participate in an on-call rotation.
• Must be willing to work extended hours and weekends as needed.
• Position is based in Hawthorne, CA and is primarily on-site.

Qualifications

• Bachelor's degree in computer science, engineering, math, or a scientific discipline; OR 2+ years of professional experience operating production infrastructure in lieu of a degree.
• 2+ years of experience with Linux operating systems in production.
• 2+ years of experience operating production infrastructure (servers, services, or networks), including monitoring, debugging, and repairing what you own.
• Professional experience in SRE, DevOps, or production infrastructure engineering is preferred.
• Familiarity with HPC clusters, schedulers (Slurm, PBS, LSF), or GPU compute is not required.
• Familiarity with scientific computing (CFD, FEA) and/or ML training workloads (PyTorch, TensorFlow, CUDA) and/or AI inference workloads is preferred.
• Eligibility for access to classified material up to TS/SCI with polygraph.