A training job that runs for weeks across hundreds of GPUs is only as healthy as its weakest machine. One failing GPU, one network adapter running below speed, or one slow node can stall or slow the whole job. Netflix has written up how it deals with that, and its answer is to stop treating GPU failures as one-off hardware problems and start treating them as a normal part of running a distributed system.
According to InfoQ's write-up of the Netflix engineering post, published on 30 October, the approach lifted training job success from about 98% to more than 99.5%, with GPU node availability above 99.95%. Netflix runs this on Kubernetes.
Why are GPU failures different from normal node failures?
A normal health check asks whether a machine is up. That is not enough for GPUs. A GPU can answer status queries and still fail when a job tries to allocate memory or run sustained work. A network adapter can be present but show fewer links than expected, which quietly cuts the bandwidth that collective operations depend on.
The cost of these partial failures is high because training runs in lockstep. Netflix's post points out that a GPU does not have to fail to hurt a job. A single slow rank holds every other GPU back. Meta reported that hardware failures caused more than 66% of training interruptions during Llama 3 training, which shows this is not a Netflix-only problem.
How does Netflix detect and fix bad GPU nodes?
Layered checks. NVIDIA DCGM watches GPU errors, memory, interconnect and temperature. Separate checks confirm the state and count of the Elastic Fabric Adapter interfaces. A node agent turns the results into Kubernetes node conditions.
Fail closed. If a diagnostic hangs, it times out and the machine is marked unhealthy. If the GPU diagnostic tool itself breaks, kernel logs are used as a fallback.
Step-by-step repair. A short-lived problem stops new work landing on the node while existing jobs carry on. A persistent problem drains the node and moves workloads. Responses run from killing a job or resetting a GPU up to rebooting or replacing the machine.
Limits on disruption. Netflix uses Karpenter's node auto-repair, with its own rules for how long each type of failure is tolerated and how much of the cluster can be taken out at once.
On the job side, checkpointing and elastic workers let training resume from the last good checkpoint after a machine is lost. Infrastructure failures are retried automatically, while application errors are classified separately so a bug in the code does not trigger an endless restart loop.
For inference, Netflix uses NVIDIA Multi-Instance GPU to let smaller workloads share a GPU with memory isolation between them. Next on its list is Kubernetes Dynamic Resource Allocation, for richer requests and placement that understands how GPUs, network adapters and CPUs are connected.
Which parts can a smaller team copy?
You do not need hundreds of GPUs to get value from this. Three parts transfer well to a cluster with a few dozen.
Check GPUs under load, not just presence. Run DCGM diagnostics when a node joins and on a schedule, publish the result as a node condition, and keep jobs off nodes that fail. Make the check fail closed, so a hung test counts as a failure.
Separate "stop scheduling" from "drain". Most teams have one response to a bad node: remove it. Adding a middle step, where a node stops taking new work but finishes what it has, avoids killing long jobs for problems that turn out to be temporary.
Make jobs survive losing a node. Checkpoint often enough that losing a node costs minutes, not days, and tell your retry logic the difference between a hardware failure and a code failure.
Then measure what you actually care about. Node uptime looks good on a dashboard. The useful number is goodput: how much of the GPU time you pay for turns into finished training progress.
Quick answers
How does Netflix improve GPU reliability?
Netflix runs layered GPU and network health checks with NVIDIA DCGM, turns the results into Kubernetes node conditions, repairs nodes step by step with Karpenter, and makes training jobs resume from checkpoints after machine loss. It reports training job success above 99.5% and GPU node availability above 99.95%.
Why is a basic node health check not enough for GPUs?
A GPU can respond to status queries and still fail under memory allocation or sustained load, and a network adapter can run with fewer links than expected. These partial failures slow or stall lockstep training jobs without making the node look down.
What is goodput in GPU training?
Goodput is the share of paid GPU time that becomes useful training progress, after subtracting time lost to failures, restarts, slow nodes and recomputation from old checkpoints.
