Autoscaling works on a simple promise. Traffic goes up, you add a replica, and capacity goes up. For a web service, that takes seconds. For a large language model server, the new pod can sit for minutes loading weights and warming up while the queue grows, and it adds nothing until it is ready.
A CNCF post from 6 November walks through where that time goes and what can be done about it today. It is the clearest summary of the topic so far.
Why do inference servers start so slowly?
Starting a model server is a long chain of work: launching processes, importing large libraries, reading and transforming model files, setting up communication between GPUs, allocating memory, compiling kernels, autotuning and capturing CUDA graphs. Some of these steps exist to make every later request faster, so they trade startup time for speed in steady state.
The slow step also moves. Once the model files are cached on the node, engine setup becomes the bottleneck instead of the download.
What can you do about it today?
The post groups the options by what work they save:
Cache model files and load them faster. Keep weights on local or nearby storage and use efficient loaders, such as llm-d's model caches or fastsafetensors.
Persist compile and autotune caches so a new replica does not redo work an earlier one already did. vLLM supports this.
Hold weights in a separate process. vLLM has merged a weight daemon that keeps weights in GPU memory and shares them over CUDA IPC. SGLang has merged a weight cache daemon.
Keep warm instances asleep. vLLM's sleep mode, SGLang's memory saver and llm-d's Fast Model Actuation let you park an instance and wake it, rather than start from scratch.
Snapshot the initialised engine. Experimental work in vLLM, SGLang and llm-d on GKE uses CRIU and CUDA checkpointing to restore a ready engine.
None of these removes all startup cost, because each one saves a different part of the chain.
How mature are snapshots?
Not very, and the post is careful to say so. vLLM's engine snapshots are documented for one NVIDIA GPU, tensor parallelism of one, on the same machine and environment. Its example was validated on v0.27.1 and does not work on v0.28. NVIDIA Dynamo's v1.4.0 docs mark its GPU memory and snapshot path as unsupported for production.
In Kubernetes itself, container checkpointing reached beta in v1.30. Pod-level checkpoint and restore is targeting alpha in v1.38, without GPU state or cross-node restore at first. GPU support is still a provisional proposal. The CRI API for pod checkpoint and restore merged in July, but the containerd and CRI-O implementations are not finished.
What should platform teams do now?
Measure the startup chain. Time image pull, weight load, engine init and warm-up separately. You cannot fix the slow step until you know which it is.
Fix storage first. Node-local model caches and pre-pulled images are the cheapest wins and work everywhere.
Persist compile caches on a volume shared by replicas of the same model and version.
Scale earlier, not faster. Trigger on queue depth or tokens per second with headroom, because a replica that takes minutes to start needs to be asked for minutes earlier.
Keep a warm floor. For latency-sensitive services, a small pool of sleeping or idle replicas is often cheaper than missed requests.
Treat snapshots as a pilot. They are promising, but the hardware, driver and version limits are real.
Quick answers
Why are LLM inference cold starts slow?
A new inference replica must load model weights, initialise GPUs, compile kernels, capture execution graphs and warm up before it can serve. Each step adds time, and the replica adds no capacity until all of them finish.
How can I reduce vLLM startup time?
Common approaches are caching model files locally, persisting compile and autotune caches, using vLLM's weight daemon or sleep mode, and, experimentally, restoring engine snapshots on a single GPU.
Does Kubernetes support checkpointing GPU pods?
Not yet. Container checkpointing is beta since v1.30, pod-level checkpoint and restore targets alpha in v1.38 without device state, and GPU support is still a provisional proposal.







