At the end of September, Ahmed Tariq wrote a short piece about a background worker that kept getting killed for no good reason. It was a Node.js process pulling jobs from Redis. It had no web endpoint, so someone gave it a liveness check that ran pgrep to confirm the worker process existed.
The team then moved to a slimmer runtime image. That image did not include pgrep. Kubernetes could not run the check, counted every attempt as a failure, and restarted a worker that had been happily processing jobs. It ended up in CrashLoopBackOff.
The article names no company and reads as a lesson from experience, not a formal post-mortem. Its value is that most platform teams will recognise it, and it is not new. In June 2021, an issue on CERN's GitLab for the EOS Helm charts reported that the slim EOS image lacked pidof, the very command the chart's liveness probe relied on. The fix was to add the package back.
How a missing tool becomes an outage
A liveness probe answers one question for the kubelet (the agent on each node that runs your containers): should I kill this container and start it again? An exec probe runs a command inside the container and treats exit code zero as healthy. Anything else is a failure, and a command that does not exist is about as clear a failure as you can get.
With Kubernetes defaults, a probe runs every 10 seconds and the container is restarted after three failures in a row. So a container with a broken liveness command lives for roughly 30 seconds before it is killed.
Then the restart backoff kicks in. Kubernetes waits 10 seconds before the next restart, then 20, then 40, doubling up to a cap of five minutes. The backoff only resets after a container has run cleanly for 10 minutes, which a broken probe never allows.
Here is the illustrative arithmetic for one pod on default settings. Within about eight minutes the backoff hits its cap. From then on, each cycle is about 30 seconds of life followed by five minutes of waiting, so the pod is up for roughly one minute in every eleven. Every replica runs the same image, so your service has lost about 90 percent of its capacity.
Nothing in the application changed. The image was smaller and safer, exactly as the security team asked.
Why image slimming walks straight into this
Distroless images, as the project describes them, hold your application and its runtime dependencies, with no shell, package manager or usual Linux utilities. Slim variants trim less, but still drop things people assume are always there.
The trap is that probes are configuration, not code. They live in a Helm chart or a manifest, often written years ago by someone else, and nobody connects them to the Dockerfile. The person who changes the base image is not looking at the probe, and the person who wrote the probe is not reviewing the image change.
The bigger problem: liveness checks that test the wrong thing
The missing binary is the easy case because it fails every time. The harder cases fail only under stress, turning a small problem into a large one.
The Kubernetes documentation is unusually blunt here. It warns that badly implemented liveness probes can cause cascading failures: containers restarting under high load, client requests failing as capacity drops, and extra pressure on the pods that are still standing. It says a liveness probe should indicate a failure the container genuinely cannot recover from, such as a deadlock.
The most common mistake is a liveness probe that checks a database or another service. When that dependency blips, every pod fails its liveness check at the same moment and Kubernetes restarts the whole fleet. Henning Jacobs made this argument in his 2019 post on liveness probes, and the official guidance agrees: if your app depends on a backend, the readiness probe can check it, while liveness should only ask whether the app itself is alive.
Readiness checks on dependencies still need care. Expo's public post-mortem of a November 2019 incident describes an engineer switching off a legacy database that sidecar routers in the API pods were still connected to. Those routers failed their readiness checks, so when European morning traffic arrived and Kubernetes started more pods, none of them ever became ready. Availability collapsed for about half an hour.
Timeouts are the quieter version. Basedash wrote earlier this year about Node.js web pods restarting nine to eleven times a day with no visible user impact. The liveness endpoint shared the event loop with everything else, garbage collection pauses of up to about 8.9 seconds blew through the default one-second timeout, and the kubelet killed the pods. Raising the liveness timeout and failure threshold stopped the restarts.
The honest caveat
None of these is a headline outage at a household name, and the missing-binary case has no company attached.
Your exposure also depends on your setup. If most services already use HTTP or gRPC probes, image slimming will not bite you this way. If a container reliably crashes on its own when it is unhealthy, the documentation notes you may not need a liveness probe at all. The risk is in the gap between who owns the probe and who owns the image.
What to do this week and this quarter
This week, ask your platform team for a count of exec liveness probes across your clusters, and for each one, whether the command exists in the image that is actually deployed. You may find one that only works by accident.
Also ask how many liveness probes call a database, cache or another service. Each is a fleet-wide restart waiting for a dependency to have a bad minute. Move those checks to readiness.
This quarter, put probe testing into CI against the final built image, not a development image with extra tools. Start the container, run the exact probe command or request, and fail the build if it does not pass. That one step would have caught both the pgrep and pidof cases before they shipped.
Then make sensible probes part of your golden path, the paved road your platform team offers for new services. The template should default to HTTP or gRPC probes (gRPC probes have been stable since Kubernetes 1.27), with liveness checking only the process and readiness checking dependencies. It should include a startup probe for slow-starting services, so nobody stretches liveness timings to cover boot time. And when someone changes a base image, the review checklist should include a look at the probes.
Health checks are meant to make systems heal themselves. Left without an owner, they become one of the most reliable ways to take down a healthy service.
