Startup, Readiness, and Liveness Probes

Every TruePPM component ships with a probe that answers one of three questions, and getting them confused is the single most common cause of a self-hosted install that looks broken but is not (or, worse, looks fine but is not serving):

• Startup — has this component finished booting? While a startup probe is configured and has not yet passed, Kubernetes suspends the liveness probe entirely, so a slow first boot cannot be mistaken for a hang.
• Readiness — can this component serve traffic right now? A readiness failure removes the pod from its Service’s endpoints (or, for a worker behind no Service, just paces a rolling update) — it does not restart anything, and it has no failure budget: one success is all it needs, so a probe that never gets one cannot be fixed by raising a threshold.
• Liveness — is the process itself still alive, as opposed to wedged? A liveness failure kills the container. This is a blunt, expensive instrument — the cost is a full restart plus whatever termination grace the component needs to shut down cleanly — so it should fail rarely and for a good reason.

The trap this page exists to name: an exec probe that does real work runs INSIDE the container it measures. If that work competes for the same CPU or process the container needs to do its actual job, a probe can fail a component that is working correctly, simply because it was too busy to also answer the probe in time. This is exactly what happened to the Celery worker’s readiness probe before 0.4 (#3236): a worker with zero restarts, processing jobs the entire time, that never once answered celery inspect ping under load. Widening the probe’s timeout could not fix it — readiness has no failure budget to widen around. The fix was to stop asking the worker to do expensive work to prove it is ready, not to give it more time to do that work.

At a glance

Component	Startup	Readiness	Liveness
api (Django / ASGI)	off by default; /health/ when enabled	/readyz — DB, cache, and migration state	/health/ — process only
channels / websockets	— (same process as api)	— (same process as api)	— (same process as api)
celery worker	heartbeat file exists	heartbeat file fresh	celery inspect ping
celery beat	—	— (see why beat has no readiness probe)	celery inspect ping (fleet)
valkey / postgres (bundled subcharts)	vendor default	vendor default	vendor default
web (nginx)	— (no boot-time dependency)	GET /asset-manifest.json — 200, build-specific	GET / — 200

Full tunable defaults for every row are in the Health probes section of the Helm values reference. Time-to-Ready and time-to-restart arithmetic at those defaults is in Deployment Sizing → Startup budget. Symptom-first diagnosis (“a pod never becomes ready”, “a container reports unhealthy while the site works fine”) is in Troubleshooting — this page explains the mechanism; that one walks you through a specific failure.

api (Django / ASGI)

Readiness — GET /api/v1/readyz. Deliberately unauthenticated (kubelet carries no credential) and deliberately deep: it round-trips the database with a bounded SELECT 1, round-trips the Valkey/Redis cache with a write-then-read, and checks whether this image’s migrations agree with the database’s recorded schema. A pod only joins the Service once all three answer ok. This is what keeps a rolling upgrade from routing traffic to a pod whose code and schema disagree — a pod mid-migration reports migration_state: behind and stays out of the Service until its own migrate init container finishes.

What to do when it fails: read the response body — checks names which dependency is down (database, cache, or migrations), and migration_state distinguishes a rollout in progress (behind) from a rolled-back image serving against a newer schema (ahead). See A pod never becomes ready, or migrations are pending and Every request returns 503.

Two things changed here in 0.4, both perf/security hardening rather than new behavior: the migration check used to rebuild the full on-disk migration graph — a filesystem scan and a Python import per app — on every single call, which at kubelet’s default 10s readiness period meant a full rebuild every 10 seconds for every Ready pod, forever. That scan is now cached for the life of the process (only the database’s recorded-migrations state, which legitimately changes mid-rollout, is re-read on every call). And the endpoint no longer gets a blanket exemption from the API’s shared rate limit the way /health/ and /edition/ do — unlike those two it does real database and cache work per call, so an unauthenticated caller who reached the pod IP directly (the probe path bypasses the Ingress) could otherwise drive unbounded load with zero rate limit. It now has its own generous, dedicated scope (env.TRUEPPM_THROTTLE_READYZ_RATE, default 2000/min) instead.

Liveness — GET /api/v1/health/. Deliberately shallow: it returns 200 whenever the ASGI process is up, with no dependency check at all. This is what keeps a transient database or cache blip from restart-looping a pod that readiness has already, correctly, pulled out of the Service — restarting the process would not fix a database outage, and would cost the boot time back on top of it.

Startup — off by default, using the same /health/ path. Turning it on (probes.api.startupEnabled: true) moves the “how long can a first boot take” question from liveness’s own initial-delay/threshold math to an explicit budget, and suspends liveness until it passes — worthwhile on a loaded cluster where a cold pod’s first boot is slower than a warm restart. See Startup probe for the numbers.

channels / websockets

There is no separate probe for real-time collaboration. WebSocket upgrades are served by the same ASGI process and the same pods as the HTTP API — Django Channels runs in-process, not as a separate Deployment — so /readyz and /health/ already answer “can this pod accept a WebSocket connection?” as a side effect of answering “can this pod accept a request at all?”. A pod that is Ready is accepting WS upgrades; a pod that is not is not.

celery worker

The worker has three probes with three different jobs, and, from 0.4, two different mechanisms:

Liveness — celery inspect ping against this pod’s own worker node. A control-plane round trip over the broker, so it detects a wedged event loop that a bare process-alive check would miss. This is the one probe still shaped this way, deliberately: a missed liveness check only restarts the container, a cost the worker’s 300s termination grace already accounts for (failureThreshold × periodSeconds is derived to match it exactly — see Tuning celery probes apart). A restart is a bounded, recoverable cost in a way a permanently-false readiness signal is not.

Readiness — a heartbeat file’s freshness, not celery inspect ping (#3346). A Celery signal handler (trueppm_api.core.worker_heartbeat) touches a file (probes.worker.heartbeatFile, default /tmp/trueppm-celery-worker-heartbeat) every time Celery’s own internal heartbeat_sent signal fires — a fixed ~2s timer that runs on the worker’s MainProcess event loop, entirely independent of whether the prefork pool is idle or fully saturated. The probe itself is a plain find <file> -newermt "-30 seconds" — a filesystem stat, not a broker round trip, not a Django import. This is what makes it safe to run far more often (every 15s, versus the pre-0.4 60s) and, more importantly, what makes it correct under load: its cost does not scale with how busy the worker is, so a worker that is fully occupied processing a queue keeps answering readiness exactly as fast as an idle one does.

Startup — the same heartbeat file’s mere existence. The file is first touched when Celery fires worker_ready — which only happens once the worker’s initial broker handshake (mingle/gossip) completes — so “the file exists at all” is precisely “this worker has established a broker connection at least once.” Until it does, liveness stays suspended, so a slow first connection cannot restart-loop the pod.

Before any of that, the worker waits for its broker (#3722). On a fresh install the worker and Valkey start at the same time. Every recorded case of a worker that connected, logged ready, and then served nothing (heartbeat file frozen at the ready timestamp, no inspect ping reply) began with the worker hitting refused broker connections during that race. The exact mechanism is not yet confirmed, so treat this as a mitigation rather than a proven fix. The worker now pings the broker before it starts consuming, retrying every 2s. If the broker still isn’t reachable after TRUEPPM_CELERY_BROKER_WAIT_SECONDS (default 120, inside the startup probe’s 150s budget), the worker exits non-zero so the container restarts cleanly. Set it to 0 to turn the wait off. Log lines prefixed worker_broker_wait: show whether the wait engaged. This covers only startup: a broker that restarts under a running worker is still recovered by the liveness probe.

What to do when readiness or startup fails: check whether the worker process is even running (kubectl logs -c celery-worker, or docker compose logs celery) before assuming a probe bug — a worker that has genuinely crashed or never connected to the broker will never touch the heartbeat file, and that is the probe doing its job. If the worker’s logs show it healthy and processing tasks but the probe still fails, check the clock and the file directly: kubectl exec ... -- stat /tmp/trueppm-celery-worker-heartbeat (or the Compose equivalent) — a stale file with a healthy-looking log usually means the worker’s --concurrency is pinned so low, or the node so starved, that even the MainProcess’s own timer thread is not getting scheduled; see Deployment Sizing for concurrency guidance. See also Celery is not processing anything for the broader diagnosis flow, and #3236 for the original incident this mechanism was built to fix. That issue’s underlying root cause — why the pre-0.4 inspect ping occasionally never got a reply from a genuinely healthy worker — was never conclusively identified; the fix taken here was to remove the class of probe that could fail that way, not to find the specific reason it did.

celery beat

Liveness — celery inspect ping against the whole fleet, not a specific node — beat runs no worker control plane of its own, so this asserts “beat can reach the broker, and at least one worker answers.” A generous threshold keeps a brief worker restart from taking beat down with it.

No readiness probe, on purpose. Beat is a pinned singleton behind no Service (replicas: 1, strategy: Recreate) — a readiness probe would gate nothing beyond rolling-update pacing that a singleton does not do, while still costing a Django import in the chart’s tightest resource budget (250m CPU / 256Mi). If beat’s readiness question — has it loaded its schedule? — matters to you operationally, the signal exists: beat writes its persistent schedule shelve (--schedule=/tmp/celerybeat-schedule) on boot, before its first tick, so the file’s existence is “schedule loaded.” The Docker Compose stacks check this directly, since Compose has no Service for a Kubernetes-style readiness probe to gate in the first place — see the table below.

For the other half of “is beat alive” — is it still ticking, days into an install, as opposed to having merely booted once — see Beat Liveness and its GET /api/v1/health/beat/ endpoint, which is a durable, database-backed detector rather than a container-level probe. It is what survives beat and every worker dying at once; the liveness probe above cannot.

valkey / postgres (bundled subcharts)

The bundled PostgreSQL and Valkey subcharts ship their vendor-standard probes unmodified (pg_isready, valkey-cli ping). TruePPM does not maintain or override them. If you run either as a managed external service instead of the bundled subchart (see Managed datastores), its health is your managed provider’s responsibility, and TruePPM only ever observes it indirectly, through /readyz’s database/cache checks above.

web (nginx)

Readiness — GET /asset-manifest.json, not /. The web tier serves a single-page app, and its catch-all location / answers any unmatched path with index.html as long as nginx is up at all — so a probe against / only ever proved “nginx is running,” never that this pod’s html root actually holds this release’s build (#4359). It is not a proxy check: a proxy_pass upstream name that does not resolve already fails nginx’s own config load at startup, crash-looping the pod before any probe — old or new — could reach it either way. asset-manifest.json is Vite’s build manifest, written at the html root only by the image’s own build — the image’s prior-release-assets mechanism (#4341, which keeps a previous release’s hashed chunks around for a tab that has not yet reloaded) only ever copies forward into assets/, never to the html root, so this file can never be a carried-over prior release’s. The chart’s web ConfigMap gives it its own location = /asset-manifest.json { try_files $uri =404; }, so a missing manifest 404s the probe instead of silently falling through to the SPA shell. What this does not prove: the manifest’s mere presence says nothing about whether the individual JS chunks it names are actually on disk and intact — a corrupt or partially missing assets/ directory still passes this probe (see scripts/check-served-assets.sh, a separate, heavier check run outside the probe path). The issue that raised this (#4359) also considered probing the SPA’s entry chunk directly; the build manifest was used instead because it is a stable, well-known path that does not change with every build’s content hash.

Liveness — still GET /. Unlike readiness, liveness exists to catch a wedged process, and a missing or corrupted bundle is a static condition that a container restart cannot fix — pointing liveness at the same build-specific check would crash-loop a pod with a genuinely broken image forever, instead of leaving it NotReady, which is the correct terminal state readiness already produces for that failure.

Both probes are answered entirely by nginx serving static files off disk — there is no backend dependency, which is also why there is no startup probe: the container is either serving within a second of starting or its process failed to start at all, and there is no meaningful boot window in between for a startup probe to cover. Timing (5s initial delay for readiness so a rollout is not held up; 10s for liveness, checked less often) was previously hardcoded in the chart; from 0.4 it is tunable via probes.web.* — see the values reference. The path itself is not a values knob — see #4359.

Docker Compose equivalents

Docker Compose has one healthcheck per service, not three, so each maps to whichever Kubernetes probe answers the question a Compose depends_on: condition: service_healthy actually needs — almost always readiness, since that is what other services should wait on before starting.

Service	Compose healthcheck	Mirrors
api	GET /api/v1/readyz	api readiness
celery (worker)	heartbeat file freshness (find ... -newermt)	worker readiness
celery-beat	test -e /tmp/celerybeat-schedule	”schedule loaded” (beat has no Kubernetes readiness probe to mirror — see above)
db / valkey	vendor healthchecks (pg_isready, valkey-cli ping)	vendor probes
nginx (production stack only)	GET /api/v1/health/ through the reverse proxy	proves the front door — TLS termination plus both upstreams — rather than either component’s own probe alone

api’s Compose healthcheck changed from /health/ to /readyz in 0.4 for the same reason the two are different Kubernetes probes: depends_on: condition: service_healthy is a readiness question (“can dependents start using this now?”), and a shallow liveness check answers the wrong one — a healthy-looking api container whose database was actually unreachable would previously report (healthy) to Compose while every real request 503’d.

The worker row has no liveness column above because Compose itself has no liveness mechanism — not because the worker has no liveness concern (#3936). restart: unless-stopped only fires on a process exit; a healthcheck reporting unhealthy changes what docker compose ps prints and nothing else, unlike a Kubernetes livenessProbe, which kubelet acts on directly. A worker whose heartbeat goes stale therefore has no path back on Compose from the healthcheck alone. Since 0.4, TRUEPPM_CELERY_WORKER_SELF_HEAL=1 (set by both docker-compose.yml and docker-compose.prod.yml) closes that gap from inside the process instead: a background thread (trueppm_api.core.worker_selfheal) reads the same heartbeat file the readiness healthcheck does and calls os._exit once it has been stale for 90 seconds — 3x the readiness check’s own 30-second window, so a transient broker blip or a scheduling pause cannot turn into a crash loop. That exit is what restart: unless-stopped needed all along. See A Celery worker wedges and never recovers for the operator-facing version of this, including the manual docker compose restart celery remedy for a deployment that disables the env var.