Troubleshooting

Find your symptom, work the causes in the order listed — they are ordered by how often they are the answer, not by how interesting they are — and confirm the fix with the check at the end of each section.

Two commands are worth running before anything else, because roughly half of all incidents are answered by them.

Docker Compose

docker compose ps
# STATUS column: every service should read "Up" and the ones with healthchecks
# (db, valkey, api) should read "(healthy)". "(unhealthy)" or "Restarting" names
# your culprit; "Exit 1" means read that service's logs first.

Helm / Kubernetes

kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<release> -o wide
# READY 1/1 and STATUS Running on every pod. Note the NODE column — it is the
# answer to "why did two replicas die together?"

Throughout this page, <release> is your Helm release name and <ns> your namespace. The application Deployments are <release>-trueppm-api, -web, -celery-worker, and -celery-beat; if your release name already contains “trueppm” the chart drops the duplicate, so a release called trueppm yields plain trueppm-api. The bundled datastores are named by their own subcharts and do not carry the trueppm segment: <release>-postgresql-0 and <release>-valkey-primary-0. Confirm with kubectl get deploy,sts -n <ns>.

---

The browser shows a blank page

What you see. The page loads, the tab title is right, and the viewport is white. No error text. The browser console usually shows a failed request for a .js bundle, or a Failed to fetch against /api/v1/….

Likely causes, in order.

1. The web tier is not serving the bundle. On Compose the web container is still building (first run compiles the SPA — see below); on Helm the -web Deployment has zero ready pods.

2. The API is unreachable from the browser, so the SPA boots and then fails its first data fetch. This is the same underlying condition as a 502, one layer up.

3. A CSP or MIME-type mismatch in front of the SPA — usually an ingress, WAF, or CDN adding its own Content-Security-Policy on top of the one the web tier already sets, so the bundle is blocked. The console names the directive.

4. The page and its assets came from two different releases. The console says Expected a JavaScript-or-Wasm module script but the server responded with a MIME type of "text/html" or Refused to apply style … MIME type ('text/html'). Each release’s index.html names its own content-hashed /assets/* files. A replica running the other release does not have those files, so its SPA fallback answers with index.html and a 200 status, and the browser refuses to run it. Releases after 0.4.0-beta.7 answer a missing /assets/ file with a real 404 instead, and the app reloads itself once when a chunk fails to load (a second failure within 10 seconds, an offline browser, or unsynced changes leave the on-screen Reload button to the user). Tags up to and including 0.4.0-beta.7 still return the 200 SPA fallback. Releases after 0.4.0-beta.7 also keep the previous release’s /assets/ files in the web image, so a tab opened before an upgrade keeps loading its own chunks from the new pods. That reaches back one release: the immediately previous published release, not counting a withdrawn one. 0.4.0-beta.7 was withdrawn, so the first image built this way carries 0.4.0-beta.6’s files. A tab opened before an upgrade that skips a release falls back to the one automatic reload. During the rollout itself, a client that got the new index.html can still ask an old pod for a new chunk; the app retries that chunk twice, over about two seconds, which normally reaches a new pod, and only then reloads. When the chunk that fails is the app’s own entry script, or one it needs before it can start, the page reloads itself after 1, 3, 8 and 15 seconds. If the page still has not started after the fourth reload, it shows TruePPM is being updated with a Reload button rather than staying blank. Neither one retries while the browser is offline; an offline browser sees You appear to be offline with the same button. If you see that message outside an upgrade, the web tier is missing files that its own index.html names. Check that every web pod runs the same image. A reload fetches the current index.html, but it can still land on the other release until the rollout finishes. Load-balancer luck decides which files fail, so the failing set changes from one reload to the next. Two situations produce it:

   • A rolling upgrade to 0.4.0-beta.7 or earlier with two or more web replicas. The chart’s production values run two, and the default rolling update keeps old and new pods serving together until it finishes. A healthy rollout produces this too, not only a stalled one. Clients that load the page during that window can blank-screen until they reload after the rollout completes.
   • An upgrade that skips a release, on any number of replicas: tabs opened before it ask for files from a release two or more back, which the new image does not keep.
   • A rollback, or a browser holding an older page. A client that fetched index.html from the new release just before a rollback asks for assets that no remaining pod has. A reload fixes it.

   Finish the rollout or the rollback, confirm that every web pod runs the same image, then reload. The previous-release assets and the chunk retry are #4341.

Commands.

# Compose (dev) — is the web service up, and did the Vite build finish?
docker compose ps web
docker compose logs --tail=50 web

# Compose (prod) — the SPA is built once by `web-build` and served by `nginx`
docker compose -f docker-compose.prod.yml logs --tail=50 web-build nginx

# Helm — is anything behind the web Service?
kubectl get deploy,endpoints -n <ns> -l app.kubernetes.io/instance=<release>
kubectl logs -n <ns> deploy/<release>-trueppm-web -c web --tail=50

Then check what the browser is actually being served:

text/html

curl -sI https://trueppm.example.com/ | head -5
# Helm — do all web pods run the same image? More than one line means two
# releases are serving together (cause 4).
kubectl get pods -n <ns> -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=web \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}' | sort | uniq -c

From a clone of the repository, scripts/check-served-assets.sh fetches every script, module preload and stylesheet that index.html references, plus every chunk and asset named in the build’s asset-manifest.json — including a route loaded behind a lazy import() that index.html never mentions. It reports each file that does not come back as a non-empty 2xx with the expected content type. To catch the mixed-release case, repeat the check a few times, because one pass can land every request on the same replica:

SERVED_ASSETS_ROUNDS=5 sh scripts/check-served-assets.sh https://trueppm.example.com

Run that after a rollout has finished. While one is still in progress, a round that gets the new index.html and asks a pod still on the old release for one of its files gets a 404, which the app recovers from on its own, so a plain run fails on it. Add SERVED_ASSETS_ROLLOUT=1 to report those as SKEW and fail only on the problems a rollout should never show: a file answered with the index.html page, a wrong content type, or a previous-release file that is not served.

Confirm the fix. Reload with the console open. GET / returns 200 text/html, every /assets/*.js returns 200 application/javascript, and the first /api/v1/ call returns 200 or 401 — not 502 or 0.

First docker compose up on a clean clone

The development Compose stack builds the API and web images from source and runs npm install in the web container. First start is a multi-minute build, not 15 seconds, and the web container serves nothing until Vite finishes. Watch it with docker compose logs -f web and wait for the ready in … line before deciding anything is broken.

---

nginx returns 502 Bad Gateway

What you see. 502 Bad Gateway from nginx (Compose-prod) or from the ingress controller (Helm), on /api/… and often on / too.

Likely causes, in order.

1. The API process is not listening. It crashed on a boot guard — settings.prod refuses to start without SECRET_KEY, ALLOWED_HOSTS, INTEGRATION_ENCRYPTION_KEY, and a storage choice — and the container is in a restart loop.
2. The API pods are all NotReady, so the Service has no endpoints and the ingress has nowhere to route. This is almost always a datastore outage: see the 503 case.
3. The API is up but slow past the proxy timeout — a Monte Carlo run or a whole-project schedule load on a large project. Check the tested envelope.

Commands.

# Compose — did the API die on a boot guard?
docker compose logs --tail=100 api | grep -Ei "improperlyconfigured|refus|traceback"
docker compose ps api

# Helm — restart count and the reason for the last exit
kubectl get pods -n <ns> -l app.kubernetes.io/component=api
kubectl describe pod -n <ns> <api-pod> | sed -n '/Last State/,/Ready/p'
kubectl logs -n <ns> <api-pod> -c api --previous --tail=100

# Is anything behind the Service at all?
kubectl get endpoints -n <ns> <release>-trueppm-api

An ENDPOINTS column reading <none> is the whole diagnosis: nothing is ready, and the 502 is the proxy telling you so.

Confirm the fix. kubectl get endpoints lists at least one pod IP, and:

curl -s -o /dev/null -w '%{http_code}\n' https://trueppm.example.com/api/v1/health/
# → 200

---

Every request returns 503 and no API pod is ready

What you see. The SPA loads, every API call returns 503, and kubectl get pods shows READY 0/1 on every API pod while the containers are Running and not restarting.

Likely causes, in order.

1. Valkey is unreachable. /api/v1/readyz performs a live set-then-get round-trip against the cache, so a broker outage marks every API pod NotReady and empties the Service. This is by far the most common cause and it surprises people, because the database is fine. See Durability & Redundancy.
2. PostgreSQL is unreachable, which fails the same probe on the database check.
3. Migrations do not match the image — see the migrations case below.

Commands. Ask the probe which of the three it is. The endpoint tells you directly, and the runtime image has no curl, so port-forward and ask from your workstation — remembering the Host header (see the false-unhealthy case):

kubectl port-forward -n <ns> svc/<release>-trueppm-api 8000:8000

curl -s -H 'Host: trueppm.example.com' http://127.0.0.1:8000/api/v1/readyz
# {"status":"fail","checks":{"database":"ok","cache":"fail","migrations":"ok"},
#  "migration_state":"in_sync"}

The failing key in checks is your component. On Compose the equivalent is:

docker compose exec api python -c \
  "import urllib.request as u, urllib.error as e; \
   print(u.urlopen('http://127.0.0.1:8000/api/v1/readyz').read().decode())" \
  2>&1 | tail -3

Then confirm the datastore itself:

docker compose exec valkey valkey-cli ping          # → PONG
docker compose exec db pg_isready -U trueppm        # → accepting connections

# Where Valkey requires a password (docker-compose.prod.yml and the Helm chart's
# default auth.enabled: true), an unauthenticated ping returns NOAUTH, not PONG:
docker compose -f docker-compose.prod.yml exec valkey \
  sh -c 'valkey-cli -a "$REDIS_PASSWORD" --no-auth-warning ping'
kubectl exec -n <ns> <release>-valkey-primary-0 -- \
  sh -c 'valkey-cli -a "$VALKEY_PASSWORD" ping'
kubectl exec -n <ns> <release>-postgresql-0 -- pg_isready -U trueppm

Confirm the fix. /api/v1/readyz returns 200 with every key in checks reading ok, and kubectl get pods returns to READY 1/1.

---

WebSockets do not connect (no live updates)

What you see. The app works, but nothing updates until you refresh, and the status bar reports the connection as lost. The browser console shows the socket closing shortly after opening.

Likely causes, in order.

1. The proxy is not upgrading the connection. The WebSocket route is /ws/v1/projects/<uuid>/. On Compose-prod, nginx/app.conf.template proxies /ws/ with proxy_http_version 1.1, the Upgrade / Connection headers, and an 86400 s read timeout. On Helm the default ingress.hosts[].paths sends /ws to the API Service — if you rewrote that list and dropped the /ws entry, or put it after /, the socket lands on the SPA instead.
2. An intermediary is closing idle connections. Cloud load balancers, corporate proxies, and some WAFs cap idle WebSocket lifetime well below the application’s. Symptom: it connects, works, and dies on a fixed interval.
3. The close code says it is not a network problem at all. 4001 is “no valid credential”, 4003 is “role below Team Member, or membership removed”, 4404 is “no consumer at that path”. The web app stops on a 4003 and shows No live updates in the status bar — that is a Viewer (or a just-demoted member), not an outage; grant Team Member if they need live updates. 4404 is rejected before the handshake is accepted, so a browser reports it as 1006 — test an unrouted path with a non-browser client if you need the code.

Commands. Read the close code first; it decides everything else.

// Browser console, on the affected page
new WebSocket("wss://trueppm.example.com/ws/v1/projects/<project-uuid>/")
  .addEventListener("close", (e) => console.log(e.code, e.reason));

# Did the request even reach Django?
kubectl logs -n <ns> deploy/<release>-trueppm-api -c api --tail=200 | grep -i "ws/v1"
docker compose logs --tail=200 api | grep -i "ws/v1"

# Helm: confirm /ws routes to the API Service and precedes /
kubectl get ingress -n <ns> <release>-trueppm -o yaml | grep -A 4 "path:"

Confirm the fix. The socket reaches readyState 1 and stays there, and a change made in a second browser appears in the first without a refresh.

---

Celery is not processing anything

TruePPM runs background work — schedule recalculation, imports, notification email — through Celery, split across two moving parts: worker processes that do the work, and Celery Beat, a single scheduler process that dispatches jobs on a timer (every 30 seconds for most of them). Both talk through a broker — a message queue, backed by Valkey/Redis — that holds jobs between “dispatched” and “picked up by a worker.” Any one of the three can fail on its own, and each looks different from the outside; this section tells them apart.

What you see. Schedules do not recalculate, imports sit at “queued”, notification email never arrives. Reads all work.

Tip

If kubectl get pods or docker compose ps already shows the worker as Ready/(healthy) and this is still happening, read Startup, Readiness, and Liveness Probes → celery worker first — from 0.4 onward “Ready” means the worker’s heartbeat file is fresh, which is a different (and stronger) claim than “a process is running.”

Likely causes, in order.

1. The worker is down or crash-looping. Nothing consumes the queue.
2. Beat is down. The worker is healthy and idle, because nothing is dispatching — the fourteen 30-second outbox drains and every nightly job are Beat-scheduled. This looks identical from the outside, which is why Beat has its own detector.
3. The broker is unreachable. In that case the API is also 503 — work the 503 case instead.
4. The work failed permanently and was dead-lettered. It is not queued, it is parked.

Commands. Distinguish (1) from (2) first — this is the single most useful check on this page:

# Beat heartbeat. 200 = beat is alive, 503 = stale (default threshold 120 s).
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $ADMIN_JWT" \
  https://trueppm.example.com/api/v1/health/beat/

# Is a worker connected to the broker at all?
docker compose exec celery celery -A trueppm_api.celery inspect ping
kubectl exec -n <ns> deploy/<release>-trueppm-celery-worker -c celery-worker -- \
  celery -A trueppm_api.celery inspect ping
# → {'celery@<host>': {'ok': 'pong'}}

# What is actually queued or running?
docker compose exec celery celery -A trueppm_api.celery inspect active
docker compose logs --tail=100 celery celery-beat
kubectl logs -n <ns> deploy/<release>-trueppm-celery-beat -c celery-beat --tail=100

If the heartbeat is fresh and workers answer pong, check whether the work was dead-lettered rather than lost:

curl -s -H "Authorization: Bearer $ADMIN_JWT" \
  https://trueppm.example.com/api/v1/health/dead-letter/
# Prometheus text: trueppm_task_dead_letter_parked{task_name="..."} <n>

Anything above zero is triaged from Settings → Workspace → System health — see Dead-letter Alerting.

Confirm the fix. inspect ping answers, /api/v1/health/beat/ returns 200, and a fresh edit to a task’s duration produces a recalculated schedule within a few seconds.

Nothing is lost while this is broken

Outbox-backed work is recorded in PostgreSQL, not in the queue. Every task carries acks_late=True and reject_on_worker_lost=True, and the drains re-dispatch orphaned rows. Restoring the worker or Beat drains the backlog. See Why losing the broker does not lose the work.

---

A background sweep stopped early (soft time limit exceeded)

Releases after 0.4.0-beta.7. Each sweep runs under a Celery soft time limit. When a very large instance runs past it, the task stops instead of skipping the slow item and carrying on, and the worker logs a WARNING naming how many items it did not reach and the first few ids:

Sweep	Schedule	Soft limit	Log text
Sprint burndown snapshots	nightly	120 s	update_sprint_burndown_snapshots: soft time limit exceeded; N item(s) not processed (first: ...)
Recurring task occurrences	hourly	120 s	generate_recurring_occurrences: soft time limit exceeded; N item(s) not processed (first: ...)
Daily forecast floor	nightly	110 s	capture_daily_forecast_floor: soft time limit exceeded; N project(s) not processed (first: ...)
Weekly digests	hourly	280 s	send_due_digests: soft time limit exceeded; N user(s) not processed for <event> (first: <ids>)

Items are processed in a fixed order, so a sweep that keeps overrunning leaves the same tail unprocessed on every run. Those items are not eventually caught up by a later run. A repeating WARNING means the instance has outgrown the limit: check worker CPU and database latency first. A persisted cursor or per-item fan-out for these sweeps is tracked in #4420, and the handlers that still swallow the time limit are tracked in #4421; neither is scheduled work. Receivers fired from a sweep’s per-item writes (for example Task post_save receivers) can still absorb the signal before it reaches the sweep’s guard; that is tracked in #4421.

A Celery worker wedges and never recovers (Docker Compose)

What you see. docker compose ps shows celery (unhealthy) and it never clears — no RESTARTS-equivalent count climbs, and docker compose logs celery goes quiet, with no new task log lines. This is Compose-specific: on Kubernetes the same underlying condition (a worker whose event loop has wedged after connecting) is recovered automatically by the chart’s celery inspect ping livenessProbe, which kubelet acts on directly.

Why. Plain Docker Compose has no mechanism that restarts a container on a failed healthcheck — restart: unless-stopped fires only on a process exit. A healthcheck transitioning to unhealthy changes only what docker compose ps reports; nothing reads that status and acts on it. A worker that wedges after connecting — the same root cause as #3722, fixed there only for the Helm chart’s cold-start race — has no path back once it is already running.

Since 0.4 (#3936), the worker closes this gap itself: trueppm_api.core.worker_selfheal runs a background watchdog thread that reads the same heartbeat file the readiness healthcheck does (trueppm_api.core.worker_heartbeat) and self-exits the process once it has been stale for 90 seconds — 3x the healthcheck’s own 30-second staleness window, so a transient broker blip or scheduling pause cannot turn into a crash loop. That exit is what restart: unless-stopped needed all along. This is enabled by TRUEPPM_CELERY_WORKER_SELF_HEAL=1, set by both docker-compose.yml and docker-compose.prod.yml.

On an older deployment, or one that has disabled the env var, recover manually:

docker compose restart celery

Commands.

# Confirmed wedge (not a crash loop): "unhealthy" with no restart count climbing
docker compose ps celery

# Did the self-heal watchdog already act? It logs before exiting.
docker compose logs celery | grep "worker_selfheal: heartbeat stale"

Confirm the fix. docker compose ps celery returns to (healthy) within a few heartbeat cycles of the restart (whether triggered by the watchdog or run manually).

See Startup, Readiness, and Liveness Probes → Docker Compose equivalents for how this fits alongside the readiness healthcheck, and Celery is not processing anything above if the worker is (healthy) but idle instead.

---

A pod never becomes ready, or migrations are pending

What you see. READY 0/1 indefinitely, or Init:0/2, or an init container in CrashLoopBackOff. On Compose, the API logs loop on DB not ready yet or exit during migrate. On a Helm install with demo mode, helm install itself can fail with failed post-install: job <release>-trueppm-demo-seed failed: BackoffLimitExceeded — the seed Job runs the same boot guards, so a guard failure surfaces there.

Likely causes, in order.

1. The database is not reachable yet. The Compose API command retries migrate every 2 s until the database answers, which is normal for the first 30–60 s and abnormal after that.

2. The image’s migrations do not match the schema. /readyz reports migration_state as one of in_sync, behind, ahead, or unknown, and a pod is only ready on in_sync. behind means this image ships migrations the database has not applied — normal during a rollout, a stuck migrate init container if it persists. ahead means the database is newer than this image, which is what a rollback without a schema restore looks like.

3. Concurrent migrate at two or more replicas. Every API pod runs the migrate init container, and Django takes no cross-process lock. Losing pods can exit non-zero and restart until the winner finishes — expect a few init restarts on the first rollout at replicaCount >= 2, and see Singletons and one-per-pod work before an upgrade carrying a long migration.

4. A boot guard. settings.prod refuses to import without SECRET_KEY, ALLOWED_HOSTS, INTEGRATION_ENCRYPTION_KEY, and a storage choice — and the guard fires in the init containers too, so the pod dies before the api container ever starts. The log line starts Refusing to start: and names the variable.

5. Local attachment storage with nowhere to write. Setting TRUEPPM_ALLOW_LOCAL_ATTACHMENT_STORAGE=true makes the boot guard prove that MEDIA_ROOT is writable, and every chart pod has a read-only root filesystem. With no volume behind the path, the migrate init container logs:

   Refusing to start: TRUEPPM_ALLOW_LOCAL_ATTACHMENT_STORAGE is set, but local
   attachment storage cannot be used: MEDIA_ROOT '/var/lib/trueppm/media' is not
   writable: Read-only file system.

   The api pod sits in Init:Error, the Celery worker in CrashLoopBackOff, and the demo seed Job exhausts its retries. Fix it one of two ways. For a real install, set persistence.media.enabled=true so the chart mounts a claim at the media path (Attachment storage), or drop the opt-in and point TRUEPPM_DEFAULT_FILE_STORAGE at object storage. For the demo overlay, values-demo.yaml already sets env.TRUEPPM_MEDIA_ROOT: /tmp; if you replaced the env: block in your own values file, put that key back. Do not enable persistence.media for a demo — the seed Job does not mount the claim.

Commands.

# Which init container, and why
kubectl describe pod -n <ns> <api-pod> | sed -n '/Init Containers/,/^Containers/p'
kubectl logs -n <ns> <api-pod> -c migrate
kubectl logs -n <ns> <api-pod> -c bootstrap

# What does the database think is applied?
kubectl exec -n <ns> deploy/<release>-trueppm-api -c api -- \
  python manage.py showmigrations --plan | grep -c '^\[X\]'
docker compose exec api python manage.py showmigrations | grep -v '\[X\]'

Confirm the fix. showmigrations shows no unapplied entries, and /api/v1/readyz returns 200 with "migration_state":"in_sync".

---

A sync or login query is slow after an upgrade, but migrations show applied

What you see. showmigrations lists every migration as applied and /api/v1/readyz reports "migration_state":"in_sync", yet the sync endpoints or login got slower after an upgrade that landed at replicaCount >= 2 — in particular if that upgrade’s rollout showed init-container restarts on the migrate step (see the previous section).

Why. A handful of migrations build their index with CREATE INDEX CONCURRENTLY so the build never blocks writes on a live table (see Concurrent migrations at replicaCount >= 2 for the advisory-lock mechanics migrate_locked uses to serialize pods). Unlike ordinary DDL, CREATE INDEX CONCURRENTLY does not roll back if it is interrupted: PostgreSQL leaves the index behind marked INVALID — ignored by the query planner but still maintained on every write — rather than dropping it. If the pod holding the migration lock dies mid-build (OOM, eviction, a node drain) before that migration finishes and gets recorded applied, the next pod to win the lock reruns the same still-unapplied migration — and the old (pre-this-fix) version of that migration’s CREATE INDEX CONCURRENTLY IF NOT EXISTS statement saw an index of that name already present (albeit INVALID), treated IF NOT EXISTS as “nothing to do,” and let the migration finish and get recorded applied with the index still broken. The current migrations now repair this themselves: each one runs a step that drops an INVALID index of its name before (re)building it, so a crash-and-retry during the same upgrade attempt self-heals on the retry, before the migration is ever recorded applied. What this does not fix is a database that is already stuck from a crash that happened before upgrading to a version carrying this fix — there, the migration is already recorded applied, so Django will never touch it again, and the INVALID index needs the manual rebuild below. Affected indexes: projects.0090 (htask_proj_hist_date_idx on projects_historicaltask), projects.0100 (task_proj_serverver_idx on projects_task, dep_pred_serverver_idx on projects_dependency), and projects.0105 (hsprint_proj_hist_date_idx on projects_historicalsprint, hrisk_proj_hist_date_idx on projects_historicalrisk) — the sync hot-path and changelog indexes.

Commands.

# List every INVALID index in the database — not just the ones named above.
docker compose exec db psql -U trueppm -d trueppm -c "
  SELECT c.relname AS index_name, t.relname AS table_name
  FROM pg_index i
  JOIN pg_class c ON c.oid = i.indexrelid
  JOIN pg_class t ON t.oid = i.indrelid
  WHERE NOT i.indisvalid;
"

# Kubernetes: run the same query against the bundled Postgres pod.
kubectl exec -n <ns> <release>-postgresql-0 -- \
  psql -U trueppm -d trueppm -c "SELECT c.relname, t.relname FROM pg_index i
    JOIN pg_class c ON c.oid = i.indexrelid
    JOIN pg_class t ON t.oid = i.indrelid WHERE NOT i.indisvalid;"

Rebuild each INVALID index found above by hand — Django will not retry an already-applied migration, so this is a direct rebuild, not a migrate re-run. DROP/CREATE INDEX CONCURRENTLY cannot run inside a transaction, so issue the drop and the create as separate psql invocations (each one its own autocommit statement), substituting the matching table/columns from the list above:

docker compose exec db psql -U trueppm -d trueppm \
  -c 'DROP INDEX CONCURRENTLY IF EXISTS <index_name>;'
docker compose exec db psql -U trueppm -d trueppm \
  -c 'CREATE INDEX CONCURRENTLY IF NOT EXISTS <index_name> ON <table> (<columns>);'

Confirm the fix. The pg_index query above returns zero rows, and EXPLAIN on the affected query (project_id, server_version lookups on projects_task / projects_dependency, or a project’s activity/changelog feed) shows an Index Scan using the rebuilt index rather than a sequential scan.

---

Image pull failure

What you see. ErrImagePull or ImagePullBackOff in kubectl get pods, or manifest unknown / denied from docker compose pull.

Likely causes, in order.

1. The tag does not exist on that registry. The two registries carry different tag forms: the GitLab Container Registry (the chart’s default image.repository) has only the v-prefixed v0.4.0-beta.4, and GHCR (ghcr.io/trueppm/{api,web}, what docker-compose.prod.yml pulls) has only the bare 0.4.0-beta.4. The chart pins itself to v<appVersion> when image.tag is empty, so pointing image.repository at GHCR without setting image.tag to the bare version fails with manifest unknown — and the reverse applies to a v-prefixed APP_VERSION in Compose. See Image tags differ by registry.

2. You installed chart 0.4.0. The chart published as 0.4.0 came from the first beta cut and defaults to image tag v0.4.0, which was never published. It has since been removed from the registry, but a release installed from it keeps running that chart — helm list shows it as trueppm-0.4.0. Upgrade to a later beta, keeping your values:

   helm upgrade <release> oci://ghcr.io/trueppm/charts/trueppm \
     --version 0.4.0-beta.4 -n <ns> -f <your-values>.yaml

3. No pull credentials for a private registry. The chart sets no imagePullSecrets, so a private registry needs one attached to the namespace’s default ServiceAccount, or supplied through your own patch.

4. No outbound network. The nodes cannot reach the registry at all — an air-gapped or egress-filtered cluster. Mirror the images into an internal registry and set image.repository / image.webRepository to point at it.

Commands.

kubectl describe pod -n <ns> <pod> | sed -n '/Events/,$p'
# The Warning Failed event carries the exact registry error.

# Which tag is the chart actually asking for?
helm get values -n <ns> <release> --all | grep -A 4 '^image:'

# Can you pull it by hand from a machine on the same network?
docker pull registry.gitlab.com/trueppm/trueppm/api:v0.4.0-beta.4   # GitLab: v-prefixed
docker pull ghcr.io/trueppm/api:0.4.0-beta.4                        # GHCR: bare version

Confirm the fix. kubectl get pods leaves ImagePullBackOff and the pod reaches Running.

---

A container reports unhealthy while the site works fine

What you see. docker compose ps shows api (unhealthy), or Kubernetes restarts the API pod on its liveness probe — yet the application serves correctly in the browser.

For what each probe actually checks (and the reverse case — a pod that stays Ready/(healthy) while genuinely not serving), see Startup, Readiness, and Liveness Probes.

Likely cause: ALLOWED_HOSTS. The health check reaches the API on localhost / the pod IP, so Django sees Host: localhost (or Host: 10.x.x.x). settings.prod sets ALLOWED_HOSTS from the environment with no implicit entries, so if your list contains only trueppm.example.com, Django rejects the probe’s request with 400 Bad Request (DisallowedHost) — while every real request, which arrives with the right Host, is served normally.

Commands.

# The tell: 400s in the log whose only client is the health check
docker compose logs --tail=100 api | grep -i "disallowedhost\|invalid HTTP_HOST"
kubectl logs -n <ns> <api-pod> -c api --tail=100 | grep -i "invalid HTTP_HOST"

# Reproduce it deliberately — same URL, two Host headers
kubectl port-forward -n <ns> svc/<release>-trueppm-api 8000:8000
curl -s -o /dev/null -w 'no host header: %{http_code}\n' http://127.0.0.1:8000/api/v1/health/
curl -s -o /dev/null -w 'real host:      %{http_code}\n' \
  -H 'Host: trueppm.example.com' http://127.0.0.1:8000/api/v1/health/
# 400 then 200 confirms it.

The fix, per platform:

• Compose — add localhost,127.0.0.1 to ALLOWED_HOSTS in .env, alongside your real hostname. The healthchecks in both Compose files request http://localhost:8000/api/v1/health/, so those two entries are what they need.
• Kubernetes — the kubelet addresses the probe to the pod IP, which changes on every reschedule, so listing it is not an option. Give the probe an explicit Host header matching a name already in ALLOWED_HOSTS, via httpGet.httpHeaders on the probe. Adding the whole pod CIDR as a wildcard is not equivalent — ALLOWED_HOSTS does not accept CIDRs, only exact names and leading-dot suffixes.

ALLOWED_HOSTS is supplied through the Secret referenced by envFrom; see Configuration.

Confirm the fix. docker compose ps reads (healthy), the DisallowedHost lines stop, and the pod’s RESTARTS count stops climbing.

---

/static/ returns 404, or Django admin is unstyled

What you see. /admin/ renders as unstyled HTML — readable, no CSS — and the browser console shows 404 for /static/admin/css/base.css. The rest of the app is unaffected, because the SPA’s own assets are served from the web tier’s own filesystem, not from /static/.

Likely causes, in order.

1. collectstatic has not populated STATIC_ROOT, or the serving container cannot see what it collected. WhiteNoise serves /static/ out of STATIC_ROOT; if that directory is empty, every file 404s. The dev Compose API command runs collectstatic --noinput on start. docker-compose.prod.yml collects in the one-shot api-init service and shares the output with the api service through the static_files named volume — both services must set STATIC_ROOT: /app/staticfiles, or api-init reports a clean collect into a directory api never reads. On Kubernetes the target is the staticfiles emptyDir, which is empty on every fresh pod until something fills it.
2. The proxy is not forwarding /static/ to the API. Both nginx configs (nginx/app.conf.template for Compose, the web tier’s ConfigMap for Helm) proxy location /static/ to the API. A custom ingress that routes only /api and /ws to the API Service sends /static/ to the SPA instead, where it hits the try_files … /index.html fallback and returns HTML with a 200.

The second cause has a distinctive signature: the request succeeds but the response is text/html, and the browser refuses to apply it as a stylesheet.

Commands.

# Is anything there to serve?
docker compose exec api ls /app/staticfiles/admin/css | head
kubectl exec -n <ns> deploy/<release>-trueppm-api -c api -- ls /app/staticfiles | head

# What is actually being returned?
curl -sI https://trueppm.example.com/static/admin/css/base.css | head -3
# Want: 200 + content-type: text/css
# 404              → cause 1 (nothing collected)
# 200 + text/html  → cause 2 (routed to the SPA fallback)

If the directory is empty, populate it:

docker compose exec api python manage.py collectstatic --noinput
kubectl exec -n <ns> deploy/<release>-trueppm-api -c api -- \
  python manage.py collectstatic --noinput

Note that on Kubernetes this fixes one pod until it restarts — the target is an emptyDir. A durable fix populates the volume as part of pod startup or bakes the collected files into the image.

Confirm the fix. curl -sI on a static path returns 200 with content-type: text/css, and /admin/ renders styled.

---

The Helm admin password is gone

What you see. kubectl exec … -- cat /run/trueppm/admin_password returns No such file or directory.

This is expected, not a fault: the file lives on a per-pod emptyDir written by the bootstrap init container, and create_admin is a deliberate no-op once a superuser exists. After any pod restart or helm upgrade the password is gone permanently, and it cannot be regenerated.

Reset it instead of hunting for it — the full procedure, including the multi-replica caveat, is in Admin password setup.

---

Related pages

• Verify your install — the per-service checks to run before you conclude anything is wrong.
• Durability & Redundancy — the failure-mode matrix behind these symptoms.
• System Health — the same diagnosis from the UI, without a shell.
• Beat Liveness — the singleton scheduler and its heartbeat.
• Deployment — the topologies these commands assume.