Skip to content

Upgrading

This page walks through moving an existing TruePPM instance to a newer version, safely, for whichever way you run it — Docker Compose, a single server, or Helm/Kubernetes. If you are standing up TruePPM for the first time rather than upgrading one, see Installation instead.

  1. Read the changelog for the target version — check CHANGELOG.md or the release notes for breaking changes and migration notes.

  2. Back up PostgreSQL with scripts/backup.sh. Valkey state is ephemeral (broker + cache); PostgreSQL is the only stateful service. The exact invocation differs by how you run TruePPM — a bare ./scripts/backup.sh --output-dir ./backups only works when DATABASE_URL is already exported and pg_dump is on the host’s PATH, which is true for neither the production Compose stack (no host database port, no pg_dump in the application images) nor a fresh shell against Helm. See Backup & Restore for the command for your stack — Compose (development), Compose (production), or Kubernetes/Helm.

    Use this rather than a hand-rolled pg_dump > file.sql. The runbook’s restore tooling reads --format=custom archives, so a plain-SQL dump is an artifact scripts/restore.sh cannot consume — you would discover that during a rollback, which is the worst moment to discover it. The custom format also preserves CREATE EXTENSION ordering, which the ltree / pg_trgm / btree_gist objects depend on; see Backup & restore.

  3. Note your current version before starting. Read it from what is actually running, not from a latest tag, which says nothing about which release a pinned deployment uses.

    Terminal window
    # Docker Compose: the image (and so the tag) each service runs
    docker compose -f docker-compose.prod.yml ps --format '{{.Service}}: {{.Image}}'
    # Kubernetes: the image of the running API Deployment
    kubectl get deploy -n trueppm -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'
    # Helm: the chart and app version of the release
    helm list -n trueppm
  4. Check your upgrade path against Supported upgrade paths.

Upgrade one minor version at a time: 0.3 to 0.4, then 0.4 to 0.5, and so on. Skipping a minor version in one step (for example 0.3 to 0.5) is not a supported path, because each release’s migrations and operator notes assume the previous minor’s schema and configuration. TruePPM is pre-GA, so this is guidance you should follow rather than a tested guarantee. From 1.0 on, every sequential minor upgrade (1.2 to 1.3, and so on) is a release requirement. Within a minor, the same caution applies: the web-asset notes under Upgrading to 0.4 cover an upgrade from the immediately previous published release, not one that skips a release.

Scheduled dates can change between releases only for a documented reason; the changelog’s changed and fixed entries list them. A frozen corpus of schedules and Monte Carlo results enforces this on every merge request.


Every release carries a short operational change note answering one question: what does an operator have to check or change before and after this upgrade? This is distinct from the changelog (user-facing changes) — it is the operator’s pre-flight. Each release’s note appears in a versioned section on this page (see Upgrading to 0.3 below for the shape); the template used to write one is:

## Upgrading to <version>
**Migration behavior:** <additive-only / includes destructive ops / data backfill>.
Downtime: <none beyond the migrate run / brief write pause / maintenance window>.
**New or changed env vars / Helm values:**
- `NEW_VAR` — <what it does, default, whether action is required>
- `changed.helm.value` — <old → new default, action required?>
**Breaking config:** <none / describe what an existing config must change>.
**New migrations operators will see:**
- `<app>.<NNNN_name>` — <one line: what schema it adds/changes>
**Pre-upgrade action:** <back up (always) / rotate a credential / set a new value>.
**Post-upgrade verification:** <what "green" looks like — see the checklist below>.
**Rollback notes:** <forward-only? safe to roll back? migration-reversibility caveat>.

Fill this in from the release’s changelog fragments, the diff of packages/helm/values.yaml, and the new files under packages/api/**/migrations/. Even an all-additive release gets a note so the operator has a complete picture rather than inferring “nothing changed.”


Breaking config: the egress allow-list is now per surface. TRUEPPM_EGRESS_ALLOWLISTED_HOSTS exempts single sign-on only from 0.5. Before you upgrade, move every non-SSO host out of it:

  • an internal SMTP relay to TRUEPPM_EGRESS_ALLOWLISTED_HOSTS_SMTP;
  • a self-hosted GitLab or Jira (Connected Accounts) to TRUEPPM_EGRESS_ALLOWLISTED_HOSTS_INTEGRATIONS;
  • a webhook receiver on a private network to TRUEPPM_EGRESS_ALLOWLISTED_HOSTS_WEBHOOKS.

A host left only on the legacy list is refused as a private address on its surface after the upgrade. Webhook deliveries to such a host fail and are parked, not retried, so a delivery attempted during the window is not replayed automatically once you fix the list. Set the new variables on the API, Celery worker, and Celery beat processes. See Networking.

Successors of weekend actuals move on recalculation

Section titled “Successors of weekend actuals move on recalculation”

A non-working actual_start or actual_finish (a Saturday, Sunday, or calendar holiday) will be read in working time in 0.5: a finish as the end of the last working day before it, a start as the opening of the next working day, on every link type at every lag. A zero-lag finish-to-start milestone after a weekend actual_finish will show at the end of Friday instead of the start of Monday. Recalculating an existing project after the upgrade will move successors of such tasks to where the equivalent weekday actual puts them.

Offline and sync clients re-sync once (sync.0008)

Section titled “Offline and sync clients re-sync once (sync.0008)”

0.5 will add a tombstone-retention floor to the delta pull (#2474). A pull whose since is older than the floor answers resync_required: true, and the client discards its local store and pulls again from since=0 (see Tombstone retention and resync_required).

sync.0008_backfill_tombstone_floor raises every project’s floor, and the program pull’s floor, just above the current cursor. Tombstones the nightly reaper removed before the upgrade never recorded a floor, so the server cannot tell which clients missed one; the backfill therefore asks every offline or sync client to re-sync once after the upgrade. Expect one full pull per client and project at the upgrade, and a short spike in sync traffic. From 0.5, deleting a task tombstones its links, relations, recurrence rule and suggested assignees with it, so reaping it later no longer forces a re-sync. Two cases still do until #4951 is fixed: reaping a task deleted before the upgrade that still had live child rows, and reaping a deleted project’s tasks.

The same migration moves every Monte Carlo run recorded at the old cursor along with it. The cursor doubles as the forecast’s plan version, so without that a forecast that was current before the upgrade would read “project changed” after it. A forecast cached in the last 24 hours is moved too, on a best-effort basis: if the cache is unreachable during migrate, that forecast reads “project changed” until it expires or someone re-runs it.

The backfill does not cover a reap that a 0.4 worker runs during the rolling update itself. Such a reap does not raise the floor, and two things can leave a client holding a deleted row afterward with no re-sync:

  • The 0.4 reap’s delete cascades to live child rows of a reaped task, such as a dependency or a task link. Those rows can carry cursors from after the backfill.
  • A tombstone that was saved again after it was deleted, on a model that ages on its deletion date (tasks, dependencies and task relations), keeps its old deletion date, so it is still eligible, but its cursor was drawn at that later save. If that save came after the backfill, its cursor sits above the floor. Risks and sprints age on their last update, so a re-save renews them.

The reap runs on the beat schedule (sync.reap_domain_tombstones, daily at 03:30 UTC). To close the window, stop the scheduler before the migration runs and let the upgrade start it again:

  • Helm: kubectl scale deployment -n <ns> -l app.kubernetes.io/component=celery-beat,app.kubernetes.io/instance=<release> --replicas=0, then helm upgrade. The upgrade restores the replica count.
  • Docker Compose: docker compose stop celery-beat, then pull and docker compose up -d.

Alternatively, schedule the upgrade away from 03:30 UTC, or roll the workers to the new image before the API pods run migrate. A 0.5 worker raises the floor itself.

Rollback notes: the migration’s reverse is a no-op. Rolling back leaves the raised floors in place, which 0.4 never reads.

Sync index rebuilds lock tables (access.0021, projects.0167, integrations.0016, timetracking.0005)

Section titled “Sync index rebuilds lock tables (access.0021, projects.0167, integrations.0016, timetracking.0005)”

0.5 will rebuild 14 delta-pull indexes so the keyset resume can seek on (owner, sync_seq, id) (#2510). Each migration drops the old index and recreates it with a plain, non-concurrent CREATE INDEX. Each migration is atomic and drops all of its old indexes before it builds the new ones, so every table it touches is locked against reads and writes until that migration’s last index is built, not just while its own index builds. The migrations run on container start, so a rolling deploy’s old pods also wait on the lock.

MigrationTables
access.0021access_project_membership, access_program_membership, plus two drop-only tables: access_user_defined_mention_group, access_program_user_defined_mention_group
projects.0167projects_task, projects_dependency, projects_task_relation, projects_label, projects_risk, projects_sprint, projects_sprintretro, projects_retroactionitem, projects_taskrecurrencerule, projects_tasksuggestedassignee, plus two drop-only tables: projects_pulseresponse, projects_retroboarditem
integrations.0016integrations_tasklink
timetracking.0005timetracking_time_entry

Five of these are on the tables that grow with project size: task_proj_syncseq_idx on projects_task, and the indexes on time entries, task links, dependencies and task relations. Build time grows with their row counts. We have not measured a per-million-rows figure; to size the window, time the upgrade on a restored copy of your database first, and schedule it away from peak edit traffic on a large install.

Rollback notes: the migrations are reversible, but reversing them rebuilds the old indexes the same way and takes the same locks.

Demo compose stack generates its own secrets

Section titled “Demo compose stack generates its own secrets”

docker-compose.demo.yml will stop baking a shared SECRET_KEY and INTEGRATION_ENCRYPTION_KEY in 0.5. A new secrets-init service will generate them on first boot into a demo_secrets volume. No action will be needed on an existing demo install: its first up after the upgrade will create the secrets, and the demo holds no data that depends on the old keys. The Postgres password is unchanged, so your existing database volume will keep working. If you ran the demo stack on a reachable host before 0.5, treat the old public key as exposed and restart it on 0.5 so the key is replaced.

Section titled “Finish-to-Finish and Start-to-Finish links into summary tasks are retired (projects.0175)”

From 0.5, the scheduler will refuse a Finish-to-Finish or Start-to-Finish link whose successor is a summary task (ADR-0370): it would force every subtask, even a short unrelated one, to finish with the predecessor, and the engine cannot model it. Plans written before that rule can already hold such a link, and an unmodified one would make every recalculation of that project fail. The migration will therefore soft-delete exactly those links: a summary here means a task with at least one live child that the scheduler schedules (not a recurring template, epic, or backlog item). A link into a parent whose only children are backlog, recurring, or epic items is accepted by the engine and is left alone. The soft-delete will set the same tombstone fields a user delete sets, so offline clients will receive the removal on their next sync, but it will write no history row, and a stale offline edit to a retired link will get a 409 conflict. It will write one retired ... link ... into summary task ... warning per link to the migration log, naming the project, the link and both tasks, so you can redraw it between subtasks. It will touch no other link type, no link out of a summary that was already legal, and nothing else, and it will be safe to run more than once. Read those lines after the upgrade if a plan shows a dependency missing from a phase. A link whose backlog child is later committed (#5046) is not covered by the migration.

New API tokens carry api:full (projects.0170)

Section titled “New API tokens carry api:full (projects.0170)”

From 0.5, a token minted without scopes will carry ["api:full"] instead of ["legacy:full"]. Both names mean the same scope, existing tokens keep legacy:full and keep working, and no data is rewritten. The migration only changes the allowed values and runs no SQL against the token table.

Two windows cannot understand the new name, because a 0.4 API pod only knows legacy:full:

  • During a rolling update, a full-access token minted on a 0.5 pod is refused (401 on the general API, 403 on task sync) by any 0.4 pod still serving. It works once the rollout completes. To avoid this, do not mint tokens until every API pod runs 0.5.
  • After a rollback to 0.4, every token minted on 0.5 without an explicit scope stops working until you re-mint it on 0.4. Tokens minted with {"scopes": ["legacy:full"]} are unaffected.

Migration behavior: includes destructive ops (see below). Downtime: a maintenance window sized to your task count, because projects.0148 blocks all reads and writes on the task table while it builds a constraint (see the projects.0148 caution below), plus the rest of the migrate run and the transient-500 rollout windows described per-migration below on multi-replica installs.

Going from 0.3 to 0.4 (the 0.4 line began at v0.4.0-beta.1 on 2026-09-15 and now continues through its release candidates) runs five migrations that contain a RemoveField, DeleteModel, or raw DROP TABLE: profiles.0008, webhooks.0009, projects.0123, projects.0149 and sso.0002. Four of them (all but sso.0002) can break requests that land on an old pod during a rolling deploy, and one of them (projects.0149) permanently deletes data. These are destructive in a way an operator needs to plan around, not just a schema-hygiene detail. If you are upgrading from 0.3, read all five entries below before you run migrate, in particular the workshop-data warning:

Blank pages during an upgrade of the web tier. Each release’s index.html names that release’s content-hashed /assets/ files, so a browser tab opened before an upgrade asks the new pods for the old release’s files the next time it opens a page it has not loaded yet. Releases after 0.4.0-beta.7 keep the previous release’s /assets/ files in the web image, so those tabs keep working on any number of web replicas. The limits:

  • One release back. Covered: an upgrade from the immediately previous published release, not counting a withdrawn one. 0.4.0-beta.7 was withdrawn, so an upgrade from 0.4.0-beta.6 is covered. Not covered: an upgrade that skips a release. Tabs opened before it reload once, automatically, when they next need a file.

  • During the rollout, the other direction. Old and new web pods serve together until the rollout finishes, and a browser that got the new index.html can ask an old pod for a new file, which answers 404. With two or more web replicas this lasts for the whole web rollout. On a single replica it lasts well under a second, but it still opens, because Kubernetes’ default rolling update starts the new pod before it stops the old one. Both ends of a page load recover on their own:

    • A page the app opens after it has loaded (a lazy route) retries the file twice, over about two seconds, before it reloads.
    • The first load of the page, where the app’s own entry script or one of the files it needs to start is the one that fails, reloads after 1, 3, 8 and 15 seconds until the page starts. If it still has not started after the fourth reload, it shows TruePPM is being updated with a Reload button instead of a blank page.

    Neither one retries while the browser is offline. An offline browser sees You appear to be offline with the same Reload button instead.

    If you would rather have no skew at all, set the web tier’s rollout strategy (rollout.web, see Helm values). strategy: Recreate, or maxSurge: 0 on a single web pod, stops every old pod before a new one starts. That removes the window, but it turns every web rollout into a full outage of the UI for as long as the new pod takes to become ready, and with one pod, a node drain is an outage too. The default rolling update keeps the UI up and lets the client recover from the window.

  • Upgrading to 0.4.0-beta.7 or earlier has neither; a browser can show a blank page until it reloads.

  • The first upgrade to a release that adds the first-load recovery does not fully have it during its own rollout. The recovery is a separate file, /boot-guard.js, which the release you are upgrading from does not have. A browser that gets the new index.html and then asks an old pod for that file starts without it, and behaves as before for that one page load: a blank page until it reloads. Every later upgrade is covered, because both releases then carry the file.

  • Upgrading from 0.4.0-beta.7, specifically. This one hop is not covered by either mitigation above, even on a release that ships both: a beta.7 web pod predates the nginx rule that answers a missing hashed asset with a clean 404 and the client-side vite:preloadError reload handler, so during the rollout it answers a request for the new release’s chunk with its SPA fallback page instead — and a browser that already has the new index.html gets no clean error to retry on. If your web tier runs more than one replica, scale it to 1 before starting an upgrade from 0.4.0-beta.7 (helm upgrade … --set replicas.web=1 with your other -f/ --set flags unchanged, or kubectl scale deployment/<release>-trueppm-web --replicas=1), wait for the rollout to finish, then scale back up. Every later beta.N → beta.(N+1) hop is covered automatically once both tags carry the fix; this is the one upgrade where it is not.

See cause 4 under The browser shows a blank page and #4341.

Known transient-500 windows on multi-replica installs. Four of the five migrations drop a column or table outright, with no intervening null=True-then-remove deprecation release, so an old pod’s code still names what the new schema no longer has. If you run the API at replicaCount >= 2 (the posture values-prod.yaml sets by default, and the posture 0.4’s basic-HA carve-out encourages), Kubernetes’ default RollingUpdate keeps an old pod serving while the new pod’s migration applies — for the duration of that rollout, each of the following reads or writes will get a transient 500 from any request landing on an old pod:

  • profiles.0008_remove_userprofile_schedule_in_deliver — drops the retired schedule_in_deliver column outright (ADR-0942 §3, #3137). Affects the old pod’s get_profile_prefs() read, used by /auth/me/ (the web shell’s bootstrap call). See #3782 for the full analysis.
  • webhooks.0009_encrypt_webhook_secret_and_failure_counters — encrypts Webhook.secret into a new secret_ciphertext column and drops the plaintext secret column in the same migration (#2885). On the old pod, webhooks/models.py still declares secret as a concrete, non-deferred CharField, so Django selects it on every Webhook query — every webhook read and every outbound delivery attempt from an old pod fails with ProgrammingError: column webhooks_webhook.secret does not exist for the duration of the window. This silently stops the entire outbound-webhook subsystem with no operator-facing warning; if you depend on webhook deliveries during the rollout, scale to a single replica for the upgrade (see the mitigation below).
  • projects.0123_remove_historicalproject_agile_features_and_more — drops Project.agile_features / HistoricalProject.agile_features outright (#2025), with no deprecation window and no migration docstring. Project and HistoricalProject reads happen on nearly every authenticated request path, so this has the broadest blast radius of the four — but no data-loss concern: the field is re-exposed as a derived, read-only SerializerMethodField (ProjectSerializer.get_agile_features), so this is dead-column removal, not a feature regression. What is missing is purely the HA-safety disclosure this entry now provides.
  • projects.0149_drop_workshops_tables — drops the workshops_session and workshops_participant tables. A 0.3 pod’s code still serves Board Workshop Mode, so its /api/v1/projects/{id}/workshop/* requests fail with relation "workshops_session" does not exist for the duration of the window. Nothing else reads those tables, so the blast radius is limited to workshop requests; the data loss itself is covered in the danger box above.

sso.0002_remove_oidcprovider_workspace_ssoproviderpolicy_and_more also drops tables, but it has no window: it first shipped in v0.4.0-beta.1 together with sso.0001, so no earlier release has code that reads the tables it removes.

This is a known, bounded, and accepted trade-off for each of the four, not a regression to report: no crash-loop, no data loss beyond what each entry states, and every window closes as soon as the rollout finishes. Single-replica installs never see any of this. If you are on replicaCount >= 2 and want to avoid the windows entirely, scale to one replica (kubectl scale deployment/trueppm-api -n trueppm --replicas=1) before the upgrade and scale back up once the rollout completes.

Already running v0.4.0-beta.1 or later? These migrations already ran — there is nothing to re-run and no remediation for the schema changes themselves. The one exception is workshop data: if you upgraded from 0.3 without exporting it first, it is already gone (see the danger box above) — this section exists so the next self-hoster upgrading from 0.3 does not lose theirs the same way.

New migrations operators will see:

  • profiles.0008_remove_userprofile_schedule_in_deliver — drops the retired schedule_in_deliver boolean preference. See above.
  • webhooks.0009_encrypt_webhook_secret_and_failure_counters — encrypts webhook signing secrets and adds failure-tracking columns. See above.
  • projects.0149_drop_workshops_tables — drops the two tables backing the removed Board Workshop Mode. Destructive, irreversible. See above.
  • projects.0123_remove_historicalproject_agile_features_and_more — drops the dead agile_features column (now derived). See above.
  • sso.0002_remove_oidcprovider_workspace_ssoproviderpolicy_and_more — migrates any OIDCProvider/OIDCIdentity rows to allauth’s SocialApp / SocialAccount (ADR-0517), then drops the bespoke OIDCProvider and OIDCIdentity models. Benign: SSO never shipped in a tagged 0.3 release, so no installation has rows to migrate or lose — no disclosure needed beyond this line.

Breaking API changes: 0.4 also removes several /api/v1/ paths that appeared in the published v0.3.0-alpha.3 schema. These are documented as deliberate deprecation-window exceptions in API Stability & Deprecation Policy — check there for the full reasoning and migration guidance for each. If you have an integration calling /api/v1/projects/{id}/workshop/*, /api/v1/tasks/{id}/scope/, /api/v1/teams/{id}/, or /api/v1/projects/{id}/history/summary/, read that section before upgrading.

Pre-upgrade action: back up PostgreSQL (always — see Before you upgrade); if you have workshop data, export or preserve it before running migrate (see the danger box above); if you cannot tolerate a transient-500 window on webhook deliveries, /auth/me/, project reads or workshop endpoints during rollout, scale to one replica first. Post-upgrade verification: see the checklist below. Rollback notes: all five migrations are destructive or table-dropping — per migration reversibility, a clean rollback means restoring the pre-upgrade backup, not a migrate reverse. projects.0149 in particular cannot be reversed by any means other than restore — its reverse_sql is a no-op by design.

Helm: the api and web pods get a default-deny ingress NetworkPolicy

Section titled “Helm: the api and web pods get a default-deny ingress NetworkPolicy”

Upgrading the Helm chart from 0.4.0-beta.3 or earlier adds default-deny ingress NetworkPolicies to the api, web, celery-worker, and celery-beat pods. The api and web policies admit traffic only from the ingress controller named by networkPolicy.ingressControllerSelector. By default, that is a namespace called ingress-nginx.

If your controller runs anywhere else and your CNI enforces NetworkPolicy, those policies would cut off all traffic to the site, and every pod would still report Ready. k3s (Traefik) and RKE2 (rke2-ingress-nginx) both run their controller in kube-system and enforce policies out of the box — and a tunnel such as Cloudflare Tunnel is not an ingress controller at all, so it never matches the default no matter which namespace it runs in. To prevent a silent outage, the first upgrade that adds these policies refuses to render until you choose one of the following:

Terminal window
# Your controller really is in the ingress-nginx namespace:
helm upgrade trueppm ... --set networkPolicy.ingressControllerConfirmed=true
# It runs elsewhere, for example kube-system (or wherever your tunnel client runs):
helm upgrade trueppm ... --set-json \
'networkPolicy.ingressControllerSelector={"namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"kube-system"}},"podSelector":{}}'
# Defer the policies (you can enable them later):
helm upgrade trueppm ... --set networkPolicy.enabled=false

The check above runs only on the upgrade that introduces the policies — fresh installs and later upgrades skip it, because it compares against a policy that must already exist for there to be a “transition” at all. A separate check fires on a fresh install too, for the two paths that bypass any in-cluster ingress controller entirely: demo.enabled (the documented Cloudflare Tunnel exposure) and service.web.type: LoadBalancer/NodePort. Neither has a prior policy to compare against, so the render simply refuses while the selector is still the default — see Public read-only demo mode. A cloud load balancer that sends traffic from outside the cluster needs an ipBlock peer instead; see networkPolicy.ingressControllerSelector.

To upgrade a release you installed with --set flags without retyping them, start from what Helm recorded at install time:

Terminal window
helm -n <namespace> get values <release> -o yaml > current-values.yaml
helm upgrade <release> oci://ghcr.io/trueppm/charts/trueppm --version <version> \
-n <namespace> -f current-values.yaml

This keeps your own settings and picks up the new chart’s defaults. --reuse-values does not: it also freezes the previous chart’s defaults.

Three more things to check for this upgrade:

  • In-cluster monitoring. If Prometheus or a Blackbox exporter in another namespace scrapes the api Service directly (the health endpoints described in Observability), set networkPolicy.monitoringSelector to that namespace. Otherwise the new policy drops those scrapes, and the dead-letter and beat-staleness alerts go quiet instead of firing. Scrapes that go through your public hostname pass through the ingress controller and are unaffected.
  • Previews. A client-side helm upgrade --dry-run cannot see the cluster, so it always reports this refusal. Use --dry-run=server to preview the upgrade accurately.
  • GitOps. Tools that render the chart with helm template and apply the result themselves, such as Argo CD, never run this check. Set the selector before you sync.

Helm: seven values keys moved into the concern-first maps

Section titled “Helm: seven values keys moved into the concern-first maps”

Upgrading the Helm chart from 0.4.0-beta.6 or earlier moves seven values keys that 0.4 prereleases introduced. The chart lays out its values concern first, then tier (resources.api, probes.worker, lifecycle.web), and these seven went the other way. They move before 0.4.0, because the schema rejects unknown keys and any later rename would break helm upgrade for everyone who had set them (ADR-1248).

Old key (0.4 prereleases)New key
web.replicaCountreplicas.web
web.service.typeservice.web.type
web.service.portservice.web.port
api.workersprocesses.api.workers
celeryWorker.concurrencyprocesses.worker.concurrency
celeryWorker.maxTasksPerChildprocesses.worker.maxTasksPerChild
celeryWorker.extraArgsprocesses.worker.extraArgs

The defaults and the behavior do not change, only where the key lives. If your values file or --set flags still use an old key, the render refuses and names it. The wording depends on your Helm version:

# Helm 3.14 and similar
(root): Additional property celeryWorker is not allowed
web: Additional property replicaCount is not allowed
# Newer Helm releases
- at '': additional properties 'celeryWorker' not allowed
- at '/web': additional properties 'replicaCount' not allowed

Nothing is silently dropped. Rename the key and run the upgrade again. As a YAML diff:

# Before (0.4.0-beta.6 and earlier) # After
api: processes:
workers: 4 api:
celeryWorker: workers: 4
concurrency: 4 worker:
web: concurrency: 4
replicaCount: 2 replicas:
service: web: 2
type: LoadBalancer service:
web:
type: LoadBalancer

Every 0.3 key keeps its name and meaning. The top-level replicaCount still sets the API and worker replicas, service.type/service.port still describe the API Service, and a values file written for 0.3 renders unchanged. Keys that exist only for the web tier’s nginx stay under web:: web.enabled, web.containerPort, web.maxBodySize, web.adminAccess.*, and web.securityHeaders.*. The api: and celeryWorker: roots no longer exist.

Upgrading from 0.4.0-beta.4 or earlier changes how the scheduler places zero-duration milestones (#4079). No migration runs and no stored date changes during the upgrade itself; the new dates appear the next time each project’s schedule recalculates, which happens on the next edit that affects the schedule.

Earlier versions gave every milestone a working day of its own, so each milestone on a path pushed everything after it back by one working day. The scheduler now follows the MS Project and Primavera P6 convention: a milestone is a point in time. A milestone that follows work sits on the day that work finishes, and the next task starts on the following working day, exactly as if the milestone were not there. A milestone with no predecessor, such as a project-start milestone, sits at the start of its day and does not delay the work after it.

What you will see after recalculation:

  • Dates move earlier. Downstream tasks, forecasts, and the project finish move earlier by one working day for every milestone on their path. A milestone that follows work is now shown on its predecessor’s finish day rather than the day after it.
  • Float and the critical path can change. A milestone no longer adds a day to the paths it sits on, so float on parallel paths is recomputed against the shorter path.
  • Monte Carlo forecasts move with the schedule. P50/P80/P95 use the same convention, so a plan with fixed durations still simulates to its CPM finish.
  • Imported MS Project plans now match their source file. Recalculated dates for an imported plan no longer drift one day per milestone from the dates in the .xml file.

Baselines captured before the upgrade keep the old dates, so the first comparison after recalculation can show milestone-driven variance that reflects the convention change, not a schedule change. To compare like with like, capture a new baseline after the project recalculates.

Data dates and SNETs before the project start

Section titled “Data dates and SNETs before the project start”

Upgrading from 0.4.0-rc.1 or earlier changes how the scheduler treats a data date (status date), or a task’s start-no-earlier-than date (SNET), that falls on or before the project start (#4635). No migration runs and no stored date changes during the upgrade itself. The new dates appear the next time each project’s schedule recalculates, which happens on the next edit that affects the schedule.

A project with no status date uses today as its data date, so this mostly affects projects that start in the future. Earlier versions measured a lag out of a milestone with no predecessor, such as a kickoff milestone, from that earlier data date rather than from the project start. The same happened with any milestone whose SNET is before the start, including a completed milestone that has no recorded dates and carries such an SNET. The work after such a milestone lost up to the gap between today and the project start from every lag. A data date or SNET on or before the project start now holds nothing back, except that an SNET still holds a task that has a Start-to-Finish link.

What you will see after recalculation:

  • Dates after a kickoff milestone move later. On a project that starts in the future, the successors of a milestone with no predecessor regain their full lag. Their dates, the forecast finish, and the critical path move later by up to the gap between today and the project start.
  • Some milestone successors move earlier. A milestone with no Start-to-Finish link that a lead places before the project start used to be held at a data date or SNET that fell between that earlier point and the start, and its successors measured their lags from there. That bound is now ignored, so they measure from where the links place the milestone. For example, with P (3 days) -FS-10d-> M -FS+5d-> X and the project starting Monday 02-02, a status date of 01-30 used to start X on 02-04; it now starts on 02-02, as with no status date.
  • Some Start-to-Finish tasks move earlier. A task with a Start-to-Finish link that its link places before the project start is no longer held at a data date that is on or before the start. It moves to where its links place it, unless its own SNET holds it. On a project that is already underway, the data date still holds it.
  • Monte Carlo forecasts move with the schedule. P50/P80/P95 use the same rule.

What the Schedule’s Snap to project start and Move project start options do for a task before the project start now depends on its links. Both write a start-no-earlier-than date (SNET) on or before the (possibly moved) start:

  • A task with a Start-to-Finish link is held there. Its links alone would put it earlier, and the SNET is the only thing holding it, so the engine keeps it.
  • A milestone with no Start-to-Finish link stays where it is shown, but its successors follow its links. A lead from its predecessor places it before the start. The milestone itself is still shown on the project start, or the new start, as before, but the lags out of it are measured from where its links place it, not from the SNET. For example, with P (3 days) -FS-10d-> M -FS+5d-> X from 02-02, snapping M to the start used to start X on 02-09 and now starts it on 02-02. Moving the start to 01-26 used to start X on 02-02 and now starts it on 01-26. The old behavior was the defect this release fixes: a pre-start SNET on such a milestone pulled its successors in a way that also depended on today’s date. The milestone’s placement comes from its links, so its SNET carries no information about when its successors can start.

Baselines and saved forecast snapshots keep the old dates, so the first comparison after recalculation shows a one-time step that reflects the rule change, not a schedule change. To compare like with like, capture a new baseline after the project recalculates.

Completed milestones after the first recalculation

Section titled “Completed milestones after the first recalculation”

The scheduler now keeps a completed milestone in place no matter how the open work before it changes (#4525). On each recalculation it reads back which edge of its day (start or end) the last schedule placed the milestone on. The first recalculation after upgrading reads what the previous version saved, so for one recalculation you may see the following:

  • A milestone that was In Progress before it was completed moves to its completion day. The row used to show the day it entered In Progress. It now shows the day it was signed off, which is where the scheduler already measured the work after it from.
  • Work after a completed milestone can move by one working day, once. This happens only when a task that leads into the milestone is still open. The scheduler keeps the edge the previous version saved, or, if that was saved under the old rule, works it out again from completed work only. Either way it happens once and then stays put. When every task that leads into the milestone is complete, the completed work decides. Neither case needs any action.

Baselines are not recalculated. If you captured a baseline after such a milestone was completed, the baseline still holds the day it entered In Progress, while the current schedule shows the day it was signed off. The first variance comparison after upgrading therefore shows a slip of the gap between those two days. That is not new slip. To compare like with like, capture a new baseline after the project recalculates.

Upgrading from 0.4.0-beta.4 or earlier makes every project and program key (the code field; see Project and program keys) unique across the workspace, compared case-insensitively. Migration projects.0154_object_keys repairs existing data before it adds the uniqueness constraint:

  • A project or program with no code gets one derived from its name. Platform Migration becomes PM, and a program called Office Move becomes office-move.
  • When several projects (or several programs) share a code, the oldest keeps it and each of the others gets a numeric suffix: PLAT2, PLAT3 for projects, atlas-2 for programs.
  • Every other code is kept exactly as it is, including an existing hyphenated project code such as GA-SEC. New project keys can’t contain hyphens, but existing ones are never rewritten.

Each code the migration changed is logged at WARNING during migrate with the prefix key repair (ADR-1237). The log is optional: the database records every rewrite too. To list them later, run this in python manage.py shell:

from trueppm_api.apps.projects.models import ObjectKey
for row in ObjectKey.objects.filter(source="backfill").select_related("project", "program"):
owner = row.project or row.program
print(row.kind, row.key, owner.name if owner else "(deleted)")

If a rewritten key isn’t what your team wants, rename it on the project’s or program’s General settings page. The old key keeps working in links and now opens the renamed project.

Rolling upgrades. The constraint excludes blank codes for this release. While an older API pod is still running during a rolling update, it can create a project with an empty code without failing. The new version gives that project a key the next time it is edited. The exclusion is planned to be removed in 0.5.

Section titled “created_by is an integer user id on share links, stakeholders, and saved board views”

Upgrading from 0.4.0-beta.6 or earlier changes one response field on three API surfaces. This affects only API clients and scripts; the web UI is updated with the server, and no data or migration is involved.

  • Share links (/api/v1/projects/{id}/share-links/) and external stakeholders (/api/v1/programs/{id}/external-stakeholders/): created_by was the creator’s display name. It is now the creator’s integer user id, and the name moves to a new created_by_name field.
  • Saved board views (/api/v1/projects/{id}/board-views/): created_by was the creator’s user id as a string, such as "7". It is now the integer 7.

If a script reads either field, update it before upgrading. See the API stability policy for why this change ships between betas.

Dead-letter and observability scrape credentials now require superuser

Section titled “Dead-letter and observability scrape credentials now require superuser”

Upgrading from 0.4.0-beta.4 or earlier changes who can reach these endpoints. Through 0.4.0-beta.4, FailedTaskViewSet (dead-letter requeue/drop/list) and the observability app’s System Health, Prometheus metrics (/api/v1/health/beat/, /api/v1/health/dead-letter/, /api/v1/health/email/), retention policy, and telemetry-export endpoints were gated with Django’s IsAdminUser, which passes for any is_staff=True account — including one that is not a superuser. They now use IsWorkspaceOperator, which checks is_superuser directly (#4009). Superusers are unaffected; a WorkspaceRole.ADMIN membership was never sufficient for these endpoints and still is not.

This is a least-privilege regression for anyone who provisioned a staff-only service account for scraping or automation against these endpoints — it now gets 403 Forbidden. Before upgrading:

  • Identify any Prometheus scrape job, alerting script, or automation that calls /api/v1/health/{beat,dead-letter,email}/, the retention endpoints, or the dead-letter queue admin API.
  • Confirm the JWT it uses belongs to a superuser account (is_superuser=True), not merely a Django-admin (is_staff=True) account. create_admin produces a superuser and is unaffected.
  • If it does not, mint a new token from a superuser account before or immediately after the upgrade, or the scrape/automation starts failing silently (a gauge that stops updating, not a loud error) the moment the new code is live.

See Dead-letter Alerting and Beat Liveness for the affected scrape configs.

Program exports and program and project webhooks honor project membership

Section titled “Program exports and program and project webhooks honor project membership”

Behavior change, no migration. Upgrading from 0.4.0-beta.6 or earlier: a program export, the program webhooks of a Program Admin, and a program-scoped API token used to reach every member project regardless of the minting admin’s own project membership, and a project webhook kept delivering after its creator lost access to that project. What changed:

  • Program export (JSON seed and async bundle). Only member projects the requesting user holds project membership on are included. A Program Admin who is not a member of a project no longer receives that project’s tasks, attachments, time entries, history, or MS Project XML from a program export. Program-level content (roster, program backlog, ceremonies) is still exported. A scheduled or queued bundle is evaluated when it is built, not when it was requested. A bundle can be downloaded only by the admin who requested it, and an admin’s export request no longer returns another admin’s job that is still in flight.

  • Program-scoped webhooks. A program webhook receives a member project’s events only while its creator (created_by) holds Project Manager or above on the specific project the event fires from — a program grant has never been a project-read grant (#4310) — and holds Program Manager or above on the program itself (#4330). The second check matches the delivery-log read gate exactly (IsProgramAdmin plus the creator-only check), which is keyed on the registrant’s program role, not their project role; the first check has no equivalent in the read gate and exists purely because dispatch, unlike reading the log, hands a specific member project’s data to someone who may hold no role on that project at all. A creator demoted to Resource Manager, Team Member, or Viewer on the firing project, or to anything below Program Manager on the program, stops receiving events even though they are still a live member in both places — either demotion alone is enough to stop delivery. Removing either membership entirely has the same effect. Events already queued, or retrying, are still delivered. A webhook whose creator account was deleted, or is deactivated, delivers no member-project events. Editing (PATCH/PUT) requires the caller to currently hold Program Manager or above and be the webhook’s creator and still hold live membership (any role — presence, not role, is what this leg checks) on every project the program covers; deleting requires only that the caller currently hold Program Manager or above, with no creator or project-membership check at all, so any current Program Admin can delete it, not only its creator. The practical consequence: a creator demoted below Program Manager on the program loses the ability to edit or delete the webhook themselves — the same demotion that stops dispatch on the program axis also removes their only path to self-service disable it, and only a different Program Admin can intervene (by deleting it; nobody but the original creator can ever PATCH it, since editing’s creator check cannot be satisfied by anyone else). A demotion on the project axis alone — the creator stays Program Manager+ but drops below Project Manager on a covered project while remaining a live member there — does not block edit or delete at all, because neither checks project role, only project membership presence for edit and nothing for delete; it only stops that project’s deliveries. Only a full removal from a covered project (not a demotion) blocks editing, via the presence check — deleting still works even then, since delete does not consult project membership. If the creator has been fully removed from any project in the program, including one added later, nobody can edit the webhook, but any Program Admin can still delete it and it can be re-created under a qualifying account. Only the creator can read its delivery log.

  • Program-scoped API tokens (task-sync and acceptance-results). These two write endpoints — inbound task-sync and CI acceptance-result ingest — now also require the token’s minter (created_by) to hold project membership on the target project, checked on every request, not just at mint time. Revoking the minter’s membership stops the token from writing into that project on the very next request, with no re-mint needed. A token whose minter was deleted (and so carries no created_by) can no longer write into any project through this path. A further role floor sits on top of this membership check: the minter must also hold Member or above, not merely live membership, so a Viewer-minted program token is refused on both write endpoints (#4331).

  • Project-scoped API tokens (task-sync and acceptance-results). These two write endpoints apply a request-time floor to a project-scoped token’s minter, based on the minter’s last recorded role on the token’s project, live or removed. A minter demoted to Viewer loses write authority on the token’s very next request. A minter removed from the project while still Member or above keeps a working token, because project-scoped tokens are org assets that survive their minter’s off-boarding. A minter demoted below Member and then removed, including one who removes themselves, stays refused (#4334). (A token whose minter’s account was deleted is a separate case: both endpoints refuse it, because it is left with no user to act as.)

  • Project-scoped webhooks. A project webhook receives that project’s events only while its creator holds Project Manager or above there, checked on the same schedule as program webhooks above. Removing the creator’s membership, demoting it below Project Manager, or deleting or deactivating their account, all stop new deliveries; events already queued, or retrying, are still delivered. Editing a project webhook, deleting it, and reading its delivery log were not affected — those already require Project Manager or above and re-check the caller’s current membership and role on every request, so a removed or demoted creator already lost those abilities today.

Operator check after the upgrade. Program webhooks now showing no deliveries for a member project usually belong to a creator without membership there, or one demoted below Project Manager on that project while remaining a member (Resource Manager, Team Member, or Viewer) — or, independently, one demoted below Program Manager on the program itself while still Project Manager on every covered project. Either restore the creator’s role (project-level, program-level, or both, depending on which check failed), or re-create the subscription under a current qualifying account. Whether the creator can self-service it depends on which axis demoted them. If only the project role dropped (the creator is still Program Manager+), they can still edit or delete the webhook themselves — editing’s project-membership leg checks live presence, not role, so a demotion (as opposed to a full removal) on the project side never blocks it, and deleting does not consult project membership at all. If the program role dropped below Program Manager, the creator loses both abilities at once: editing and deleting a program-scoped webhook both require the caller to currently hold Program Manager or above on the program, with no exception for the webhook’s own creator. In that case the creator cannot disable it themselves at all — restoring their program role, or having a different current Program Admin delete the subscription (delete has no creator check, unlike edit), is the only remedy; nobody but the original creator can ever edit it, since editing’s creator check cannot be satisfied by anyone else, demoted or not. A full removal from a covered project (not merely a demotion) blocks editing via the presence check even with Program Manager intact, but a still-Program-Manager creator can still delete it in that case too, since delete never checks project membership. A project webhook showing no deliveries belongs to a creator removed from that project, or demoted below Project Manager on it; re-add or re-promote them, or have a different, still-qualifying Project Manager/Admin disable or re-create the subscription — unlike the program case, a project webhook’s own edit/delete already requires Project Manager or above (matching the dispatch and delivery-log gates), so a demoted creator loses the ability to disable their own stale webhook too, and cannot self-service it back to health. A test ping is not a reliable health check here: it is gated on the caller’s own current role, not the registrant’s membership or role, so a current Admin’s test ping still succeeds against a dormant webhook and shows up in the delivery log looking healthy while real events are silently dropped. A program-scoped integration token that starts returning 401 on task-sync or acceptance-results usually means its minter lost project membership (on 0.5 and later, also program membership, or was demoted below Member in the program); add them back, or have a Program Admin (or above) re-mint the token. The manage.py export_program command is unaffected: it runs as the operator with shell access and exports every member project.

WebSocket replay seq becomes per-project (sync.0006)

Section titled “WebSocket replay seq becomes per-project (sync.0006)”

The seq on a real-time board event is now a per-project, dense counter instead of the replay buffer’s global row id (#4386). Operators have nothing to configure.

What the migration does. sync.0006_boardevent_per_project_seq does three things:

  • It adds a nullable seq column to sync_boardevent. It does not rewrite any existing rows; they keep seq empty and are delivered live but never replayed.
  • It creates the per-project counter table. It seeds each project’s counter at the highest replay id that project used plus 2³². The pods still running the old release keep issuing global ids during the rollout. The offset keeps those ids, and every cursor saved before the upgrade, below the new per-project numbers.
  • It replaces the (project, id) index with a (project, seq) unique constraint.

Building that index scans the table. The whole migration runs in one transaction and holds an ACCESS EXCLUSIVE lock on sync_boardevent until it commits. The table holds at most TRUEPPM_BOARD_EVENT_RETENTION_HOURS of events (24 h by default). We measured about 0.12 s for 300,000 rows across 500 projects on a developer laptop, local PostgreSQL 16. That figure is from one machine. Expect it to scale roughly with row count, and to be slower on shared or network storage.

Near the end of the migration, the counter table’s foreign key to projects is created. That takes a SHARE ROW EXCLUSIVE lock on projects_project, held from then until the migration commits. The locks are taken in this order: sync_boardevent, then projects_project. While they are held, old pods that are still serving wait on them:

  • a board-event broadcast or a reconnect replay, for the whole migration;
  • an INSERT, UPDATE or DELETE on a project row, for the tail of the migration. This includes the sync-cursor bump each sync upload makes, and project edits. Plain SELECT … FOR UPDATE on a project row does not wait.

Each pauses for that long, then continues. Nothing fails. We found no old-pod transaction that writes a project row and then touches sync_boardevent, so a deadlock is not expected. This is not guaranteed by construction.

During a rolling update (replicaCount >= 2, as in values-prod.yaml), pods on the old and new code run side by side for a few minutes, using two different seq number spaces:

  • A browser can briefly miss live updates or apply them late.
  • A tab whose socket is still on an old pod can receive one event from a new pod. This applies to a newly loaded tab as well as an existing one. After that event, the tab ignores further events from old pods. Changes old pods make from then on are not shown until the data is refetched or the page is reloaded. Reconnecting to a new pod does not correct this on its own.
  • Events written by an old pod during the window are delivered live but are not replayable after a reconnect.

Reloading the page clears all of these. Every client that was connected across the upgrade refetches its project data once, on its first reconnect. If you script against the WebSocket API, any cursor saved before the upgrade gets one resync_required the first time you reconnect with it. See Sequence numbers and reconnect replay.

Rollback: the migration is reversible. Unapplying it drops the column and the counter table. After a rollback, reload every open browser tab.

The reload is required because a tab still holds a per-project cursor above 2³², far above every global id the rolled-back release issues. The rolled-back server neither replays to such a cursor nor resyncs it. The tab then drops every live board event as already seen, until it is reloaded. For a scripted client, discard any cursor saved since the upgrade.

Upgrading across the #4359 readiness-probe change

Section titled “Upgrading across the #4359 readiness-probe change”

Helm only — sets a floor on the web image, not just the chart. The web pod’s readiness probe moved from GET / to GET /asset-manifest.json, a build-specific file Vite writes at the html root (#4359). An image built from v0.4.0-beta.7 or earlier has no asset-manifest.json at its html root — it predates the file’s existence — so that probe path 404s unconditionally and the web pod never turns Ready. On a fresh install, this means no web endpoints join the Service at all; on an upgrade with --wait, the command hangs until it times out. There is no per-tier tag knob — both the api and web images resolve the same image.tag (trueppm.imageTag, packages/helm/templates/_helpers.tpl) — so this can only happen by explicitly overriding image.tag to v0.4.0-beta.7 or earlier on a chart version that already carries #4359’s deployment.yaml; leaving image.tag at its default (which resolves to the chart’s own appVersion) is unaffected. The only other way to hit this is pointing image.webRepository at an old web image some other way the chart does not model, such as a pinned digest or a mirror that serves a stale tag.


0.3 adds new database tables and columns for the agile-team feature set. A migration is a script that changes TruePPM’s database structure to match a new version of the code (adding a table or column, for example) — TruePPM ships them, so you never write or edit one yourself. All of the migrations in 0.3 are additive (new tables and nullable columns — no destructive operations), so the upgrade is a standard migrate run with no manual data steps and no downtime beyond that run. Every deploy path runs that step for you on startup — the Compose api / api-init service, or the Helm chart’s migrate init container — and each section below shows how to watch it complete. The new schema:

  • Forecast snapshots (scheduling.0007_projectforecastsnapshot) — a new ProjectForecastSnapshot table that persists each project’s P50/P80/P95 Monte Carlo forecast over time, so the Schedule view can show a forecast history. Retention is bounded by MC_HISTORY_CAP (see configuration).
  • Sprint outcomes (projects.0064_sprinttaskoutcome, projects.0065_historicalsprint_goal_outcome_sprint_goal_outcome) — a new SprintTaskOutcome table plus a goal_outcome column on Sprint (MET / PARTIAL / MISSED), capturing the sprint close-out snapshot.
  • Scope-change audit (projects.0054_sprintscopechange_goal_impact_and_more) — a goal_impact column on SprintScopeChange, recording whether a post-activation scope change affected the sprint goal.

If you maintain a fork, note that 0.3 also collapses each app’s migration history into a 0001_squashed_… migration via Django’s replaces= (issue #1286). Because the original migrations remain on disk and applyable, an existing database records the squashed migration as already-applied and upgrades as a no-op — there is no drop, recreate, or data step.


Terminal window
git fetch --tags && git checkout v<version>
docker compose build
docker compose up -d

<version> is the release you are moving to; the current tag is on the Releases page. Check out the release tag rather than pulling main, whose tip can be ahead of any release.

build, not pull (#3189). The dev stack’s api and web services are build:-based, and celery references image: trueppm-api:local with no build: of its own — so docker compose pull has nothing to fetch for the services that changed, and reports success. Building is what actually picks up the new code.

Migrations run automatically when the api container starts.


Terminal window
cd /opt/trueppm
git fetch --tags && git checkout v<version>
# Update the target version in .env (the bare version, no "v"; find the
# current tag on the Releases page):
# APP_VERSION=<version>
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d

The api-init service runs migrate --noinput before the API starts. Watch it complete:

Terminal window
docker compose -f docker-compose.prod.yml logs -f api-init
# Should end with: "0 unapplied migration(s)." or a list of applied migrations.

Terminal window
helm upgrade trueppm oci://ghcr.io/trueppm/charts/trueppm \
--version <version> \
--namespace trueppm \
-f my-values.yaml

<version> is the release version without a leading v; the current tag is on the Releases page. Always pass it: without it Helm resolves the newest stable chart and skips pre-release versions (--devel includes them), so an upgrade can move further than you meant.

Migrations run in a migrate init container of the api Deployment before the new pods start serving. Check its logs:

Terminal window
kubectl logs -n trueppm deployment/trueppm-api -c migrate

The Helm chart now installs secure by default: it generates the PostgreSQL and Valkey passwords and stores them in a chart-owned connection Secret (<release>-trueppm-connection, annotated helm.sh/resource-policy: keep), injects DATABASE_URL / REDIS_URL via secretKeyRef, enables Valkey auth by default, and applies restricted container security contexts. A few notes when upgrading from a pre-hardening release:

  • Rotate the old default password. Earlier chart versions shipped a default database/cache password of trueppm. If you ran with that default, rotate it. The simplest path is to set explicit, strong passwords on the upgrade so the chart writes them into the connection Secret and the bundled datastores pick them up:
    Terminal window
    helm upgrade trueppm oci://ghcr.io/trueppm/charts/trueppm \
    --namespace trueppm \
    -f my-values.yaml \
    --set postgresql.auth.password="<new-strong-password>" \
    --set valkey.auth.password="<new-strong-password>"
    Prefer supplying these through an external Secret over --set. After the rollout settles you can clear the explicit values and let the chart manage the password from the connection Secret going forward.
  • Leave the passwords blank to keep the generated ones. On an upgrade where a connection Secret already exists, leaving postgresql.auth.password / valkey.auth.password empty makes the chart read the existing password back rather than minting a new one — so re-running helm upgrade never churns the credential or orphans the database PVC.
  • The connection Secret survives helm uninstall. The resource-policy: keep annotation means an accidental uninstall/reinstall reuses the same password and keeps the existing data reachable. If you intend a clean wipe, delete the Secret and the PersistentVolumeClaims explicitly.
  • Using managed datastores? When postgresql.enabled / valkey.enabled are false, env.DATABASE_URL and env.REDIS_URL are now required — the chart fails the render if either is missing. Add them (ideally via an external Secret) before upgrading.
  • App-side auth/CSP defaults. The refresh token now rides an httpOnly Secure cookie and a strict CSP header is sent on every response. A standard deploy needs no changes: TruePPM is served from a single origin, which is what these defaults assume. Splitting the SPA and API across hostnames is not supported — see Split-origin deploys.

Migration reversibility — read this first

Section titled “Migration reversibility — read this first”

The safe rollback path depends entirely on what the upgrade’s migrations did, so classify them before you touch anything (the release’s operational change note states this):

  • Additive-only (new tables, new nullable columns, new indexes — the common case, and every 0.3 migration). The new schema is a superset of the old, so the previous image runs against it unchanged. Roll back the image/chart revision only — do not reverse the migrations and do not restore the database. The extra tables/columns sit unused until you roll forward again. The readiness probe cannot verify “additive-only” for you, so it holds the rolled-back pods out of the Service until you confirm that classification — see the caution below for the one-line opt-out.
  • Destructive or transforming (a column drop/rename, a type change, or a data backfill that rewrites rows). The old code cannot run against the new schema, and reversing the migration loses the data the new schema captured. Here a clean rollback means restore the pre-upgrade backup — a migrate reverse is not a substitute, because Django’s reverse operations recreate structure but cannot recover dropped or transformed data. This is why the pre-upgrade backup is mandatory, not optional.

A few data migrations are intentionally irreversible: migrate <app> <previous> runs, but the reverse step is a no-op or cannot rebuild what the forward step removed. They are deliberate (restoring the old state would recreate a bug or needs data that no longer exists), so the only way back across them is the pre-upgrade backup described above.

MigrationWhat the forward step doesWhat reversing does not restore
projects.0075Renumbers risk short IDs from 8-character hex to a contiguous decimal sequence per project.The original hex IDs. The decimal IDs stay valid and unique.
projects.0121Clears the three-point estimate fields on any task where they were mis-ordered (optimistic above most likely, or most likely above pessimistic), so the new check constraint can validate. The task falls back to its fixed duration.The mis-ordered values, which were already unusable.
projects.0137Strips the redundant mcp:read scope from API tokens that also carry legacy:full.The stripped scope. Stored state matches what the token already resolved to.
projects.0149Drops the retired Board Workshop Mode tables and their django_migrations rows.The tables and their rows. Restore from a backup taken before the upgrade.
Terminal window
# Restore the previous APP_VERSION in .env, then:
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d

If the migration applied schema changes, restore from the pre-upgrade backup:

Terminal window
docker compose -f docker-compose.prod.yml down
docker compose -f docker-compose.prod.yml up db -d
./scripts/restore.sh --artifact ./backups/trueppm-backup-<timestamp>.tar.gz --yes
# Then bring up the full stack at the previous version.

Concurrent migrations at replicaCount >= 2

Section titled “Concurrent migrations at replicaCount >= 2”

migrate runs as a per-pod init container, not as a pre-upgrade hook Job, so at two or more API replicas every pod runs it concurrently against one database. Django has no concurrency control of its own: each process reads django_migrations, decides the same migration is unapplied, and both run it.

From 0.4 the chart runs manage.py migrate_locked, which serializes them behind a PostgreSQL advisory lock. The losers block rather than fail, and by the time they acquire the lock the winner has finished, so their own migrate is a no-op. Advisory locks release automatically when the holder’s connection dies, so a killed init container cannot wedge the next rollout.

Nothing to configure. If a migration legitimately runs longer than ten minutes, raise the wait with --lock-timeout in the init container’s command, or scale the API to one replica for that upgrade.

Demo-mode upgrades: --wait and the seed hook timeout

Section titled “Demo-mode upgrades: --wait and the seed hook timeout”

Demo mode (demo.enabled) adds a post-install,post-upgrade hook Job that runs the same migrate_locked wait described above before re-seeding the sample project and share links. Helm does not wait for the api Deployment to become Ready before firing that hook, so on every demo install or upgrade the hook’s own migrate initContainer can genuinely contend for the advisory lock against the api Deployment’s init container — and migrate_locked’s default wait for that lock is up to ten minutes, longer than Helm’s own default --timeout of five minutes.

Always run demo installs and upgrades with an explicit wait and a longer timeout:

Terminal window
helm upgrade trueppm ./packages/helm -f packages/helm/values-demo.yaml \
--wait --timeout 15m

Without --wait, Helm does not block on the hook at all, so a slow seed races the Deployments silently. With the default --timeout, a lock wait anywhere near ten minutes can outlast Helm’s patience — but by then the api, web, and worker Deployments have already been applied, so Helm marks the release failed while the pods are already serving the new version; the hook Job itself keeps running and is only cleaned up by the next upgrade’s before-hook-creation policy.

Do not add --atomic for a demo-mode upgrade. --atomic turns a hook failure into an automatic rollback, which — per the Helm rollback skew window below, stacked on top of the stall this section describes — leaves the deployment in a worse, harder-to-diagnose state than a failed release that a human then re-runs.

Terminal window
helm rollback trueppm -n trueppm --wait --timeout 10m

Always pass --wait. Without it, helm rollback patches the three Deployments and returns exit 0 immediately — before the web rollout finishes, before the api tier’s readiness probe has run even once, and well before any of the sequence below is visible. A bare helm rollback exiting 0 is not evidence the rollback finished; --wait makes the command itself fail (timeout or error) when a target Deployment never becomes ready, instead of reporting success on a rollback that is still stuck.

Across a release that added migrations, the three tiers do not roll back together — read this before you run it:

  • Web rolls back immediately. The web tier’s readiness probe checks /asset-manifest.json (a build-specific sentinel, #4359), not any api-dependent path (packages/helm/templates/web/deployment.yaml); it does not gate on the api’s version. At replicaCount >= 2 the rollback is an ordinary rolling update in reverse, so old- and new-release web pods serve together for the few seconds the rollout takes — the same mixed-asset mechanism as a forward upgrade (see cause 4 and #4341), just now serving the older bundle once it settles.
  • The api tier stalls, by design, and stays on the newer version. Each rolled-back api pod still runs the migrate init container (migrate_locked) first. Because the init container’s migration graph is the old image’s — it has no record of migrations the newer release added — it finds nothing to apply and exits 0; this is not a crash loop. The api container then starts and /api/v1/readyz reports "migration_state": "ahead" (the database records migrations this image does not ship) and gates the pod Ready: false, because TRUEPPM_READYZ_ALLOW_DB_AHEAD defaults to false (packages/api/src/trueppm_api/settings/base.py). The chart sets no explicit strategy: on the api Deployment, so Kubernetes’ default RollingUpdate (25%/25%, rounded) gives maxUnavailable: 0 at the 2–3 replica posture values-prod.yaml ships — the rollout cannot scale down the still-Ready newer-version pods until the rolled-back replacement passes readiness, which it never does. The api tier is therefore left running the newer version indefinitely, with the rolled-back replica stuck NotReady beside it; the Deployment’s progressDeadlineSeconds (the chart does not override it, so Kubernetes’ default of 600 s applies) only flips the Progressing condition to False after ten minutes — it does not stop anything or roll back further on its own.
  • The worker has no migration gate at all and rolls back immediately. The Celery worker’s readiness probe is a heartbeat-freshness check (packages/helm/templates/celery-worker/deployment.yaml, trueppm.celeryProbe with mode: heartbeat-fresh) — it never reads migration_state. The rolled-back worker starts consuming tasks against the newer schema with nothing holding it back, which is the least visible leg of the split: nothing reports it as degraded.

The result is web on the old release, api on the new one, and worker on the old one running against the new schema — not a transient skew window, but a stable split state that --wait will time out on. Resolve it deliberately:

  1. Classify the release’s migrations using the migration reversibility rule above, from its operational change note.
  2. Confirm the api pod is gated on ahead, not behind, before touching the override — kubectl logs on the stuck pod shows the one-time TRUEPPM_READYZ_ALLOW_DB_AHEAD WARNING, or query the probe directly:
    Terminal window
    kubectl exec -n trueppm deploy/trueppm-api -- \
    curl -s localhost:8000/api/v1/readyz
    # → {"status": "fail", "checks": {...}, "migration_state": "ahead"}
  3. Additive-only: re-open the gate and let the stalled rollout complete. helm rollback takes no --set, so apply the override with helm upgrade against the same, explicit previous chart version — an unpinned --reuse-values can silently resolve back to the newest chart and undo the rollback instead of completing it:
    Terminal window
    helm upgrade trueppm oci://ghcr.io/trueppm/charts/trueppm \
    --version <previous-chart-version> -n trueppm \
    --reuse-values --set env.TRUEPPM_READYZ_ALLOW_DB_AHEAD=true \
    --wait --timeout 10m
    Once the api tier is healthy and you are ready to roll forward again, remove the override (--set env.TRUEPPM_READYZ_ALLOW_DB_AHEAD=false, or drop the key and helm upgrade --reuse-values) — leaving it set hides the next rollback’s drift behind the same green status/checks it hides this one behind.
  4. Destructive or transforming: do not set the override — forcing the api pod ready only makes it serve code that no longer matches the schema it just rejected. Follow migration reversibility above: restore the pre-upgrade backup, then roll the image back onto the restored schema. Scale the worker to zero replicas first (kubectl scale deployment/trueppm-celery-worker --replicas=0 -n trueppm) — it has no gate of its own and will otherwise keep consuming tasks against the schema you are about to replace underneath it.
  5. Verify the fleet agrees on one version before calling the rollback done — nothing in kubectl get pods summarizes the split for you today, so check the image tag per tier directly:
    Terminal window
    kubectl get pods -n trueppm -o jsonpath='{range .items[*]}{.metadata.labels.app\.kubernetes\.io\/component}{"\t"}{.spec.containers[0].image}{"\n"}{end}' | sort -u

Terminal window
# Check all containers are healthy
docker compose -f docker-compose.prod.yml ps
# or
kubectl get pods -n trueppm
# Hit the health endpoint
curl https://trueppm.example.com/api/v1/health/
# → {"status": "ok"}
# Confirm the expected version is running — check the deployed image tag
kubectl get deployment -n trueppm trueppm-api \
-o jsonpath='{.spec.template.spec.containers[0].image}'
# or: helm list -n trueppm

Reverse proxies and WebSockets (0.5). Starting with 0.5, a proxy that rewrites the Host or Origin the API sees will make WebSockets fail with 1006. Set TRUEPPM_WS_ALLOWED_ORIGINS to the origin the browser sends.


Migrations fail on startup

Check that DATABASE_URL is correct and the database is reachable. Run migrations manually to see the full traceback:

Terminal window
docker compose -f docker-compose.prod.yml exec api python manage.py migrate --noinput

Static files not updating

Trigger a collectstatic run:

Terminal window
docker compose -f docker-compose.prod.yml exec api python manage.py collectstatic --noinput --clear

WebSocket connections drop after upgrade

Expected — clients reconnect automatically within a few seconds. The Channels layer (Valkey) is not drained between upgrades; in-flight messages are lost but clients recover via the reconnect loop.