Upgrading
This page walks through moving an existing TruePPM instance to a newer version, safely, for whichever way you run it — Docker Compose, a single server, or Helm/Kubernetes. If you are standing up TruePPM for the first time rather than upgrading one, see Installation instead.
Before you upgrade
Section titled “Before you upgrade”-
Read the changelog for the target version — check
CHANGELOG.mdor the release notes for breaking changes and migration notes. -
Back up PostgreSQL with
scripts/backup.sh. Valkey state is ephemeral (broker + cache); PostgreSQL is the only stateful service. The exact invocation differs by how you run TruePPM — a bare./scripts/backup.sh --output-dir ./backupsonly works whenDATABASE_URLis already exported andpg_dumpis on the host’sPATH, which is true for neither the production Compose stack (no host database port, nopg_dumpin the application images) nor a fresh shell against Helm. See Backup & Restore for the command for your stack — Compose (development), Compose (production), or Kubernetes/Helm.Use this rather than a hand-rolled
pg_dump > file.sql. The runbook’s restore tooling reads--format=customarchives, so a plain-SQL dump is an artifactscripts/restore.shcannot consume — you would discover that during a rollback, which is the worst moment to discover it. The custom format also preservesCREATE EXTENSIONordering, which theltree/pg_trgm/btree_gistobjects depend on; see Backup & restore. -
Note your current version before starting. Read it from what is actually running, not from a
latesttag, which says nothing about which release a pinned deployment uses.Terminal window # Docker Compose: the image (and so the tag) each service runsdocker compose -f docker-compose.prod.yml ps --format '{{.Service}}: {{.Image}}'# Kubernetes: the image of the running API Deploymentkubectl get deploy -n trueppm -o jsonpath='{.items[*].spec.template.spec.containers[*].image}'# Helm: the chart and app version of the releasehelm list -n trueppm -
Check your upgrade path against Supported upgrade paths.
Supported upgrade paths
Section titled “Supported upgrade paths”Upgrade one minor version at a time: 0.3 to 0.4, then 0.4 to 0.5, and so on. Skipping a minor version in one step (for example 0.3 to 0.5) is not a supported path, because each release’s migrations and operator notes assume the previous minor’s schema and configuration. TruePPM is pre-GA, so this is guidance you should follow rather than a tested guarantee. From 1.0 on, every sequential minor upgrade (1.2 to 1.3, and so on) is a release requirement. Within a minor, the same caution applies: the web-asset notes under Upgrading to 0.4 cover an upgrade from the immediately previous published release, not one that skips a release.
Scheduled dates can change between releases only for a documented reason; the
changelog’s changed and fixed entries list them. A frozen corpus of schedules
and Monte Carlo results enforces this on every merge request.
Per-release operational change notes
Section titled “Per-release operational change notes”Every release carries a short operational change note answering one question: what does an operator have to check or change before and after this upgrade? This is distinct from the changelog (user-facing changes) — it is the operator’s pre-flight. Each release’s note appears in a versioned section on this page (see Upgrading to 0.3 below for the shape); the template used to write one is:
## Upgrading to <version>
**Migration behavior:** <additive-only / includes destructive ops / data backfill>.Downtime: <none beyond the migrate run / brief write pause / maintenance window>.
**New or changed env vars / Helm values:**- `NEW_VAR` — <what it does, default, whether action is required>- `changed.helm.value` — <old → new default, action required?>
**Breaking config:** <none / describe what an existing config must change>.
**New migrations operators will see:**- `<app>.<NNNN_name>` — <one line: what schema it adds/changes>
**Pre-upgrade action:** <back up (always) / rotate a credential / set a new value>.**Post-upgrade verification:** <what "green" looks like — see the checklist below>.**Rollback notes:** <forward-only? safe to roll back? migration-reversibility caveat>.Fill this in from the release’s changelog fragments, the diff of
packages/helm/values.yaml, and the new files under
packages/api/**/migrations/. Even an all-additive release gets a note so the
operator has a complete picture rather than inferring “nothing changed.”
Upgrading to 0.5
Section titled “Upgrading to 0.5”Breaking config: the egress allow-list is now per surface.
TRUEPPM_EGRESS_ALLOWLISTED_HOSTS exempts single sign-on only from 0.5. Before
you upgrade, move every non-SSO host out of it:
- an internal SMTP relay to
TRUEPPM_EGRESS_ALLOWLISTED_HOSTS_SMTP; - a self-hosted GitLab or Jira (Connected Accounts) to
TRUEPPM_EGRESS_ALLOWLISTED_HOSTS_INTEGRATIONS; - a webhook receiver on a private network to
TRUEPPM_EGRESS_ALLOWLISTED_HOSTS_WEBHOOKS.
A host left only on the legacy list is refused as a private address on its surface after the upgrade. Webhook deliveries to such a host fail and are parked, not retried, so a delivery attempted during the window is not replayed automatically once you fix the list. Set the new variables on the API, Celery worker, and Celery beat processes. See Networking.
Successors of weekend actuals move on recalculation
Section titled “Successors of weekend actuals move on recalculation”A non-working actual_start or actual_finish (a Saturday, Sunday, or calendar
holiday) will be read in working time in 0.5: a finish as the end of the last working
day before it, a start as the opening of the next working day, on every link type
at every lag. A zero-lag finish-to-start milestone after a weekend actual_finish
will show at the end of Friday instead of the start of Monday. Recalculating an
existing project after the upgrade will move successors of such tasks to where the
equivalent weekday actual puts them.
Offline and sync clients re-sync once (sync.0008)
Section titled “Offline and sync clients re-sync once (sync.0008)”0.5 will add a tombstone-retention floor to the delta pull (#2474). A pull whose
since is older than the floor answers resync_required: true, and the client
discards its local store and pulls again from since=0 (see
Tombstone retention and resync_required).
sync.0008_backfill_tombstone_floor raises every project’s floor, and the
program pull’s floor, just above the current cursor. Tombstones the nightly
reaper removed before the upgrade never recorded a floor, so the server cannot
tell which clients missed one; the backfill therefore asks every offline or
sync client to re-sync once after the upgrade. Expect one full pull per client
and project at the upgrade, and a short spike in sync traffic. From 0.5, deleting a
task tombstones its links, relations, recurrence rule and suggested assignees
with it, so reaping it later no longer forces a re-sync. Two cases still do until
#4951 is fixed: reaping a task deleted before the upgrade that still had live
child rows, and reaping a deleted project’s tasks.
The same migration moves every Monte Carlo run recorded at the old cursor
along with it. The cursor doubles as the forecast’s plan version, so without
that a forecast that was current before the upgrade would read “project
changed” after it. A forecast cached in the last 24 hours is moved too, on a
best-effort basis: if the cache is unreachable during migrate, that forecast
reads “project changed” until it expires or someone re-runs it.
Stop the reaper before you migrate
Section titled “Stop the reaper before you migrate”The backfill does not cover a reap that a 0.4 worker runs during the rolling update itself. Such a reap does not raise the floor, and two things can leave a client holding a deleted row afterward with no re-sync:
- The 0.4 reap’s delete cascades to live child rows of a reaped task, such as a dependency or a task link. Those rows can carry cursors from after the backfill.
- A tombstone that was saved again after it was deleted, on a model that ages on its deletion date (tasks, dependencies and task relations), keeps its old deletion date, so it is still eligible, but its cursor was drawn at that later save. If that save came after the backfill, its cursor sits above the floor. Risks and sprints age on their last update, so a re-save renews them.
The reap runs on the beat schedule (sync.reap_domain_tombstones, daily at
03:30 UTC). To close the window, stop the scheduler before the migration runs
and let the upgrade start it again:
- Helm:
kubectl scale deployment -n <ns> -l app.kubernetes.io/component=celery-beat,app.kubernetes.io/instance=<release> --replicas=0, thenhelm upgrade. The upgrade restores the replica count. - Docker Compose:
docker compose stop celery-beat, then pull anddocker compose up -d.
Alternatively, schedule the upgrade away from 03:30 UTC, or roll the workers
to the new image before the API pods run migrate. A 0.5 worker raises the
floor itself.
Rollback notes: the migration’s reverse is a no-op. Rolling back leaves the raised floors in place, which 0.4 never reads.
Sync index rebuilds lock tables (access.0021, projects.0167, integrations.0016, timetracking.0005)
Section titled “Sync index rebuilds lock tables (access.0021, projects.0167, integrations.0016, timetracking.0005)”0.5 will rebuild 14 delta-pull indexes so the keyset resume can seek on
(owner, sync_seq, id) (#2510). Each migration drops the old index and recreates
it with a plain, non-concurrent CREATE INDEX. Each migration is atomic and drops
all of its old indexes before it builds the new ones, so every table it touches
is locked against reads and writes until that migration’s last index is
built, not just while its own index builds. The migrations run on container
start, so a rolling deploy’s old pods also wait on the lock.
| Migration | Tables |
|---|---|
access.0021 | access_project_membership, access_program_membership, plus two drop-only tables: access_user_defined_mention_group, access_program_user_defined_mention_group |
projects.0167 | projects_task, projects_dependency, projects_task_relation, projects_label, projects_risk, projects_sprint, projects_sprintretro, projects_retroactionitem, projects_taskrecurrencerule, projects_tasksuggestedassignee, plus two drop-only tables: projects_pulseresponse, projects_retroboarditem |
integrations.0016 | integrations_tasklink |
timetracking.0005 | timetracking_time_entry |
Five of these are on the tables that grow with project size: task_proj_syncseq_idx
on projects_task, and the indexes on time entries, task links, dependencies and
task relations. Build time grows with their row counts. We have not measured a
per-million-rows figure; to size the window, time the upgrade on a restored copy
of your database first, and schedule it away from peak edit traffic on a large
install.
Rollback notes: the migrations are reversible, but reversing them rebuilds the old indexes the same way and takes the same locks.
Demo compose stack generates its own secrets
Section titled “Demo compose stack generates its own secrets”docker-compose.demo.yml will stop baking a shared SECRET_KEY and
INTEGRATION_ENCRYPTION_KEY in 0.5. A new secrets-init service will generate
them on first boot into a demo_secrets volume. No action will be needed on
an existing demo install: its first up after the upgrade will create the
secrets, and the demo holds no data that depends on the old keys. The Postgres
password is unchanged, so your existing database volume will keep working. If you
ran the demo stack on a reachable host before 0.5, treat the old public key as
exposed and restart it on 0.5 so the key is replaced.
Finish-to-Finish and Start-to-Finish links into summary tasks are retired (projects.0175)
Section titled “Finish-to-Finish and Start-to-Finish links into summary tasks are retired (projects.0175)”From 0.5, the scheduler will refuse a Finish-to-Finish or Start-to-Finish link whose
successor is a summary task (ADR-0370): it would force every subtask, even a short
unrelated one, to finish with the predecessor, and the engine cannot model it. Plans
written before that rule can already hold such a link, and an unmodified one would make
every recalculation of that project fail. The migration will therefore soft-delete
exactly those links: a summary here means a task with at least one live child that the
scheduler schedules (not a recurring template, epic, or backlog item). A link into a
parent whose only children are backlog, recurring, or epic items is accepted by the
engine and is left alone. The soft-delete will set the same tombstone fields a user
delete sets, so offline clients will receive the removal on their next sync, but it will
write no history row, and a stale offline edit to a retired link will get a 409
conflict. It will write one retired ... link ... into summary task ... warning per
link to the migration log, naming the project, the link and both tasks, so you can
redraw it between subtasks. It will touch no other link type, no link out of a summary
that was already legal, and nothing else, and it will be safe to run more than once.
Read those lines after the upgrade if a plan shows a dependency missing from a phase.
A link whose backlog child is later committed (#5046) is not covered by the migration.
New API tokens carry api:full (projects.0170)
Section titled “New API tokens carry api:full (projects.0170)”From 0.5, a token minted without scopes will carry ["api:full"] instead of
["legacy:full"]. Both names mean the same scope, existing tokens keep
legacy:full and keep working, and no data is rewritten. The migration only
changes the allowed values and runs no SQL against the token table.
Two windows cannot understand the new name, because a 0.4 API pod only knows
legacy:full:
- During a rolling update, a full-access token minted on a 0.5 pod is refused
(
401on the general API,403on task sync) by any 0.4 pod still serving. It works once the rollout completes. To avoid this, do not mint tokens until every API pod runs 0.5. - After a rollback to 0.4, every token minted on 0.5 without an explicit
scope stops working until you re-mint it on 0.4. Tokens minted with
{"scopes": ["legacy:full"]}are unaffected.
Upgrading to 0.4
Section titled “Upgrading to 0.4”Migration behavior: includes destructive ops (see below). Downtime: a
maintenance window sized to your task count, because projects.0148 blocks all
reads and writes on the task table while it builds a constraint (see the
projects.0148 caution below), plus the rest of the migrate run and the
transient-500 rollout windows described per-migration below on multi-replica
installs.
Going from 0.3 to 0.4 (the 0.4 line began at v0.4.0-beta.1 on 2026-09-15 and
now continues through its release candidates) runs five migrations that
contain a RemoveField, DeleteModel, or raw DROP TABLE:
profiles.0008, webhooks.0009, projects.0123, projects.0149 and
sso.0002. Four of them (all but sso.0002) can break requests that land on an
old pod during a rolling deploy, and one of them (projects.0149) permanently
deletes data. These are destructive in a way an operator needs to plan around,
not just a schema-hygiene detail. If you are upgrading from 0.3, read all five
entries below before you run migrate, in particular the workshop-data
warning:
Blank pages during an upgrade of the web tier. Each release’s index.html
names that release’s content-hashed /assets/ files, so a browser tab opened
before an upgrade asks the new pods for the old release’s files the next time
it opens a page it has not loaded yet. Releases after 0.4.0-beta.7 keep the
previous release’s /assets/ files in the web image, so those tabs keep
working on any number of web replicas. The limits:
-
One release back. Covered: an upgrade from the immediately previous published release, not counting a withdrawn one. 0.4.0-beta.7 was withdrawn, so an upgrade from 0.4.0-beta.6 is covered. Not covered: an upgrade that skips a release. Tabs opened before it reload once, automatically, when they next need a file.
-
During the rollout, the other direction. Old and new web pods serve together until the rollout finishes, and a browser that got the new
index.htmlcan ask an old pod for a new file, which answers404. With two or more web replicas this lasts for the whole web rollout. On a single replica it lasts well under a second, but it still opens, because Kubernetes’ default rolling update starts the new pod before it stops the old one. Both ends of a page load recover on their own:- A page the app opens after it has loaded (a lazy route) retries the file twice, over about two seconds, before it reloads.
- The first load of the page, where the app’s own entry script or one of the files it needs to start is the one that fails, reloads after 1, 3, 8 and 15 seconds until the page starts. If it still has not started after the fourth reload, it shows TruePPM is being updated with a Reload button instead of a blank page.
Neither one retries while the browser is offline. An offline browser sees You appear to be offline with the same Reload button instead.
If you would rather have no skew at all, set the web tier’s rollout strategy (
rollout.web, see Helm values).strategy: Recreate, ormaxSurge: 0on a single web pod, stops every old pod before a new one starts. That removes the window, but it turns every web rollout into a full outage of the UI for as long as the new pod takes to become ready, and with one pod, a node drain is an outage too. The default rolling update keeps the UI up and lets the client recover from the window. -
Upgrading to 0.4.0-beta.7 or earlier has neither; a browser can show a blank page until it reloads.
-
The first upgrade to a release that adds the first-load recovery does not fully have it during its own rollout. The recovery is a separate file,
/boot-guard.js, which the release you are upgrading from does not have. A browser that gets the newindex.htmland then asks an old pod for that file starts without it, and behaves as before for that one page load: a blank page until it reloads. Every later upgrade is covered, because both releases then carry the file. -
Upgrading from 0.4.0-beta.7, specifically. This one hop is not covered by either mitigation above, even on a release that ships both: a beta.7 web pod predates the nginx rule that answers a missing hashed asset with a clean
404and the client-sidevite:preloadErrorreload handler, so during the rollout it answers a request for the new release’s chunk with its SPA fallback page instead — and a browser that already has the newindex.htmlgets no clean error to retry on. If your web tier runs more than one replica, scale it to 1 before starting an upgrade from 0.4.0-beta.7 (helm upgrade … --set replicas.web=1with your other-f/--setflags unchanged, orkubectl scale deployment/<release>-trueppm-web --replicas=1), wait for the rollout to finish, then scale back up. Every laterbeta.N→beta.(N+1)hop is covered automatically once both tags carry the fix; this is the one upgrade where it is not.
See cause 4 under The browser shows a blank page and #4341.
Known transient-500 windows on multi-replica installs. Four of the five
migrations drop a column or table outright, with no intervening
null=True-then-remove deprecation release, so an old pod’s code still names
what the new schema no longer has. If you run the API at replicaCount >= 2
(the posture values-prod.yaml sets by
default, and the posture 0.4’s basic-HA carve-out encourages), Kubernetes’
default RollingUpdate keeps an old pod serving while the new pod’s migration
applies — for the duration of that rollout, each of the following reads or
writes will get a transient 500 from any request landing on an old pod:
profiles.0008_remove_userprofile_schedule_in_deliver— drops the retiredschedule_in_delivercolumn outright (ADR-0942 §3, #3137). Affects the old pod’sget_profile_prefs()read, used by/auth/me/(the web shell’s bootstrap call). See #3782 for the full analysis.webhooks.0009_encrypt_webhook_secret_and_failure_counters— encryptsWebhook.secretinto a newsecret_ciphertextcolumn and drops the plaintextsecretcolumn in the same migration (#2885). On the old pod,webhooks/models.pystill declaressecretas a concrete, non-deferredCharField, so Django selects it on everyWebhookquery — every webhook read and every outbound delivery attempt from an old pod fails withProgrammingError: column webhooks_webhook.secret does not existfor the duration of the window. This silently stops the entire outbound-webhook subsystem with no operator-facing warning; if you depend on webhook deliveries during the rollout, scale to a single replica for the upgrade (see the mitigation below).projects.0123_remove_historicalproject_agile_features_and_more— dropsProject.agile_features/HistoricalProject.agile_featuresoutright (#2025), with no deprecation window and no migration docstring.ProjectandHistoricalProjectreads happen on nearly every authenticated request path, so this has the broadest blast radius of the four — but no data-loss concern: the field is re-exposed as a derived, read-onlySerializerMethodField(ProjectSerializer.get_agile_features), so this is dead-column removal, not a feature regression. What is missing is purely the HA-safety disclosure this entry now provides.projects.0149_drop_workshops_tables— drops theworkshops_sessionandworkshops_participanttables. A 0.3 pod’s code still serves Board Workshop Mode, so its/api/v1/projects/{id}/workshop/*requests fail withrelation "workshops_session" does not existfor the duration of the window. Nothing else reads those tables, so the blast radius is limited to workshop requests; the data loss itself is covered in the danger box above.
sso.0002_remove_oidcprovider_workspace_ssoproviderpolicy_and_more also drops
tables, but it has no window: it first shipped in v0.4.0-beta.1 together with
sso.0001, so no earlier release has code that reads the tables it removes.
This is a known, bounded, and accepted trade-off for each of the four, not a
regression to report: no crash-loop, no data loss beyond what each entry
states, and every window closes as soon as the rollout finishes.
Single-replica installs never see any of this. If you are on
replicaCount >= 2 and want to avoid the windows entirely, scale to one
replica (kubectl scale deployment/trueppm-api -n trueppm --replicas=1)
before the upgrade and scale back up once the rollout completes.
Already running v0.4.0-beta.1 or later? These migrations already ran —
there is nothing to re-run and no remediation for the schema changes
themselves. The one exception is workshop data: if you upgraded from 0.3
without exporting it first, it is already gone (see the danger box above) —
this section exists so the next self-hoster upgrading from 0.3 does not lose
theirs the same way.
New migrations operators will see:
profiles.0008_remove_userprofile_schedule_in_deliver— drops the retiredschedule_in_deliverboolean preference. See above.webhooks.0009_encrypt_webhook_secret_and_failure_counters— encrypts webhook signing secrets and adds failure-tracking columns. See above.projects.0149_drop_workshops_tables— drops the two tables backing the removed Board Workshop Mode. Destructive, irreversible. See above.projects.0123_remove_historicalproject_agile_features_and_more— drops the deadagile_featurescolumn (now derived). See above.sso.0002_remove_oidcprovider_workspace_ssoproviderpolicy_and_more— migrates anyOIDCProvider/OIDCIdentityrows toallauth’sSocialApp/SocialAccount(ADR-0517), then drops the bespokeOIDCProviderandOIDCIdentitymodels. Benign: SSO never shipped in a tagged 0.3 release, so no installation has rows to migrate or lose — no disclosure needed beyond this line.
Breaking API changes: 0.4 also removes several /api/v1/ paths that
appeared in the published v0.3.0-alpha.3 schema. These are documented as
deliberate deprecation-window exceptions in API Stability & Deprecation
Policy — check there for the full
reasoning and migration guidance for each. If you have an integration calling
/api/v1/projects/{id}/workshop/*, /api/v1/tasks/{id}/scope/,
/api/v1/teams/{id}/, or /api/v1/projects/{id}/history/summary/, read that
section before upgrading.
Pre-upgrade action: back up PostgreSQL (always — see Before you
upgrade); if you have workshop data, export or preserve
it before running migrate (see the danger box above); if you cannot tolerate
a transient-500 window on webhook deliveries, /auth/me/, project reads or
workshop endpoints during rollout, scale to one replica first.
Post-upgrade verification: see the checklist below.
Rollback notes: all five migrations are destructive or table-dropping —
per migration reversibility, a
clean rollback means restoring the pre-upgrade backup, not a migrate reverse.
projects.0149 in particular cannot be reversed by any means other than
restore — its reverse_sql is a no-op by design.
Helm: the api and web pods get a default-deny ingress NetworkPolicy
Section titled “Helm: the api and web pods get a default-deny ingress NetworkPolicy”Upgrading the Helm chart from 0.4.0-beta.3 or earlier adds default-deny
ingress NetworkPolicies to the api, web, celery-worker, and celery-beat
pods. The api and web policies admit traffic only from the ingress controller
named by networkPolicy.ingressControllerSelector. By default, that is a
namespace called ingress-nginx.
If your controller runs anywhere else and your CNI enforces NetworkPolicy, those
policies would cut off all traffic to the site, and every pod would still report
Ready. k3s (Traefik) and RKE2 (rke2-ingress-nginx) both run their controller in
kube-system and enforce policies out of the box — and a tunnel such as
Cloudflare Tunnel is not an ingress controller at all, so it never matches the
default no matter which namespace it runs in. To prevent a silent outage, the
first upgrade that adds these policies refuses to render until you choose one
of the following:
# Your controller really is in the ingress-nginx namespace:helm upgrade trueppm ... --set networkPolicy.ingressControllerConfirmed=true
# It runs elsewhere, for example kube-system (or wherever your tunnel client runs):helm upgrade trueppm ... --set-json \ 'networkPolicy.ingressControllerSelector={"namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"kube-system"}},"podSelector":{}}'
# Defer the policies (you can enable them later):helm upgrade trueppm ... --set networkPolicy.enabled=falseThe check above runs only on the upgrade that introduces the policies — fresh
installs and later upgrades skip it, because it compares against a policy that
must already exist for there to be a “transition” at all. A separate check
fires on a fresh install too, for the two paths that bypass any in-cluster
ingress controller entirely: demo.enabled (the documented Cloudflare Tunnel
exposure) and service.web.type: LoadBalancer/NodePort. Neither has a prior
policy to compare against, so the render simply refuses while the selector is
still the default — see Public read-only demo
mode. A cloud load
balancer that sends traffic from outside the cluster needs an ipBlock peer
instead; see
networkPolicy.ingressControllerSelector.
To upgrade a release you installed with --set flags without retyping them,
start from what Helm recorded at install time:
helm -n <namespace> get values <release> -o yaml > current-values.yamlhelm upgrade <release> oci://ghcr.io/trueppm/charts/trueppm --version <version> \ -n <namespace> -f current-values.yamlThis keeps your own settings and picks up the new chart’s defaults.
--reuse-values does not: it also freezes the previous chart’s defaults.
Three more things to check for this upgrade:
- In-cluster monitoring. If Prometheus or a Blackbox exporter in another
namespace scrapes the api Service directly (the health endpoints described in
Observability), set
networkPolicy.monitoringSelectorto that namespace. Otherwise the new policy drops those scrapes, and the dead-letter and beat-staleness alerts go quiet instead of firing. Scrapes that go through your public hostname pass through the ingress controller and are unaffected. - Previews. A client-side
helm upgrade --dry-runcannot see the cluster, so it always reports this refusal. Use--dry-run=serverto preview the upgrade accurately. - GitOps. Tools that render the chart with
helm templateand apply the result themselves, such as Argo CD, never run this check. Set the selector before you sync.
Helm: seven values keys moved into the concern-first maps
Section titled “Helm: seven values keys moved into the concern-first maps”Upgrading the Helm chart from 0.4.0-beta.6 or earlier moves seven values keys
that 0.4 prereleases introduced. The chart lays out its values concern first, then
tier (resources.api, probes.worker, lifecycle.web), and these seven went the
other way. They move before 0.4.0, because the schema rejects unknown keys and any
later rename would break helm upgrade for everyone who had set them
(ADR-1248).
| Old key (0.4 prereleases) | New key |
|---|---|
web.replicaCount | replicas.web |
web.service.type | service.web.type |
web.service.port | service.web.port |
api.workers | processes.api.workers |
celeryWorker.concurrency | processes.worker.concurrency |
celeryWorker.maxTasksPerChild | processes.worker.maxTasksPerChild |
celeryWorker.extraArgs | processes.worker.extraArgs |
The defaults and the behavior do not change, only where the key lives. If your
values file or --set flags still use an old key, the render refuses and names
it. The wording depends on your Helm version:
# Helm 3.14 and similar(root): Additional property celeryWorker is not allowedweb: Additional property replicaCount is not allowed
# Newer Helm releases- at '': additional properties 'celeryWorker' not allowed- at '/web': additional properties 'replicaCount' not allowedNothing is silently dropped. Rename the key and run the upgrade again. As a YAML diff:
# Before (0.4.0-beta.6 and earlier) # Afterapi: processes: workers: 4 api:celeryWorker: workers: 4 concurrency: 4 worker:web: concurrency: 4 replicaCount: 2 replicas: service: web: 2 type: LoadBalancer service: web: type: LoadBalancerEvery 0.3 key keeps its name and meaning. The top-level replicaCount still
sets the API and worker replicas, service.type/service.port still describe
the API Service, and a values file written for 0.3 renders unchanged. Keys that
exist only for the web tier’s nginx stay under web:: web.enabled,
web.containerPort, web.maxBodySize, web.adminAccess.*, and
web.securityHeaders.*. The api: and celeryWorker: roots no longer exist.
Milestone date shift after recalculation
Section titled “Milestone date shift after recalculation”Upgrading from 0.4.0-beta.4 or earlier changes how the scheduler places
zero-duration milestones (#4079). No migration runs and no stored date changes
during the upgrade itself; the new dates appear the next time each project’s
schedule recalculates, which happens on the next edit that affects the schedule.
Earlier versions gave every milestone a working day of its own, so each milestone on a path pushed everything after it back by one working day. The scheduler now follows the MS Project and Primavera P6 convention: a milestone is a point in time. A milestone that follows work sits on the day that work finishes, and the next task starts on the following working day, exactly as if the milestone were not there. A milestone with no predecessor, such as a project-start milestone, sits at the start of its day and does not delay the work after it.
What you will see after recalculation:
- Dates move earlier. Downstream tasks, forecasts, and the project finish move earlier by one working day for every milestone on their path. A milestone that follows work is now shown on its predecessor’s finish day rather than the day after it.
- Float and the critical path can change. A milestone no longer adds a day to the paths it sits on, so float on parallel paths is recomputed against the shorter path.
- Monte Carlo forecasts move with the schedule. P50/P80/P95 use the same convention, so a plan with fixed durations still simulates to its CPM finish.
- Imported MS Project plans now match their source file. Recalculated dates
for an imported plan no longer drift one day per milestone from the dates in
the
.xmlfile.
Baselines captured before the upgrade keep the old dates, so the first comparison after recalculation can show milestone-driven variance that reflects the convention change, not a schedule change. To compare like with like, capture a new baseline after the project recalculates.
Data dates and SNETs before the project start
Section titled “Data dates and SNETs before the project start”Upgrading from 0.4.0-rc.1 or earlier changes how the scheduler treats a data
date (status date), or a task’s start-no-earlier-than date (SNET), that falls on
or before the project start (#4635).
No migration runs and no stored date changes during the upgrade itself. The new
dates appear the next time each project’s schedule recalculates, which happens on
the next edit that affects the schedule.
A project with no status date uses today as its data date, so this mostly affects projects that start in the future. Earlier versions measured a lag out of a milestone with no predecessor, such as a kickoff milestone, from that earlier data date rather than from the project start. The same happened with any milestone whose SNET is before the start, including a completed milestone that has no recorded dates and carries such an SNET. The work after such a milestone lost up to the gap between today and the project start from every lag. A data date or SNET on or before the project start now holds nothing back, except that an SNET still holds a task that has a Start-to-Finish link.
What you will see after recalculation:
- Dates after a kickoff milestone move later. On a project that starts in the future, the successors of a milestone with no predecessor regain their full lag. Their dates, the forecast finish, and the critical path move later by up to the gap between today and the project start.
- Some milestone successors move earlier. A milestone with no Start-to-Finish
link that a lead places before the project start used to be held at a data date
or SNET that fell between that earlier point and the start, and its successors
measured their lags from there. That bound is now ignored, so they measure from
where the links place the milestone. For example, with
P (3 days) -FS-10d-> M -FS+5d-> Xand the project starting Monday 02-02, a status date of 01-30 used to startXon 02-04; it now starts on 02-02, as with no status date. - Some Start-to-Finish tasks move earlier. A task with a Start-to-Finish link that its link places before the project start is no longer held at a data date that is on or before the start. It moves to where its links place it, unless its own SNET holds it. On a project that is already underway, the data date still holds it.
- Monte Carlo forecasts move with the schedule. P50/P80/P95 use the same rule.
What the Schedule’s Snap to project start and Move project start options do for a task before the project start now depends on its links. Both write a start-no-earlier-than date (SNET) on or before the (possibly moved) start:
- A task with a Start-to-Finish link is held there. Its links alone would put it earlier, and the SNET is the only thing holding it, so the engine keeps it.
- A milestone with no Start-to-Finish link stays where it is shown, but its
successors follow its links. A lead from its predecessor places it before the
start. The milestone itself is still shown on the project start, or the new
start, as before, but the lags out of it are measured from where its links place
it, not from the SNET. For example, with
P (3 days) -FS-10d-> M -FS+5d-> Xfrom 02-02, snappingMto the start used to startXon 02-09 and now starts it on 02-02. Moving the start to 01-26 used to startXon 02-02 and now starts it on 01-26. The old behavior was the defect this release fixes: a pre-start SNET on such a milestone pulled its successors in a way that also depended on today’s date. The milestone’s placement comes from its links, so its SNET carries no information about when its successors can start.
Baselines and saved forecast snapshots keep the old dates, so the first comparison after recalculation shows a one-time step that reflects the rule change, not a schedule change. To compare like with like, capture a new baseline after the project recalculates.
Completed milestones after the first recalculation
Section titled “Completed milestones after the first recalculation”The scheduler now keeps a completed milestone in place no matter how the open work before it changes (#4525). On each recalculation it reads back which edge of its day (start or end) the last schedule placed the milestone on. The first recalculation after upgrading reads what the previous version saved, so for one recalculation you may see the following:
- A milestone that was In Progress before it was completed moves to its completion day. The row used to show the day it entered In Progress. It now shows the day it was signed off, which is where the scheduler already measured the work after it from.
- Work after a completed milestone can move by one working day, once. This happens only when a task that leads into the milestone is still open. The scheduler keeps the edge the previous version saved, or, if that was saved under the old rule, works it out again from completed work only. Either way it happens once and then stays put. When every task that leads into the milestone is complete, the completed work decides. Neither case needs any action.
Baselines are not recalculated. If you captured a baseline after such a milestone was completed, the baseline still holds the day it entered In Progress, while the current schedule shows the day it was signed off. The first variance comparison after upgrading therefore shows a slip of the gap between those two days. That is not new slip. To compare like with like, capture a new baseline after the project recalculates.
Project and program keys (projects.0154)
Section titled “Project and program keys (projects.0154)”Upgrading from 0.4.0-beta.4 or earlier makes every project and program key
(the code field; see Project and program keys)
unique across the workspace, compared case-insensitively. Migration
projects.0154_object_keys repairs existing data before it adds the uniqueness
constraint:
- A project or program with no code gets one derived from its name. Platform
Migration becomes
PM, and a program called Office Move becomesoffice-move. - When several projects (or several programs) share a code, the oldest keeps
it and each of the others gets a numeric suffix:
PLAT2,PLAT3for projects,atlas-2for programs. - Every other code is kept exactly as it is, including an existing hyphenated
project code such as
GA-SEC. New project keys can’t contain hyphens, but existing ones are never rewritten.
Each code the migration changed is logged at WARNING during migrate with the
prefix key repair (ADR-1237). The log is optional: the database records every
rewrite too. To list them later, run this in python manage.py shell:
from trueppm_api.apps.projects.models import ObjectKey
for row in ObjectKey.objects.filter(source="backfill").select_related("project", "program"): owner = row.project or row.program print(row.kind, row.key, owner.name if owner else "(deleted)")If a rewritten key isn’t what your team wants, rename it on the project’s or program’s General settings page. The old key keeps working in links and now opens the renamed project.
Rolling upgrades. The constraint excludes blank codes for this release. While an older API pod is still running during a rolling update, it can create a project with an empty code without failing. The new version gives that project a key the next time it is edited. The exclusion is planned to be removed in 0.5.
created_by is an integer user id on share links, stakeholders, and saved board views
Section titled “created_by is an integer user id on share links, stakeholders, and saved board views”Upgrading from 0.4.0-beta.6 or earlier changes one response field on three API
surfaces. This affects only API clients and scripts; the web UI is updated with
the server, and no data or migration is involved.
- Share links (
/api/v1/projects/{id}/share-links/) and external stakeholders (/api/v1/programs/{id}/external-stakeholders/):created_bywas the creator’s display name. It is now the creator’s integer user id, and the name moves to a newcreated_by_namefield. - Saved board views (
/api/v1/projects/{id}/board-views/):created_bywas the creator’s user id as a string, such as"7". It is now the integer7.
If a script reads either field, update it before upgrading. See the API stability policy for why this change ships between betas.
Dead-letter and observability scrape credentials now require superuser
Section titled “Dead-letter and observability scrape credentials now require superuser”Upgrading from 0.4.0-beta.4 or earlier changes who can reach these endpoints.
Through 0.4.0-beta.4, FailedTaskViewSet (dead-letter requeue/drop/list) and
the observability app’s System Health, Prometheus metrics
(/api/v1/health/beat/, /api/v1/health/dead-letter/, /api/v1/health/email/),
retention policy, and telemetry-export endpoints were gated with Django’s
IsAdminUser, which passes for any is_staff=True account — including one
that is not a superuser. They now use IsWorkspaceOperator, which checks
is_superuser directly (#4009). Superusers are unaffected; a WorkspaceRole.ADMIN
membership was never sufficient for these endpoints and still is not.
This is a least-privilege regression for anyone who provisioned a staff-only
service account for scraping or automation against these endpoints — it now
gets 403 Forbidden. Before upgrading:
- Identify any Prometheus scrape job, alerting script, or automation that calls
/api/v1/health/{beat,dead-letter,email}/, the retention endpoints, or the dead-letter queue admin API. - Confirm the JWT it uses belongs to a superuser account (
is_superuser=True), not merely a Django-admin (is_staff=True) account.create_adminproduces a superuser and is unaffected. - If it does not, mint a new token from a superuser account before or immediately after the upgrade, or the scrape/automation starts failing silently (a gauge that stops updating, not a loud error) the moment the new code is live.
See Dead-letter Alerting and Beat Liveness for the affected scrape configs.
Program exports and program and project webhooks honor project membership
Section titled “Program exports and program and project webhooks honor project membership”Behavior change, no migration. Upgrading from 0.4.0-beta.6 or earlier: a program
export, the program webhooks of a Program Admin, and a program-scoped API token used to
reach every member project regardless of the minting admin’s own project membership, and
a project webhook kept delivering after its creator lost access to that project. What
changed:
-
Program export (JSON seed and async bundle). Only member projects the requesting user holds project membership on are included. A Program Admin who is not a member of a project no longer receives that project’s tasks, attachments, time entries, history, or MS Project XML from a program export. Program-level content (roster, program backlog, ceremonies) is still exported. A scheduled or queued bundle is evaluated when it is built, not when it was requested. A bundle can be downloaded only by the admin who requested it, and an admin’s export request no longer returns another admin’s job that is still in flight.
-
Program-scoped webhooks. A program webhook receives a member project’s events only while its creator (
created_by) holds Project Manager or above on the specific project the event fires from — a program grant has never been a project-read grant (#4310) — and holds Program Manager or above on the program itself (#4330). The second check matches the delivery-log read gate exactly (IsProgramAdminplus the creator-only check), which is keyed on the registrant’s program role, not their project role; the first check has no equivalent in the read gate and exists purely because dispatch, unlike reading the log, hands a specific member project’s data to someone who may hold no role on that project at all. A creator demoted to Resource Manager, Team Member, or Viewer on the firing project, or to anything below Program Manager on the program, stops receiving events even though they are still a live member in both places — either demotion alone is enough to stop delivery. Removing either membership entirely has the same effect. Events already queued, or retrying, are still delivered. A webhook whose creator account was deleted, or is deactivated, delivers no member-project events. Editing (PATCH/PUT) requires the caller to currently hold Program Manager or above and be the webhook’s creator and still hold live membership (any role — presence, not role, is what this leg checks) on every project the program covers; deleting requires only that the caller currently hold Program Manager or above, with no creator or project-membership check at all, so any current Program Admin can delete it, not only its creator. The practical consequence: a creator demoted below Program Manager on the program loses the ability to edit or delete the webhook themselves — the same demotion that stops dispatch on the program axis also removes their only path to self-service disable it, and only a different Program Admin can intervene (by deleting it; nobody but the original creator can everPATCHit, since editing’s creator check cannot be satisfied by anyone else). A demotion on the project axis alone — the creator stays Program Manager+ but drops below Project Manager on a covered project while remaining a live member there — does not block edit or delete at all, because neither checks project role, only project membership presence for edit and nothing for delete; it only stops that project’s deliveries. Only a full removal from a covered project (not a demotion) blocks editing, via the presence check — deleting still works even then, since delete does not consult project membership. If the creator has been fully removed from any project in the program, including one added later, nobody can edit the webhook, but any Program Admin can still delete it and it can be re-created under a qualifying account. Only the creator can read its delivery log. -
Program-scoped API tokens (task-sync and acceptance-results). These two write endpoints — inbound task-sync and CI acceptance-result ingest — now also require the token’s minter (
created_by) to hold project membership on the target project, checked on every request, not just at mint time. Revoking the minter’s membership stops the token from writing into that project on the very next request, with no re-mint needed. A token whose minter was deleted (and so carries nocreated_by) can no longer write into any project through this path. A further role floor sits on top of this membership check: the minter must also hold Member or above, not merely live membership, so a Viewer-minted program token is refused on both write endpoints (#4331). -
Project-scoped API tokens (task-sync and acceptance-results). These two write endpoints apply a request-time floor to a project-scoped token’s minter, based on the minter’s last recorded role on the token’s project, live or removed. A minter demoted to Viewer loses write authority on the token’s very next request. A minter removed from the project while still Member or above keeps a working token, because project-scoped tokens are org assets that survive their minter’s off-boarding. A minter demoted below Member and then removed, including one who removes themselves, stays refused (#4334). (A token whose minter’s account was deleted is a separate case: both endpoints refuse it, because it is left with no user to act as.)
-
Project-scoped webhooks. A project webhook receives that project’s events only while its creator holds Project Manager or above there, checked on the same schedule as program webhooks above. Removing the creator’s membership, demoting it below Project Manager, or deleting or deactivating their account, all stop new deliveries; events already queued, or retrying, are still delivered. Editing a project webhook, deleting it, and reading its delivery log were not affected — those already require Project Manager or above and re-check the caller’s current membership and role on every request, so a removed or demoted creator already lost those abilities today.
Operator check after the upgrade. Program webhooks now showing no deliveries for a
member project usually belong to a creator without membership there, or one demoted
below Project Manager on that project while remaining a member (Resource Manager, Team
Member, or Viewer) — or, independently, one demoted below Program Manager on the program
itself while still Project Manager on every covered project. Either restore the
creator’s role (project-level, program-level, or both, depending on which check failed),
or re-create the subscription under a current qualifying account. Whether the creator
can self-service it depends on which axis demoted them. If only the project role
dropped (the creator is still Program Manager+), they can still edit or delete the
webhook themselves — editing’s project-membership leg checks live presence, not role,
so a demotion (as opposed to a full removal) on the project side never blocks it, and
deleting does not consult project membership at all. If the program role dropped below
Program Manager, the creator loses both abilities at once: editing and deleting a
program-scoped webhook both require the caller to currently hold Program Manager or
above on the program, with no exception for the webhook’s own creator. In that case the
creator cannot disable it themselves at all — restoring their program role, or having a
different current Program Admin delete the subscription (delete has no creator
check, unlike edit), is the only remedy; nobody but the original creator can ever edit
it, since editing’s creator check cannot be satisfied by anyone else, demoted or not. A
full removal from a covered project (not merely a demotion) blocks editing via the
presence check even with Program Manager intact, but a still-Program-Manager creator can
still delete it in that case too, since delete never checks project membership. A project webhook showing
no deliveries belongs to a creator removed from that project, or demoted below Project
Manager on it; re-add or re-promote them, or have a different, still-qualifying Project
Manager/Admin disable or re-create the subscription — unlike the program case, a project
webhook’s own edit/delete already requires Project Manager or above (matching the
dispatch and delivery-log gates), so a demoted creator loses the ability to disable their
own stale webhook too, and cannot self-service it back to health. A test ping is not a
reliable health check here: it is gated on the caller’s own current role, not the
registrant’s membership or role, so a current Admin’s test ping still succeeds against a
dormant webhook and shows up in the delivery log looking healthy while real events
are silently dropped. A program-scoped integration token that starts returning 401 on
task-sync or acceptance-results usually means its minter lost project membership (on 0.5 and later, also program
membership, or was demoted below Member in the program); add
them back, or have a Program Admin (or above) re-mint the token. The
manage.py export_program command is unaffected: it runs as the operator with shell
access and exports every member project.
WebSocket replay seq becomes per-project (sync.0006)
Section titled “WebSocket replay seq becomes per-project (sync.0006)”The seq on a real-time board event is now a per-project, dense counter instead of
the replay buffer’s global row id (#4386). Operators have nothing to configure.
What the migration does. sync.0006_boardevent_per_project_seq does three things:
- It adds a nullable
seqcolumn tosync_boardevent. It does not rewrite any existing rows; they keepseqempty and are delivered live but never replayed. - It creates the per-project counter table. It seeds each project’s counter at the highest replay id that project used plus 2³². The pods still running the old release keep issuing global ids during the rollout. The offset keeps those ids, and every cursor saved before the upgrade, below the new per-project numbers.
- It replaces the
(project, id)index with a(project, seq)unique constraint.
Building that index scans the table. The whole migration runs in one transaction
and holds an ACCESS EXCLUSIVE lock on sync_boardevent until it commits. The
table holds at most TRUEPPM_BOARD_EVENT_RETENTION_HOURS of events (24 h by
default). We measured about 0.12 s for 300,000 rows across 500 projects on a
developer laptop, local PostgreSQL 16. That figure is from one machine. Expect it to
scale roughly with row count, and to be slower on shared or network storage.
Near the end of the migration, the counter table’s foreign key to projects is
created. That takes a SHARE ROW EXCLUSIVE lock on projects_project, held from
then until the migration commits. The locks are taken in this order:
sync_boardevent, then projects_project. While they are held, old pods that are
still serving wait on them:
- a board-event broadcast or a reconnect replay, for the whole migration;
- an
INSERT,UPDATEorDELETEon a project row, for the tail of the migration. This includes the sync-cursor bump each sync upload makes, and project edits. PlainSELECT … FOR UPDATEon a project row does not wait.
Each pauses for that long, then continues. Nothing fails. We found no old-pod
transaction that writes a project row and then touches sync_boardevent, so a
deadlock is not expected. This is not guaranteed by construction.
During a rolling update (replicaCount >= 2, as in values-prod.yaml), pods on
the old and new code run side by side for a few minutes, using two different seq
number spaces:
- A browser can briefly miss live updates or apply them late.
- A tab whose socket is still on an old pod can receive one event from a new pod. This applies to a newly loaded tab as well as an existing one. After that event, the tab ignores further events from old pods. Changes old pods make from then on are not shown until the data is refetched or the page is reloaded. Reconnecting to a new pod does not correct this on its own.
- Events written by an old pod during the window are delivered live but are not replayable after a reconnect.
Reloading the page clears all of these. Every client that was connected across
the upgrade refetches its project data once, on its first reconnect. If you script
against the WebSocket API, any cursor saved before the upgrade gets one
resync_required the first time you reconnect with it. See
Sequence numbers and reconnect replay.
Rollback: the migration is reversible. Unapplying it drops the column and the counter table. After a rollback, reload every open browser tab.
The reload is required because a tab still holds a per-project cursor above 2³², far above every global id the rolled-back release issues. The rolled-back server neither replays to such a cursor nor resyncs it. The tab then drops every live board event as already seen, until it is reloaded. For a scripted client, discard any cursor saved since the upgrade.
Upgrading across the #4359 readiness-probe change
Section titled “Upgrading across the #4359 readiness-probe change”Helm only — sets a floor on the web image, not just the chart. The web
pod’s readiness probe moved from GET / to GET /asset-manifest.json, a
build-specific file Vite writes at the html root (#4359). An image built from
v0.4.0-beta.7 or earlier has no asset-manifest.json at its html root — it
predates the file’s existence — so that probe path 404s unconditionally and
the web pod never turns Ready. On a fresh install, this means no web
endpoints join the Service at all; on an upgrade with --wait, the
command hangs until it times out. There is no per-tier tag knob — both the
api and web images resolve the same image.tag (trueppm.imageTag,
packages/helm/templates/_helpers.tpl) — so this can only happen by
explicitly overriding image.tag to v0.4.0-beta.7 or earlier on a chart
version that already carries #4359’s deployment.yaml; leaving image.tag
at its default (which resolves to the chart’s own appVersion) is
unaffected. The only other way to hit this is pointing image.webRepository
at an old web image some other way the chart does not model, such as a
pinned digest or a mirror that serves a stale tag.
Upgrading to 0.3
Section titled “Upgrading to 0.3”0.3 adds new database tables and columns for the agile-team feature set. A
migration is a script that changes TruePPM’s database structure to match a
new version of the code (adding a table or column, for example) — TruePPM ships
them, so you never write or edit one yourself. All of the migrations in 0.3 are
additive (new tables and nullable columns — no destructive operations), so
the upgrade is a standard migrate run with no manual data steps and no downtime
beyond that run. Every deploy path runs that step for you on startup — the
Compose api / api-init service, or the Helm chart’s migrate init container
— and each section below shows how to watch it complete. The new schema:
- Forecast snapshots (
scheduling.0007_projectforecastsnapshot) — a newProjectForecastSnapshottable that persists each project’s P50/P80/P95 Monte Carlo forecast over time, so the Schedule view can show a forecast history. Retention is bounded byMC_HISTORY_CAP(see configuration). - Sprint outcomes (
projects.0064_sprinttaskoutcome,projects.0065_historicalsprint_goal_outcome_sprint_goal_outcome) — a newSprintTaskOutcometable plus agoal_outcomecolumn onSprint(MET / PARTIAL / MISSED), capturing the sprint close-out snapshot. - Scope-change audit (
projects.0054_sprintscopechange_goal_impact_and_more) — agoal_impactcolumn onSprintScopeChange, recording whether a post-activation scope change affected the sprint goal.
If you maintain a fork, note that 0.3 also collapses each app’s migration history
into a 0001_squashed_… migration via Django’s replaces= (issue #1286). Because
the original migrations remain on disk and applyable, an existing database records
the squashed migration as already-applied and upgrades as a no-op — there is no
drop, recreate, or data step.
Docker Compose (development)
Section titled “Docker Compose (development)”git fetch --tags && git checkout v<version>docker compose builddocker compose up -d<version> is the release you are moving to; the current tag is on the
Releases page. Check out the
release tag rather than pulling main, whose tip can be ahead of any release.
build, not pull (#3189). The dev stack’s api and web services are
build:-based, and celery references image: trueppm-api:local with no
build: of its own — so docker compose pull has nothing to fetch for the
services that changed, and reports success. Building is what actually picks up
the new code.
Migrations run automatically when the api container starts.
Single-server with Docker Compose
Section titled “Single-server with Docker Compose”cd /opt/trueppmgit fetch --tags && git checkout v<version>
# Update the target version in .env (the bare version, no "v"; find the# current tag on the Releases page):# APP_VERSION=<version>
docker compose -f docker-compose.prod.yml pulldocker compose -f docker-compose.prod.yml up -dThe api-init service runs migrate --noinput before the API starts. Watch it complete:
docker compose -f docker-compose.prod.yml logs -f api-init# Should end with: "0 unapplied migration(s)." or a list of applied migrations.Helm / Kubernetes
Section titled “Helm / Kubernetes”helm upgrade trueppm oci://ghcr.io/trueppm/charts/trueppm \ --version <version> \ --namespace trueppm \ -f my-values.yaml<version> is the release version without a leading v; the current tag is
on the Releases page. Always pass it: without it Helm resolves the newest
stable chart and skips pre-release versions (--devel includes them), so an upgrade
can move further than you meant.
Migrations run in a migrate init container of the api Deployment before the new pods start serving. Check its logs:
kubectl logs -n trueppm deployment/trueppm-api -c migrateUpgrading to the hardened Helm chart
Section titled “Upgrading to the hardened Helm chart”The Helm chart now installs secure by default: it generates the PostgreSQL and
Valkey passwords and stores them in a chart-owned connection Secret
(<release>-trueppm-connection, annotated helm.sh/resource-policy: keep),
injects DATABASE_URL / REDIS_URL via secretKeyRef, enables Valkey auth by
default, and applies restricted container security contexts. A few notes when
upgrading from a pre-hardening release:
- Rotate the old default password. Earlier chart versions shipped a default
database/cache password of
trueppm. If you ran with that default, rotate it. The simplest path is to set explicit, strong passwords on the upgrade so the chart writes them into the connection Secret and the bundled datastores pick them up:Prefer supplying these through an external Secret overTerminal window helm upgrade trueppm oci://ghcr.io/trueppm/charts/trueppm \--namespace trueppm \-f my-values.yaml \--set postgresql.auth.password="<new-strong-password>" \--set valkey.auth.password="<new-strong-password>"--set. After the rollout settles you can clear the explicit values and let the chart manage the password from the connection Secret going forward. - Leave the passwords blank to keep the generated ones. On an upgrade where a
connection Secret already exists, leaving
postgresql.auth.password/valkey.auth.passwordempty makes the chart read the existing password back rather than minting a new one — so re-runninghelm upgradenever churns the credential or orphans the database PVC. - The connection Secret survives
helm uninstall. Theresource-policy: keepannotation means an accidental uninstall/reinstall reuses the same password and keeps the existing data reachable. If you intend a clean wipe, delete the Secret and the PersistentVolumeClaims explicitly. - Using managed datastores? When
postgresql.enabled/valkey.enabledarefalse,env.DATABASE_URLandenv.REDIS_URLare now required — the chart fails the render if either is missing. Add them (ideally via an external Secret) before upgrading. - App-side auth/CSP defaults. The refresh token now rides an httpOnly Secure cookie and a strict CSP header is sent on every response. A standard deploy needs no changes: TruePPM is served from a single origin, which is what these defaults assume. Splitting the SPA and API across hostnames is not supported — see Split-origin deploys.
Rollback
Section titled “Rollback”Migration reversibility — read this first
Section titled “Migration reversibility — read this first”The safe rollback path depends entirely on what the upgrade’s migrations did, so classify them before you touch anything (the release’s operational change note states this):
- Additive-only (new tables, new nullable columns, new indexes — the common case, and every 0.3 migration). The new schema is a superset of the old, so the previous image runs against it unchanged. Roll back the image/chart revision only — do not reverse the migrations and do not restore the database. The extra tables/columns sit unused until you roll forward again. The readiness probe cannot verify “additive-only” for you, so it holds the rolled-back pods out of the Service until you confirm that classification — see the caution below for the one-line opt-out.
- Destructive or transforming (a column drop/rename, a type change, or a data
backfill that rewrites rows). The old code cannot run against the new schema,
and reversing the migration loses the data the new schema captured. Here a
clean rollback means restore the pre-upgrade backup — a
migratereverse is not a substitute, because Django’s reverse operations recreate structure but cannot recover dropped or transformed data. This is why the pre-upgrade backup is mandatory, not optional.
Known migration constraints
Section titled “Known migration constraints”A few data migrations are intentionally irreversible: migrate <app> <previous>
runs, but the reverse step is a no-op or cannot rebuild what the forward step
removed. They are deliberate (restoring the old state would recreate a bug or needs
data that no longer exists), so the only way back across them is the pre-upgrade
backup described above.
| Migration | What the forward step does | What reversing does not restore |
|---|---|---|
projects.0075 | Renumbers risk short IDs from 8-character hex to a contiguous decimal sequence per project. | The original hex IDs. The decimal IDs stay valid and unique. |
projects.0121 | Clears the three-point estimate fields on any task where they were mis-ordered (optimistic above most likely, or most likely above pessimistic), so the new check constraint can validate. The task falls back to its fixed duration. | The mis-ordered values, which were already unusable. |
projects.0137 | Strips the redundant mcp:read scope from API tokens that also carry legacy:full. | The stripped scope. Stored state matches what the token already resolved to. |
projects.0149 | Drops the retired Board Workshop Mode tables and their django_migrations rows. | The tables and their rows. Restore from a backup taken before the upgrade. |
Docker Compose rollback
Section titled “Docker Compose rollback”# Restore the previous APP_VERSION in .env, then:docker compose -f docker-compose.prod.yml pulldocker compose -f docker-compose.prod.yml up -dIf the migration applied schema changes, restore from the pre-upgrade backup:
docker compose -f docker-compose.prod.yml downdocker compose -f docker-compose.prod.yml up db -d./scripts/restore.sh --artifact ./backups/trueppm-backup-<timestamp>.tar.gz --yes# Then bring up the full stack at the previous version.Concurrent migrations at replicaCount >= 2
Section titled “Concurrent migrations at replicaCount >= 2”migrate runs as a per-pod init container, not as a pre-upgrade hook Job,
so at two or more API replicas every pod runs it concurrently against one
database. Django has no concurrency control of its own: each process reads
django_migrations, decides the same migration is unapplied, and both run it.
From 0.4 the chart runs manage.py migrate_locked, which serializes them behind
a PostgreSQL advisory lock. The losers block rather than fail, and by the time
they acquire the lock the winner has finished, so their own migrate is a
no-op. Advisory locks release automatically when the holder’s connection dies,
so a killed init container cannot wedge the next rollout.
Nothing to configure. If a migration legitimately runs longer than ten minutes,
raise the wait with --lock-timeout in the init container’s command, or scale
the API to one replica for that upgrade.
Demo-mode upgrades: --wait and the seed hook timeout
Section titled “Demo-mode upgrades: --wait and the seed hook timeout”Demo mode (demo.enabled) adds a post-install,post-upgrade hook Job that
runs the same migrate_locked wait described above before re-seeding the
sample project and share links. Helm does not wait for the api
Deployment to become Ready before firing that hook, so on every demo
install or upgrade the hook’s own migrate initContainer can genuinely
contend for the advisory lock against the api Deployment’s init container —
and migrate_locked’s default wait for that lock is up to ten minutes,
longer than Helm’s own default --timeout of five minutes.
Always run demo installs and upgrades with an explicit wait and a longer timeout:
helm upgrade trueppm ./packages/helm -f packages/helm/values-demo.yaml \ --wait --timeout 15mWithout --wait, Helm does not block on the hook at all, so a slow seed
races the Deployments silently. With the default --timeout, a lock wait
anywhere near ten minutes can outlast Helm’s patience — but by then the api,
web, and worker Deployments have already been applied, so Helm marks the
release failed while the pods are already serving the new version; the
hook Job itself keeps running and is only cleaned up by the next upgrade’s
before-hook-creation policy.
Do not add --atomic for a demo-mode upgrade. --atomic turns a hook
failure into an automatic rollback, which — per the Helm
rollback skew window below, stacked on top of the stall this
section describes — leaves the deployment in a worse, harder-to-diagnose
state than a failed release that a human then re-runs.
Helm rollback
Section titled “Helm rollback”helm rollback trueppm -n trueppm --wait --timeout 10mAlways pass --wait. Without it, helm rollback patches the three Deployments
and returns exit 0 immediately — before the web rollout finishes, before the
api tier’s readiness probe has run even once, and well before any of the
sequence below is visible. A bare helm rollback exiting 0 is not evidence the
rollback finished; --wait makes the command itself fail (timeout or error)
when a target Deployment never becomes ready, instead of reporting success on a
rollback that is still stuck.
Across a release that added migrations, the three tiers do not roll back together — read this before you run it:
- Web rolls back immediately. The web tier’s readiness probe checks
/asset-manifest.json(a build-specific sentinel, #4359), not any api-dependent path (packages/helm/templates/web/deployment.yaml); it does not gate on the api’s version. AtreplicaCount>= 2 the rollback is an ordinary rolling update in reverse, so old- and new-release web pods serve together for the few seconds the rollout takes — the same mixed-asset mechanism as a forward upgrade (see cause 4 and #4341), just now serving the older bundle once it settles. - The api tier stalls, by design, and stays on the newer version. Each
rolled-back api pod still runs the
migrateinit container (migrate_locked) first. Because the init container’s migration graph is the old image’s — it has no record of migrations the newer release added — it finds nothing to apply and exits 0; this is not a crash loop. The api container then starts and/api/v1/readyzreports"migration_state": "ahead"(the database records migrations this image does not ship) and gates the podReady: false, becauseTRUEPPM_READYZ_ALLOW_DB_AHEADdefaults tofalse(packages/api/src/trueppm_api/settings/base.py). The chart sets no explicitstrategy:on the api Deployment, so Kubernetes’ default RollingUpdate (25%/25%, rounded) givesmaxUnavailable: 0at the 2–3 replica posturevalues-prod.yamlships — the rollout cannot scale down the still-Readynewer-version pods until the rolled-back replacement passes readiness, which it never does. The api tier is therefore left running the newer version indefinitely, with the rolled-back replica stuckNotReadybeside it; the Deployment’sprogressDeadlineSeconds(the chart does not override it, so Kubernetes’ default of 600 s applies) only flips theProgressingcondition toFalseafter ten minutes — it does not stop anything or roll back further on its own. - The worker has no migration gate at all and rolls back immediately. The
Celery worker’s readiness probe is a heartbeat-freshness check
(
packages/helm/templates/celery-worker/deployment.yaml,trueppm.celeryProbewithmode: heartbeat-fresh) — it never readsmigration_state. The rolled-back worker starts consuming tasks against the newer schema with nothing holding it back, which is the least visible leg of the split: nothing reports it as degraded.
The result is web on the old release, api on the new one, and worker on
the old one running against the new schema — not a transient skew window, but
a stable split state that --wait will time out on. Resolve it deliberately:
- Classify the release’s migrations using the migration reversibility rule above, from its operational change note.
- Confirm the api pod is gated on
ahead, notbehind, before touching the override —kubectl logson the stuck pod shows the one-timeTRUEPPM_READYZ_ALLOW_DB_AHEADWARNING, or query the probe directly:Terminal window kubectl exec -n trueppm deploy/trueppm-api -- \curl -s localhost:8000/api/v1/readyz# → {"status": "fail", "checks": {...}, "migration_state": "ahead"} - Additive-only: re-open the gate and let the stalled rollout complete.
helm rollbacktakes no--set, so apply the override withhelm upgradeagainst the same, explicit previous chart version — an unpinned--reuse-valuescan silently resolve back to the newest chart and undo the rollback instead of completing it:Once the api tier is healthy and you are ready to roll forward again, remove the override (Terminal window helm upgrade trueppm oci://ghcr.io/trueppm/charts/trueppm \--version <previous-chart-version> -n trueppm \--reuse-values --set env.TRUEPPM_READYZ_ALLOW_DB_AHEAD=true \--wait --timeout 10m--set env.TRUEPPM_READYZ_ALLOW_DB_AHEAD=false, or drop the key andhelm upgrade --reuse-values) — leaving it set hides the next rollback’s drift behind the same greenstatus/checksit hides this one behind. - Destructive or transforming: do not set the override — forcing the api
pod ready only makes it serve code that no longer matches the schema it
just rejected. Follow migration reversibility
above: restore the pre-upgrade backup, then roll the image back onto the
restored schema. Scale the worker to zero replicas first (
kubectl scale deployment/trueppm-celery-worker --replicas=0 -n trueppm) — it has no gate of its own and will otherwise keep consuming tasks against the schema you are about to replace underneath it. - Verify the fleet agrees on one version before calling the rollback
done — nothing in
kubectl get podssummarizes the split for you today, so check the image tag per tier directly:Terminal window kubectl get pods -n trueppm -o jsonpath='{range .items[*]}{.metadata.labels.app\.kubernetes\.io\/component}{"\t"}{.spec.containers[0].image}{"\n"}{end}' | sort -u
Post-upgrade verification
Section titled “Post-upgrade verification”# Check all containers are healthydocker compose -f docker-compose.prod.yml ps# orkubectl get pods -n trueppm
# Hit the health endpointcurl https://trueppm.example.com/api/v1/health/# → {"status": "ok"}
# Confirm the expected version is running — check the deployed image tagkubectl get deployment -n trueppm trueppm-api \ -o jsonpath='{.spec.template.spec.containers[0].image}'# or: helm list -n trueppmReverse proxies and WebSockets (0.5). Starting with 0.5, a proxy that rewrites the Host or Origin the API sees will make WebSockets fail with 1006. Set TRUEPPM_WS_ALLOWED_ORIGINS to the origin the browser sends.
Common issues
Section titled “Common issues”Migrations fail on startup
Check that DATABASE_URL is correct and the database is reachable. Run migrations manually to see the full traceback:
docker compose -f docker-compose.prod.yml exec api python manage.py migrate --noinputStatic files not updating
Trigger a collectstatic run:
docker compose -f docker-compose.prod.yml exec api python manage.py collectstatic --noinput --clearWebSocket connections drop after upgrade
Expected — clients reconnect automatically within a few seconds. The Channels layer (Valkey) is not drained between upgrades; in-flight messages are lost but clients recover via the reconnect loop.