Dead-letter Alerting

Added in 0.2 (alpha)

This page documents functionality added in TruePPM 0.2, available since the 0.2.0-alpha.1 pre-release (May 31, 2026). 0.2 is an alpha release; the first beta is planned for 0.4.

Ships in 0.4

One section on this page is not in the latest release (v0.3.0-alpha.3): Outbound webhook deliveries. Webhook delivery failures reach the dead-letter queue from 0.4 onward. On the current release a permanently failed delivery is recorded only on its own delivery row — it produces no FailedTask, no alert line, and no movement in trueppm_task_dead_letter_parked.

When a background Celery task in TruePPM exhausts its retries, the work is permanently abandoned — a CPM recalculation, a notification email, an MS Project import, all driven asynchronously. Without a signal, a solo operator has no way to know that work silently died short of reading the failed-task admin viewset by hand.

Dead-letter alerting closes that gap. Every permanently-failed task records a durable row, emits a structured alert log line, and is counted on a Prometheus endpoint you can scrape and alert on.

Edition

Routing dead-letter alerts to PagerDuty or Slack is an Enterprise extension (see Routing alerts off-box). This page covers the basic detection layer every adopter gets: the durable record, the alert log line, and the Prometheus gauge.

What happens when a task is dead-lettered

A task is dead-lettered when it has exhausted all configured retries and will not run again. At that point, three things happen exactly once per failed task:

1. A durable FailedTask row is recorded (status DEAD). This is the system of record — it survives restarts and is what the metrics endpoint counts. It is the same row the failed-task admin viewset reads. From 0.4, the row will also carry a project_id column (nullable — not every dead-lettered task is project-scoped) so failures will be attributable to a project directly from the admin API or the ORM, not only from the alert log line’s extra field.

2. A structured WARNING log line is emitted by the trueppm_api.apps.scheduling.receivers logger:

   WARNING dead-letter alert: task scheduling.recalculate (a1b2c3d4-…) permanently failed: connection timed out

   The line carries machine-readable extra fields for log-based alerting:

   Field	Meaning
   task_name	The registered Celery task name (e.g. scheduling.recalculate)
   task_id	The Celery task id of the failed run
   exception_type	The exception class name that caused the final failure
   project_id	The owning project id, or null when the task is not project-scoped

3. A celery_task_permanently_failed signal fires, which Enterprise (or your own code) can subscribe to for off-box alerting. The OSS receiver only logs — see Routing alerts off-box.

The alert fires once per newly dead-lettered task: a re-delivered failure for a task id that is already parked does not re-alert, so you do not get an alert storm from a single stuck task.

Note

The alert WARNING is deliberately a distinct, lower-severity line from the ERROR that records the underlying task failure. The ERROR records what failed; the WARNING records that an operator alert was raised. The alerting receiver is exception-safe — a failure inside alerting can never mask or re-raise into the dead-letter recording path. See ADR-0084 for the full rationale.

Alerting on the log line

If you forward worker logs to an aggregator (Loki, CloudWatch, Datadog, an ELK stack), alert on the dead-letter alert: message from the trueppm_api.apps.scheduling.receivers logger. The extra fields are emitted as structured fields, so you can route or group by task_name or project_id without parsing the message string. This is the fallback signal for deployments with no Prometheus.

The /api/v1/health/dead-letter/ metrics endpoint

For metrics-based alerting, scrape the Prometheus-text endpoint. It requires a staff (admin) account — it exposes operational state, so it is gated with IsAdminUser and is bearer-scrapeable, mirroring /api/v1/health/beat/ (see Beat Liveness & Durability).

It emits a single gauge, trueppm_task_dead_letter_parked, labeled by task name:

# HELP trueppm_task_dead_letter_parked Permanently dead-lettered Celery tasks currently awaiting operator action, by task name.
# TYPE trueppm_task_dead_letter_parked gauge
trueppm_task_dead_letter_parked{task_name="scheduling.recalculate"} 3

curl -fsS -H "Authorization: Bearer $ADMIN_JWT" \
  https://trueppm.example.com/api/v1/health/dead-letter/

The metric is a gauge, not a counter: it counts dead-lettered tasks that are currently parked — awaiting operator action. The value falls when a parked task is dismissed, retried, or purged. A value of 0 (or the series disappearing for a given task_name) means nothing is currently dead-lettered for that task.

Note

The gauge is derived from the FailedTask table on each scrape (COUNT(*) … GROUP BY task_name … WHERE status = DEAD), not from an in-memory counter. This is correct by construction across processes: the Celery worker records the dead-letter, and the web process serves the scrape — a counter held in worker memory would be invisible to the scraper. Reading committed FailedTask rows means the web process sees exactly what the worker recorded.

Wiring it into Prometheus

The endpoint is authenticated, so it is not a drop-in unauthenticated scrape target. Configure the scrape job with a bearer token:

scrape_configs:
  - job_name: trueppm-dead-letter
    metrics_path: /api/v1/health/dead-letter/
    scheme: https
    authorization:
      type: Bearer
      credentials: <admin-jwt>      # or credentials_file: /etc/prometheus/trueppm-token
    static_configs:
      - targets: ["trueppm.example.com"]

A useful alert rule fires when anything is parked:

groups:
  - name: trueppm-dead-letter
    rules:
      - alert: TruePPMTaskDeadLettered
        expr: sum(trueppm_task_dead_letter_parked) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "TruePPM has permanently-failed background tasks awaiting action"

Because the metric is a gauge of currently parked work, the alert clears automatically once the parked tasks are retried, dismissed, or purged — no manual alert reset.

Routing alerts off-box (Enterprise)

The OSS edition only logs the alert and counts it on the metrics endpoint. It never calls out to PagerDuty, Slack, or any external pager — that is an Enterprise capability.

The integration point is the celery_task_permanently_failed Django signal in scheduling/signals.py. Enterprise (or your own custom app) registers an additional receiver against it from its own AppConfig.ready() — the OSS log-only receiver stays in place and the two run side by side:

# In an enterprise / custom app's apps.py
from django.apps import AppConfig


class AlertingConfig(AppConfig):
    name = "trueppm_enterprise.alerting"

    def ready(self) -> None:
        # Importing scheduling.signals gives the stable extension contract;
        # connecting a new receiver does not modify or replace the OSS one.
        from trueppm_api.apps.scheduling import signals

        @signals.celery_task_permanently_failed.connect(weak=False, dispatch_uid="enterprise.page_on_dead_letter")
        def page_on_dead_letter(sender, *, task_name, task_id, exception, project_id, **kwargs):
            pagerduty.trigger(
                summary=f"TruePPM task {task_name} permanently failed",
                dedup_key=task_id,
                custom_details={
                    "exception": type(exception).__name__,
                    "project_id": project_id,
                },
            )

The signal payload carries task_id, task_name, exception, traceback_str, and project_id (None when the task is not project-scoped).

Always pass weak=False (and a dispatch_uid) from ready()

A receiver defined as a nested function inside AppConfig.ready() — as page_on_dead_letter is above — has no reference keeping it alive once ready() returns other than the signal’s own bookkeeping. Django and Celery signals default connect() to a weak reference, so without weak=False the receiver is garbage-collected almost immediately and silently stops firing; there is no error, the alert just never arrives. A stable dispatch_uid additionally prevents a second ready() call (Django’s test runner and autoreloader can trigger one) from registering a second, now-permanent receiver and double-firing every alert.

Apache 2.0 boundary

The OSS core never imports trueppm_enterprise. The dependency is one-way: Enterprise registers against the OSS signal, never the reverse. The signal’s shape is a stable extension contract — OSS does not change it without treating it as a breaking change for Enterprise. See ADR-0084.

Outbound webhook deliveries

Outbound webhook delivery is covered by this page’s machinery, with one wrinkle worth knowing: the delivery task does not fail the way a normal Celery task does.

A delivery that will never be attempted again — a permanent 4xx, an exhausted retry chain, a host blocked by the SSRF egress guard, or a missing signing secret — marks its own delivery row failed and returns normally. Nothing raises, because raising would hand the retry decision back to Celery for something the task has already decided not to retry. The task therefore parks itself explicitly, and the result is indistinguishable from any other dead-lettered task:

WARNING dead-letter alert: task webhooks.deliver_webhook (a1b2c3d4-…) permanently failed: WebhookDeliveryFailed

trueppm_task_dead_letter_parked{task_name="webhooks.deliver_webhook"} 2

The FailedTask row carries the delivery id in args, the webhook id in kwargs, and the project (for project-scoped webhooks) in project_id. The exception_message is the human-readable cause — Permanent client error HTTP 410 — not retried, HTTP 503 after 5 attempts, and so on.

Two things this does not cover:

• A delivery to an inactive webhook is not parked. Skipping a paused subscription is a deliberate state, not a fault; parking every skipped delivery would bury real failures under noise from a webhook nobody expects to fire.

  The consequence is worth internalizing, because it inverts the usual reading of your alert stream: when the automatic guard deactivates a subscription, that webhook’s dead-letter records stop. Silence after a burst of webhooks.deliver_webhook failures means the integration is dead, not that it recovered. The parked record for the failure that crossed the threshold carries SUBSCRIPTION DEACTIVATED in its exception_message precisely so the transition is greppable — alert on that string, not only on the parked count.

• Requeuing a parked delivery does not resend it. The dead-letter admin’s requeue action re-dispatches scheduling tasks; there is no replay path for a webhook delivery. Treat a parked delivery as evidence for the operator, and reconcile missed events through the API.

Alongside this, a webhook that fails terminally five times in a row is deactivated automatically, which is the project-admin-facing half of the same signal.

Related

• Webhooks — outbound delivery, its retry classification, and the automatic-deactivation guard.
• Beat Liveness & Durability — the sibling /api/v1/health/beat/ detector for a dead Celery Beat process.
• Outbox & Record Retention — FailedTask retention is governed by the retention purge; parked dead-letters are reaped along with other terminal records.