Run workflow recovery safely

This page explains how to inspect a failed workflow step and deliberately queue it for recovery. Use this procedure only for an existing dead-letter entry. Workspace re-executes either an uncommitted transition or the exact persisted action; it never reruns the entire workflow indiscriminately.

Prerequisites

You need:

  • an authenticated nucli tenant alias for the affected tenant,
  • permission to read and replay workflow dead-letter entries,
  • the entry UUID from workflow monitoring or GET /api/v1/workflow-dead-letters.

For a reachable server, always use nucli or the tenant-bound API. Do not modify workflow or dead-letter data directly in PostgreSQL. numin is not a remote recovery client for this procedure.

Review the preflight

Run the read-only preflight first:

bash
nucli --tenant <tenant-alias> workflow recovery replay <dead-letter-uuid>

Add --json for machine-readable output. The preflight reads the entry and shows its recovery kind, instance, status, and target without mutating server state. Confirm that:

  • you selected the expected tenant,
  • the instance, transition, or action belongs to the expected business case,
  • the entry is still open and the original business or technical failure has been addressed,
  • the action is explicitly replay-safe; unclassified actions remain blocked.

Queue recovery

After the preflight identifies the expected entry, explicitly queue recovery:

bash
nucli --tenant <tenant-alias> workflow recovery replay <dead-letter-uuid> --apply

Before the mutating request, nucli repeats the preflight. The server returns queued only after it has persisted the recovery request. A worker then claims it with a lease and monotonic fence.

Transition recovery re-executes only the transition that was not committed. Action recovery re-executes only the action identified by its persistent UUID and does not change workflow state a second time. Successful action checkpoints are not repeated.

Verify the outcome

Read the entry and workflow instance again. Successful recovery moves the entry to a resolved status. The instance remains visibly dead-lettered while another unresolved entry still exists for it.

Treat these responses as stop signals:

  • 401: authenticate the selected tenant alias again.
  • 403: do not obtain broader permissions without operational approval.
  • 404: verify the UUID and tenant; entries from another tenant remain hidden.
  • 409: the entry is already being processed, is an unreplayable legacy entry, or references an action that is not approved for replay.

Do not work around a conflict in the database. Review the entry, action classification, and active worker first.