Run workflow recovery safely

This operating guide shows administrators how to inspect a failed workflow step and deliberately queue it for recovery. After completing the procedure, a replay-safe entry is queued or has been identified as not replayable. Workspace re-executes either an uncommitted transition or the exact persisted action; it never reruns the entire workflow indiscriminately.

Prerequisites

You need:

  • a Workspace session for the affected tenant or an authenticated nucli tenant alias,
  • permission to read and replay workflow dead-letter entries,
  • access to Automation > Workflows > Recovery or the entry UUID from GET /api/v1/workflow-dead-letters.

For a reachable server, always use nucli or the tenant-bound API. Do not modify workflow or dead-letter data directly in PostgreSQL. numin is not a remote recovery client for this procedure.

Review the entry in Workspace

  1. Open Automation > Workflows > Recovery.
  2. Narrow the worklist by status.
  3. Review the workflow step, instance, error, attempts, and dead-letter time.
  4. Correct the business or technical cause.
  5. Select Replay only for an Open or Retry scheduled entry and confirm that you want to queue it.

The action is available only with read and replay permissions. Queued and Running entries are already being processed. Resolved entries need no further action. Do not bypass a Not replayable result with a database change.

Review the preflight with nucli

Use nucli when you need to script, automate, or review the procedure outside the administration UI. Run the read-only preflight first:

bash
nucli --tenant <tenant-alias> workflow recovery replay <dead-letter-uuid>

Add --json for machine-readable output. The preflight reads the entry and shows its recovery kind, instance, status, and target without mutating server state. Confirm that:

  • you selected the expected tenant,
  • the instance, transition, or action belongs to the expected business case,
  • the entry is still open and the original business or technical failure has been addressed,
  • the action is explicitly replay-safe; unclassified actions remain blocked.

Queue recovery with nucli

After the preflight identifies the expected entry, explicitly queue recovery:

bash
nucli --tenant <tenant-alias> workflow recovery replay <dead-letter-uuid> --apply

Before the mutating request, nucli repeats the preflight. The server returns queued only after it has persisted the recovery request. A worker then claims it with a lease and monotonic fence.

Transition recovery re-executes only the transition that was not committed. Action recovery re-executes only the action identified by its persistent UUID and does not change workflow state a second time. Successful action checkpoints are not repeated.

Replay a failed trigger materialization

A failed internal trigger transport is not a workflow dead-letter entry. Review it separately when an accepted event or scheduled trigger execution did not materialize a delivery. You need the workflow_triggers:replay permission.

When the global workflow trigger runtime is disabled, queued materialization and delivery events remain paused without consuming another failed attempt. Re-enable the runtime only after checking the technical cause. Processing then resumes automatically; manual replay is necessary only for materializations that already reached a terminal failure.

For delayed materialization, the persisted source-event creation time remains authoritative. Workspace assigns the event only to triggers whose half-open activation window contains that time. A trigger activated later therefore does not consume older events of the same topic that are still queued. If a tenant-specific override was not active at that time, the system definition that was effective then remains authoritative.

If only the internal transport of an already persisted delivery fails before business execution starts, the delivery remains intact. After the short transport lease expires, Workspace automatically reserves a higher transport generation. You do not need to replay a dead-letter entry or modify database data. Check the runtime and correct the technical cause if higher generations also fail repeatedly.

If the business trigger delivery instead reaches its configured attempt limit, its dead-letter entry appears with the Not replayable status. Workspace does not offer a replay action for this entry because the existing recovery worker supports only transitions and explicitly replay-safe actions. Do not confuse this state with a failed trigger materialization. Do not modify the delivery or dead-letter row directly in the database; document the case and clarify the supported operational resolution.

List open failures for the selected tenant first:

bash
nucli --tenant <tenant-alias> workflow triggers failures list

Add --json for a machine-readable page. If the response contains a nextCursor, pass it to the next request with --cursor. The response contains only safe operational fields. It does not expose raw event payloads or internal database errors.

Run the read-only preflight for one failed event:

bash
nucli --tenant <tenant-alias> workflow triggers failures replay <event-uuid>

Correct the technical cause first. Then explicitly create the replay event:

bash
nucli --tenant <tenant-alias> workflow triggers failures replay <event-uuid> --apply

Before applying the change, nucli repeats the preflight. Workspace locks the current business continuation, verifies its tenant and progress, and creates at most one deterministic successor event. Repeating the apply command uses the same successor. After a successful apply, the predecessor no longer appears in the open-failure list.

Stop on 404 or 409. Verify the tenant, event UUID, and current workflow state. Do not bypass a stale continuation or authority conflict with a database change.

Verify the outcome

Refresh the worklist and read the entry and workflow instance again. The normal sequence starts with Queued, moves to Running, and ends with Resolved after successful recovery. A transient failure can move the entry to Retry scheduled. The instance remains visibly dead-lettered while another unresolved entry still exists for it.

The task is complete when the entry is Resolved and the instance has no other unresolved dead-letter entry. If the entry remains Not replayable, document the case and clarify the supported operator closure; do not modify the record directly.

Treat these responses as stop signals:

  • 401: authenticate the selected tenant alias again.
  • 403: do not obtain broader permissions without operational approval.
  • 404: verify the UUID and tenant; entries from another tenant remain hidden.
  • 409: the entry is already being processed, is an unreplayable legacy entry, or references an action that is not approved for replay.

Do not work around a conflict in the database. Review the entry, action classification, and active worker first.