Operate workflow retention and archives safely

This guide explains how to inspect workflow retention, prepare archive profiles, archive individual instances with verification, and preserve protected evidence. Workspace performs periodic deletion through tenant-bound Taskstream jobs. The administration UI is read-only; use nucli against the running server for mutating operations.

Retention jobs are installed in a paused state. Enable them only after schema, inventory, provider, and canary checks. Never run an old local workflow janitor in parallel.

Retention contract

retention_days has three meanings:

• omitted: inherit the tenant default,
• 0: retain workflow instances without a time limit,
• greater than 0: use the workflow-specific period.

Workspace starts the period at an immutable terminal epoch. A workflow that leaves a terminal state and later becomes terminal again receives a new epoch. The migration does not invent terminal timestamps for legacy rows, so legacy instances without authoritative evidence remain protected.

Workspace does not delete an instance while any of these blockers exists:

• a dead-letter entry,
• a running transition request or execution,
• an active trigger delivery, a non-terminal occurrence target, or an artifact operation,
• an active legal hold.

Terminal request, execution, action-checkpoint, and trigger history remains available for at least 365 days. Completed request replays remain available for at least 30 days. Resolved and not-replayable dead letters follow their separate default 30-day horizon; open dead letters are not deleted. Open claims and running work are never deleted as history.

Inspect inventory and dry-run

Read the tenant-bound inventory first:

nucli --tenant <tenant-alias> workflow retention inventory --json

Review missing terminal epochs, dead letters, active requests, executions, deliveries, artifact operations, legal holds, and the current eligible count.

Run reconciliation without mutation:

nucli --tenant <tenant-alias> workflow retention reconcile --json

Without --apply, nucli sends only the preflight. For an explicitly approved bounded run, nucli repeats the preflight before the apply request:

nucli --tenant <tenant-alias> workflow retention reconcile --apply --json

One run processes no more than 20 batches of 500 instances, 10,000 instances in total, and five minutes. Concurrent workers skip locked rows and do not delete an instance twice.

Configure an archive profile

The filesystem provider uses a server-controlled data root and needs no credentials:

nucli --tenant <tenant-alias> workflow retention profile set \
  --provider filesystem \
  --retention-days 365 \
  --config '{"prefix":"workflow"}'

For S3-compatible storage, store the access key, secret key, and optional session token as a tenant-bound secret. Pass only its UUID to the profile:

nucli --tenant <tenant-alias> workflow retention profile set \
  --provider s3-compatible \
  --credential-secret-id <secret-uuid> \
  --retention-days 365 \
  --config '{"endpoint":"https://objects.example","bucket":"workflow-archive","region":"eu-central-1","prefix":"nucleus"}'

Never place credential values in --config, logs, work orders, or shell history. Workspace persists only the secret reference in the profile. Archive retention is unlimited (0) or at least 365 days.

S3-compatible profiles also require an installation-wide archive egress approval. The locked global keys workflow.archive.egress.mode and workflow.archive.egress.allowed_destinations default to allowlist_only with an empty list. Add the normalized endpoint with its scheme, host, and port to the allowlist. Workspace revalidates DNS and the destination IP for each operation, pins the transport to the approved address, and never follows redirects. A missing or invalid approval keeps the profile fail-closed.

Verify the provider with a write-once/read-back probe:

nucli --tenant <tenant-alias> workflow retention profile readiness

A profile becomes ready only after Workspace reads the unchanged probe object. Constructing a client or reaching the endpoint alone is not sufficient.

Archive one instance explicitly

Run the non-mutating preflight:

nucli --tenant <tenant-alias> workflow retention archive <instance-uuid> --json

The preflight requires an expired terminal epoch, a ready profile, and no open recovery, request, trigger, or artifact operation. An active legal hold blocks periodic deletion but does not block this explicit archive path. Then explicitly archive and delete the instance:

nucli --tenant <tenant-alias> workflow retention archive <instance-uuid> --apply --json

Workspace creates a versioned manifest containing the definition, context, transition history, action checkpoints, recovery, trigger occurrences, occurrence targets, deliveries, and artifact evidence. The complete evidence also includes earlier terminal epochs, every context bound to the instance, and soft-deleted runtime children; foreign-tenant rows are not silently filtered out. Workspace encrypts the manifest under nucleus/workflow-archive.v1 and writes it once. The encrypted envelope must not exceed 64 MiB. Workspace reads the provider response as a bounded stream, verifies its actual size and SHA-256 digest, and does not trust Content-Length or other provider metadata. Workspace then binds an active legal hold to that evidence before a CAS deletes the unchanged instance. If the later delete CAS fails, the verified evidence remains available and is checked again on retry. Any write, read, size, decrypt, digest, tenant, eligibility, or CAS failure leaves the source intact. Foreign-tenant context, log, definition, or runtime rows are data drift and block archiving before the provider write; Workspace does not silently filter them.

The paused-by-default Taskstream retention job also evaluates verified archive objects. After the retention period captured with the snapshot expires, it deletes the provider object only with the verified opaque object version. An active legal hold or unlimited archive retention blocks physical deletion. The tenant-bound verification record remains as a tombstone. Provider lifecycle rules must not shorten the recorded period or override a legal hold.

Deleting a store no longer triggers a hidden partial archive. Explicitly archive bound workflow instances first. The store can then detach the already removed binding.

Legal hold and recovery closure

Append a legal hold with an auditable reason:

nucli --tenant <tenant-alias> workflow retention hold <instance-uuid> \
  --reason "Preserve for case 4711"

A release appends a separate event and does not overwrite the hold:

nucli --tenant <tenant-alias> workflow retention hold <instance-uuid> \
  --release --reason "Case 4711 closed"

If a dead letter must not be replayed, close recovery only with the current fence, a reason, and explicit apply:

nucli --tenant <tenant-alias> workflow recovery cancel <dead-letter-uuid> \
  --expected-fence <fence> \
  --reason "Rejected by business decision" \
  --apply

Workspace records the actor, correlation, and fence. The dead letter remains a retention blocker until its history horizon expires.

Permissions and stop signals

Grant only the permissions required for the operation:

• workflow_retention:read
• workflow_retention:reconcile
• workflow_retention:archive
• workflow_retention:profile_manage
• workflow_retention:legal_hold
• workflow_recovery:cancel

Stop when a profile is not ready, terminal epochs are missing, blockers are unexpected, the tenant does not match, verification fails, or the generated envelope exceeds 64 MiB. The apply endpoint reports the safe code WORKFLOW_ARCHIVE_OBJECT_LIMIT_EXCEEDED for this case. Envelope v1 is not intended for larger archives. Do not edit workflow data directly in PostgreSQL, and do not use numin as a remote retention client. Profile and archive responses expose only stable IDs, status, retention, and readiness codes. Endpoint, bucket, prefix, secret reference, object path, object version, digest, and cryptographic metadata remain internal.