Recover from critical runtime failures safely

An Emergency Release fixes a critical runtime defect with a signed OCI image without changing the database, Product Release targets, configuration, or storage contract. Use this lane only when schukai supplies a valid EmergencyRelease manifest with stateChangeClass: runtime_only for the exact image.

Verify signatures, digests, and database compatibility first. Test the image in a canary next. Switch production only after explicit approval. A missing or failed result stops the process.

Situation	Correct lane
Signed OCI image with `runtime_only` and unchanged Product Release identity	Emergency Release on this page
Schema, Product Release, configuration, or storage change	Normal update and recovery under Run product updates without manual repair
DEB, RPM, or standalone binary	Normal package update
Missing manifest, signature, evidence, or exact digest	Stop and request a complete release bundle

Prerequisites

You need:

source and target images as immutable REGISTRY/REPOSITORY@sha256:<digest> references;
emergency-release.json and the detached emergency-release.sigstore.json from the same release bundle;
the Emergency Release public key obtained separately;
the separately trusted nucleus-emergency-release-contract validator and nucleus-emergency-browser-canary, plus Cosign and jq;
the issuing CA installed locally for the registry, curl, and the canary browser;
a mode-0600 credential file for an explicitly provided canary account;
an isolated canary or designated canary instance.

Do not obtain the verifier from the unverified target image. Distribute the public key and verifier through your approved operator-tooling channel.

Agent assistance without automatic approval

An agent can explain the workflow and deterministically classify existing gate results. It must neither attest signatures itself nor switch a production pin. Start with:

bash

nucli skills show emergency-release

MCP provides two local, read-only tools:

Tool	Purpose	Changes anything?
`nucleus_emergency_release_skill`	Returns the complete signature, live-plan, Canary, external approval, readback, and recovery workflow.	No
`nucleus_emergency_release_assess`	Classifies separately verified boolean gate results and identifies the next safe handoff.	No

The assessment requires every observation below. A boolean value records the result of a separate check; it is never the evidence itself:

Input	Source
`phase`	`pre_activation`, `post_activation`, or `rollback`, matching the current operating phase
`exactDigestsVerified`	Comparison of immutable source, target, manifest, and evidence digests
`signatureVerified`, `attestationVerified`	Separately trusted verifier with the pinned public key and registry CA
`runtimeOnly`	Verified manifest plus unchanged Product Release, schema, config, and storage contracts
`deploymentPlanReady`	Matching target `numin`, exact target digest, and exact manifest against live state
`canaryPassed`	Digest-bound CanaryEvidence from health, fresh API/browser sessions, and the problem smoke
`externalApprovalGranted`	Explicit infrastructure approval bound to environment, target, manifest, and Canary digests
`primaryReadbackPassed`	At least five minutes of readback on the activated primary environment
`stateChanged`	Recheck of Product Release identity, schema, transition digest, config, and storage
`rollbackAllowed`	A newly executed live plan still permits the exact source digest

Possible outcomes stop on mutable identity, incomplete supply-chain verification, a state-changing artifact, a missing live plan, a failed Canary, or missing external approval. Only external_activation_handoff_ready lets infrastructure switch the affected pin outside the MCP tool. After activation, primary_readback_required requires primary readback. A source switch is only a candidate with binary_switch_eligible; otherwise use forward recovery or a complete restore.

Do not pass credential files, cookies, private signing keys, manifest secrets, or unfiltered customer logs. The assessment verifies no artifact, grants no approval, and replaces no live check.

Verify the image and manifest

Run verification before starting the target image:

bash

export NUCLEUS_EMERGENCY_CONTRACT_BIN=/opt/workspace-operator/bin/nucleus-emergency-release-contract
export NUCLEUS_EMERGENCY_BROWSER_CANARY_BIN=/opt/workspace-operator/bin/nucleus-emergency-browser-canary

/opt/workspace-operator/bin/verify-emergency-release.sh \
  --image registry.example/workspace@sha256:<target-digest> \
  --manifest /secure/emergency-release.json \
  --manifest-bundle /secure/emergency-release.sigstore.json \
  --public-key /etc/workspace/trust/emergency-release.pub \
  --registry-ca /etc/ssl/certs/registry-ca.pem

The command confirms the signature, image digest, canonical manifest, and OCI attestation. It fails on unknown fields, mutable tags, or an attestation that contains another manifest.

Check the database contract before activation

Use numin from the exact target image. This command reads the persistent state and changes no data:

bash

/path/to/target/numin \
  --config /path/to/nucleus.config.enc \
  deployment plan \
  --artifact-digest <target-digest> \
  --emergency-manifest /secure/emergency-release.json \
  --json

Continue only when all of these fields match:

text

ready: true
releaseMode: runtime_only
rollbackToSource.allowed: true
rollbackToSource.mode: binary_switch

The hotfix may have a newer runtime version. Its productReleaseVersion, Product Release manifest digest, schema baseline, and transition digest must match the source image and persistent state.

Run the canary

The credential file contains exactly the email and password fields. Create it as a protected file:

bash

chmod 0600 /secure/canary-credentials.json

/opt/workspace-operator/bin/run-emergency-canary.sh \
  --image registry.example/workspace@sha256:<target-digest> \
  --manifest /secure/emergency-release.json \
  --manifest-bundle /secure/emergency-release.sigstore.json \
  --public-key /etc/workspace/trust/emergency-release.pub \
  --base-url https://canary.example.com \
  --credentials-file /secure/canary-credentials.json \
  --ca-file /etc/ssl/certs/workspace-ca.pem \
  --environment production_canary \
  --output /secure/canary-evidence.json

The fixed smoke catalog checks Liveness, Readiness, the public login route, a fresh API session, and a fresh browser session. The manifest cannot run arbitrary shell commands. Keep the emitted canary_evidence_digest for the approval record.

Approve the production switch

Infrastructure approval must bind at least:

target environment;
target image digest;
Emergency Release manifest digest;
Canary Evidence digest.

Automate verification and canary checks. Change the production pin only after explicit approval. Change only the affected runtime pin. Do not move stable or latest tags.

For the supplied Docker Compose installation, the approved switch is:

bash

./start.sh update \
  registry.example/workspace@sha256:<target-digest> \
  --emergency-manifest /secure/emergency-release.json

The helper still creates and restore-tests a backup, runs the target DeploymentPlan, and waits for Readiness.

For the supplied Kustomize installation, use the same approved handoff:

bash

./update.sh \
  registry.example/workspace@sha256:<target-digest> \
  /secure/backup-evidence.json \
  --emergency-manifest /secure/emergency-release.json

The script exposes the manifest and backup evidence only to the one-time, unprivileged preflight as ConfigMaps. It changes the Deployment only when numin from the exact target image approves the live state.

Observe the primary system for five minutes

Run a new primary readback for at least 300 seconds after the switch:

bash

/opt/workspace-operator/bin/run-emergency-canary.sh \
  --image registry.example/workspace@sha256:<target-digest> \
  --manifest /secure/emergency-release.json \
  --manifest-bundle /secure/emergency-release.sigstore.json \
  --public-key /etc/workspace/trust/emergency-release.pub \
  --base-url https://workspace.example.com \
  --credentials-file /secure/canary-credentials.json \
  --ca-file /etc/ssl/certs/workspace-ca.pem \
  --environment primary_readback \
  --observation-seconds 300 \
  --output /secure/primary-readback.json

Keep the source image for at least 24 hours after a successful readback. Do not delete baseline-readiness evidence or images it references during the support period.

Switch back to the source image

Verify the signature and manifest again. Then let the active hotfix validate the current database with the same manifest. The Docker Compose installation combines this check with the exact switch:

bash

./start.sh emergency-rollback \
  registry.example/workspace@sha256:<source-digest> \
  --emergency-manifest /secure/emergency-release.json

The helper stops if the source digest, active target digest, or persistent state differs. If the plan no longer permits a binary switch, recover forward with a compatible version or restore the database, storage, encrypted configuration, and runtime key as one set.

Next steps

Use numin for the read-only DeploymentPlan.
Use nucli after Readiness for authenticated status and session checks.
Follow Run product updates without manual repair whenever a hotfix changes a persistent contract.