Run product updates without manual repair

Workspace applies every safe, product-owned transition for a supported update. You do not need to inspect database columns, system artifacts, Taskstream sources, or revision anchors separately.

Before every update, the staged target release creates a read-only DeploymentPlan. It binds the source and target baselines, target artifact, backup, transition class, and point of no return. Activate the target only after this plan is ready.

Do I need another release after an update fails?

Not merely because you see HTTP 503, a visible sign-in page, or retry_wait. Decide from the persisted transition state:

A schema update may have succeeded even though Product Release fails later. The overall startup then failed, but the database change is real. Never start an older binary as an experiment. Inspect schemaTransitionClass, pointOfNoReturnAt, minimumRuntimeBaseline, and rollbackAfterSchemaTransition before deciding how to recover.

Choose the deployment mode

Decide which availability level the installation must provide before you start:

An uninterrupted rolling update requires the running source release to be inside the target release's supported mixed-version window. The DeploymentPlan must report transitionClass: compatible_expand and a minimumRuntimeBaseline that includes the source release. A forward_only transition requires a maintenance window. Workspace keeps compatible old nodes serving during reconciliation. Incompatible nodes stop accepting new work and enter drained in a controlled way.

Understand the point of no return

The physical schema baseline describes the database state. The minimumRuntimeBaseline identifies the oldest server baseline that may still open that state.

Preflight the target release

Stage or pull the target without activating it. Then create a complete backup and perform an actual restore test. Start the target binary outside the load balancer in deployment-preflight mode. This mode exposes only the diagnostic endpoint, sets PostgreSQL to default_transaction_read_only=on for every connection, and never reports itself as a business-ready Workspace instance:

nucleus deployment-preflight \
  --config /path/to/nucleus.config.enc \
  --listen 127.0.0.1:8444 \
  --target-digest sha256:<digest> \
  --backup-evidence /secure/backup-evidence.json

nucli \
  --host https://workspace.example.com \
  --tenant system \
  system update doctor \
  --target-url https://127.0.0.1:8444 \
  --target-digest sha256:<digest> \
  --json

nucli reads the authoritative source state from the running server and compares it with the database state observed by the target binary. It binds the release, manifest, schema baseline, target digest, and System tenant into short-lived evidence. The command changes no data.

The reported updateMode is rolling, maintenance_required, bridge, or offline. --watch waits only for transient Product Release work that is already running. A preflight process is never promoted to a normal server. Stop it, then start the exact same binary or OCI digest in normal mode.

If the running server is unreachable, use numin deployment plan from the exact target artifact as the privileged offline fallback. Do not use numin to bypass authentication or server-side contracts on a reachable server.

Correct invalid fulfillment follow-up modes

If Doctor reports the fulfillment_follow_up_mode_contract check with action update_fulfillment_follow_up_mode_invalid, stop the update. For security, the remote preflight reports only affectedCount, not tenant or product variant identities.

Run the local read-only diagnosis with the numin version from the staged target artifact:

numin --config /data/nucleus.config.enc schema migrations status --json

preflightFindings contains the total count and up to 500 deterministically sorted tenant and variant IDs. A truncated: true value means that more rows are affected. Correct only the listed product variants to manual or automatic through the existing tenant-bound PIM workflow. numin intentionally provides no --apply path for this finding.

After each correction round, rerun the local diagnosis and then nucli system update doctor. Activate the target release only after the finding is gone and remote preflight returns exit code 0 again.

Run a rolling or blue/green update

1. Create the backup required by your deployment.
2. Configure the load balancer to route business traffic only to instances whose /health/ready check succeeds.
3. Run target preflight and require exit code 0 with updateMode: rolling.
4. Stop preflight. Start the exact same digest next to at least one compatible, ready old instance. Do not stop the old instance yet.
5. Watch the protected status of the new instance:

   nucli --tenant system system product-release status --watch

1. Wait until the new instance returns HTTP 200 from /health/ready.
2. Add the new instance to business traffic. Remove the old instance only after that.

Update a single instance

1. Stage the target release without activating it.
2. Create the required backup and verify its restore.
3. Run target preflight and require exit code 0.
4. Start the maintenance window only after the plan is ready.
5. Activate the exact target artifact that passed preflight.
6. Watch the protected status:

   nucli --tenant system system product-release status --watch

1. Return the installation to service only after /health/ready returns HTTP 200.

Release bundles automate this sequence:

# Docker Compose
./start.sh backup
./start.sh update git.schukai.me/releases/nucleus@sha256:<digest>

# DEB or RPM: package installation only stages the release
sudo apt install ./nucleus-server_<version>-1_amd64.deb
# or: sudo rpm -Uvh nucleus-server-<version>-1.x86_64.rpm
sudo nucleus-server-backup
sudo nucleus-server-update \
  --target-version <version> \
  --backup-evidence /var/lib/nucleus/backups/<evidence>.json

# Kubernetes
./update.sh \
  git.schukai.me/releases/nucleus@sha256:<digest> \
  /secure/backup-evidence.json

Docker and Kubernetes require an immutable image digest. Native package installation never starts the new release or migrates the database.

/health/live remains available while Workspace reconciles the release contract. /health/ready returns HTTP 503 until verification succeeds. /health/product-release exposes the limited release state. All three endpoints remain available after traffic promotion; /health/ready and /health/product-release then return HTTP 200 and report the verified release status succeeded.

Browsers can continue to load already published public sites and the administration sign-in page during this period. Workspace serves only the published live artifact through GET and HEAD. Preview modes, builds, analytics recording, APIs, and write operations remain unavailable. An unresolved site receives a localized maintenance page with HTTP 503. Explicit JSON clients continue to receive the limited maintenance status.

Workers, analytics, SFTP and SMTP runtimes, APIs, and write-capable business routes start only after verification. A visible administration shell does not mean that Workspace is ready for writes. Use /health/ready as the release criterion. Already published sites and the sign-in page remain visible, but a single instance cannot process business work during this period.

A routine update does not require numin ... --apply, system-init apply, direct database access, or an operator guessing an ordered maintenance procedure from release notes.

The protected status reports transient operational failures as retry_wait; Workspace retries the fenced release run automatically with bounded backoff. blocked identifies a product-state conflict, while failed identifies a contract failure in the running binary. These terminal states require a correction and are not hidden behind endless retries.

What Workspace verifies automatically

The build-bound contract covers the supported schema and contract window, global system artifacts, workflow library, Taskstream catalog, executors, schedulers, producer sources, and positive tenant and IAM revision anchors. It also compiles every pinned workflow version, managed binding, active tenant override, and effective system fallback for every non-deleted tenant. This prevents an invalid workflow from surfacing only when the first business operation uses it. Skipped intermediate releases do not matter when the target binary explicitly lists the last completely verified source release and its complete schema transition chain. Unknown or unpublished source states fail closed.

Workspace completes the full preflight before it changes product state. It then applies only operations declared as automatic_safe. Persistent checkpoints, leases, and fence tokens make the process idempotent across concurrent instances. Every component uses the same tenant snapshot. Incompatible old nodes stop accepting work, wait for active Taskstream and workflow artifact attempts, and enter drained before mutation starts.

A new workflows module version does not update existing tenants in the background. To give an existing tenant the current managed workflow selection, explicitly start the provisioning run offered for that purpose. Workspace binds the run to its current module target. It neither rewrites nor silently continues an older completed run.

Inspect a blocked update

Existing invalid revisions, inactive producer sources, unknown persisted Taskstream types, uncompilable workflows, and incompatible customer overlays remain unchanged.

nucli --tenant system system product-release status --json

The protected status endpoint remains available through the product release gate. It reports the run, retry time, tenant snapshot, components, runtime evidence, blockers, owning area, and next action without exposing SQL, driver, path, or secret details. It also reports schemaBaseline, minimumRuntimeBaseline, schemaTransitionClass, schemaTransitionManifestDigest, pointOfNoReturnAt, and rollbackAfterSchemaTransition. numin remains the local read-only diagnostic for a privileged operating environment; use nucli whenever the server is reachable.

An agent reads the binding workflow with nucli skills show deployment. An MCP client can then call nucleus_deployment_assess with normalized, non-secret status fields. The local assessment returns versionAction, recoveryMode, next steps, and stop conditions, but grants no approval and changes neither the pin nor the database. Do not pass tokens, passwords, credential files, or unfiltered logs.

The first installation does not have a login session yet. Pass the System API key from the SYSTEM INITIALIZED log block through stdin only:

read -r -s NUCLEUS_BOOTSTRAP_TOKEN
printf '%s\n' "$NUCLEUS_BOOTSTRAP_TOKEN" |
  nucli \
    --host https://workspace.example.com \
    --tenant system \
    system product-release status \
    --watch \
    --token-stdin
unset NUCLEUS_BOOTSTRAP_TOKEN

--token-stdin skips login and optional discovery, stores no token, and still uses server-side System tenancy and diagnostics permissions.

Resolve the reported product conflict through its supported contract, then restart the same binary. Reconciliation resumes from verified checkpoints.

If sign-in and server status are unavailable, use the matching target tool locally:

/path/to/target/numin \
  --config /path/to/nucleus.config.enc \
  deployment plan --artifact-digest <sha256> \
  --backup-evidence /secure/backup-evidence.json --json
/path/to/target/numin \
  --config /path/to/nucleus.config.enc \
  schema migrations status --json

Never substitute numin from an older installation.

Critical runtime-only defects

A signed OCI hotfix may use the faster Emergency lane only when its manifest declares stateChangeClass: runtime_only and leaves Product Release, schema, configuration, and storage unchanged. Signature verification, canary checks, and production approval remain separate gates. The Emergency lane does not apply to DEB, RPM, or standalone binaries.

Follow Recover from critical runtime failures safely to verify the image and manifest, run the target DeploymentPlan, test fresh API and browser sessions, and confirm the exact return path.

New tenants

New tenants automatically use standard.v1 with iam, workflows, crm, localization, commerce, pim, and forum. Workspace creates positive authority anchors before the first producer or job artifact. Business routes remain unavailable until every profile step has been verified.

POST /api/v1/system/tenants creates the tenant, owner binding, and durable provisioning run atomically. Automation must send an Idempotency-Key. Workspace returns HTTP 202, Location, Retry-After, the tenant, the run, and a status URL. Reusing the key with the same normalized request returns the same run; reusing it for another request returns HTTP 409.

System administrators can inspect a tenant with nucli --tenant system system tenant-provisioning status <tenant-id> or GET /api/v1/system/tenants/{tenant-id}/provisioning. Interrupted runs resume in bounded pages with persistent backoff and retain the same run. GET /api/v1/system/tenants?include=provisioning adds each latest status to a paginated tenant list with one batch query. Business routes remain unavailable until the run reaches succeeded.