Find slow requests with performance diagnostics

This operations guide helps you identify slow HTTP route groups and determine how much database access contributes to their latency. After the measurement, you have a privacy-conscious ranking that can guide a separately tested optimization.

Prerequisites

You need a valid nucli alias for the System Tenant and the system_diagnostics:read scope. Verify the context first:

nucli --tenant system whoami --scopes

Run measurements in production or staging only with explicit approval. The command reads metrics and does not change product data.

Measure routes

Choose a window with representative traffic:

nucli --tenant system performance routes --window 30s --min-requests 5 --limit 20

Add --json for automation. The command reads one cumulative snapshot at the start and another at the end. It filters low-volume routes and ranks the remaining routes by P95 latency. A server restart or regressing counter aborts the analysis; rerun it with a complete window.

P95 is an approximate histogram bucket boundary. A value prefixed with >= means the observation reached the unbounded bucket. DB AVG shows cumulative database time per request, while DB Q/REQ shows the average query count. Parallel queries can accumulate more database time than the HTTP wall time.

High HTTP latency with low database time points toward application work, external services, or response size. High database time or query counts guide you toward query plans, indexes, N+1 access, or connection-pool waits. Treat these signals as evidence for a separate endpoint change, not as an automatic repair.

Prometheus is optional

Workspace produces the metrics itself, so nucli performance routes does not need a Prometheus server. Add an external Prometheus-compatible collector only when you need retained time series, dashboards, or alerts.

The protected operations endpoint is /api/v1/system/metrics. Its route labels do not contain concrete request paths, query strings, SQL, identities, or tenants. It also provides database-pool counters and installation-wide queue depth, age, throughput, and failure metrics.

Tenant-related or credential-adjacent series are separated under /api/v1/system/metrics/restricted. Access requires the System Tenant and the additional system_observability_restricted:read scope. Grant that scope only to the technical diagnostics role. Use separate collector targets, credentials, and storage paths for the two endpoints; collectors must not merge these data planes later.

On nucleus-ingress, the two planes are exposed at /api/control-plane/metrics and /api/control-plane/metrics/restricted. Each path uses its own file-backed bearer token; the general control-plane token is not valid for either endpoint. A path returns 404 when its token file is not configured and 401 for a wrong token. Identical tokens or unreadable files prevent startup.

Minimize retained data

Retain high-resolution operations metrics for no more than 30 days and restricted tenant series for no more than 14 days. The external metrics store enforces these limits; Workspace does not retain time series. Dashboards and alerts may use only the metric families and bounded labels emitted by their endpoint. Raw paths, query values, hosts, identities, request IDs, SQL, table or column names, free-form error text, user-agent values, and secrets must not appear in labels or annotations.

Grafana Alloy may scrape both Prometheus endpoints separately and translate them into internal OpenTelemetry or remote-write pipelines. Do not enable generic HTTP auto-instrumentation unless its attributes comply with the same metadata contract.

Produce defensible capacity evidence

A successful load run or an observed tenant count is not capacity evidence by itself. Bind each result to the product, infrastructure, and load-generator revisions, test environment, load profile, synthetic cohort size, measurement window, and guardrails. Required scenarios cover a baseline, homogeneous ramp, mixed load, noisy neighbor, background import, and backup and restore.

During an approved synthetic load phase, let nucli sample application signals at a fixed interval:

nucli --tenant system performance capacity \
  --window 30m --interval 5s \
  --installation shopcloud-capacity-environment-01 \
  --infrastructure-source /path/to/infra-worktree \
  --infrastructure-repository ssh://git.example.invalid/shopcloud/infrastructure \
  --load-generator-source /path/to/generator-worktree \
  --load-generator-repository ssh://git.example.invalid/shopcloud/load-generator \
  --load-profile-version mixed-v1 \
  --load-profile-file /path/to/mixed-v1.json \
  --stage 20 --json

Both source paths must be clean detached checkouts of fixed commits; a branch checkout is rejected. They must not be changed or shared with another process during the measurement. Their normalized origin identity must match the explicitly supplied expected repository identity. nucli resolves repository and commit itself before the run, then checks them again after the final snapshot. That before/after check cannot detect a temporary change which is fully reverted before the final check, so exclusive fixed-commit checkouts are a prerequisite. The profile version comes from strictly parsed JSON, and its bytes are bound by a SHA-256 digest.

The sampling interval must be at least five seconds. A window is limited to 24 hours and 20,000 samples. Across the entire run, at most 4,096 distinct HTTP series identities are accepted, including series that have disappeared. Exceeding any of these limits aborts the measurement rather than compacting incomplete evidence.

The report contains the HTTP analysis, sampled pool maxima, pool-wait counter deltas, maxima and deltas for registered queues, and the count, average, and P95 of queue processing durations. If any snapshot lacks a required signal or the window contains no application traffic beyond the metrics scrape, the report status is incomplete and unknown names the gap. The technical nucleus-capacity-sample/v2 report remains with Operations. It contains measurements, provenance, completeness status, and unknown required values, but no approved guardrails and no pass/fail decision. A future producer for the Commercial handoff/v1 contract may accept only approved aggregates; raw samples and tenant identifiers must not enter that handoff. That producer and the approval of load profiles and guardrails are not part of this tool.

Only a downstream approved validator can decide whether every required value meets its guardrail. unknown must never count as a pass. Safe technical density must not exceed 70 percent of the last fully passing level. Commercial receives only aggregated, revision-bound evidence from at least ten synthetic tenants, never raw series or tenant identifiers. The result does not automatically define prices, packages, or SLA limits.