Documentation · The Lagstyr handbook
Responding to incidents
Start with the affected work and the last verified state. Record a case reference, time, release and configuration identity, symptoms, impact and an incident owner. Keep “unavailable”, “stale” and “unknown” distinct: each calls for a different investigation.
The console’s Incidents page, under Govern, lists the incidents that are still live with
their severity, category, status and owner. It is also where an inc_ reference on an
integration break leads: a break is owned by an incident, and that incident’s status is what
says whether the break is being worked. Closed incidents are not listed there.
Contain unsafe work and preserve evidence
For suspected security or privacy events, stop unsafe processing, revoke affected credentials and preserve the authoritative history and custody record. Use the narrowest effective containment: pause the integration, action class or AI system, withdraw the affected actor’s authority, or stop the implicated service when host-level containment is required. Follow the relevant supported emergency/governed procedure and record who acted and why.
Stopping a worker prevents further processing; it does not undo a provider effect, settle a receipt or repair authority. Pausing an integration is a re-authenticated operator act that records who paused it; it blocks new claims but allows exact retained-release completion and verification so in-flight outcomes can drain. Disconnecting a provider connection is equally direct: the connector loses its access token at once, and the token broker revokes and destroys the refresh token. When revocation is unconfirmed, remove the grant in the provider’s own console. Revocation must be checked at the API/transaction boundary as well as in the secret store. If the host itself may be compromised, use a clean administration/recovery environment rather than trusting local verification alone.
Preserve release/preflight identities, unit state, bounded operational evidence, affected task, intent and receipt references and integrity results in your controlled case store. Do not edit authoritative rows, delete migration history, discard failed deploy attempts or rewrite evidence to make a check pass. Share only reviewed, minimised diagnostics through the agreed support path.
Diagnose by boundary
| Symptom | First checks and next action |
|---|---|
| Kernel will not start | Compare supported PostgreSQL/pgvector versions, SQLx ledger checksums/head, confined login, exact role attestation and DML-key wiring. Re-run the password-authenticated confinement proof through the deploy/migrator procedure; do not give the service owner privileges. |
| Liveness succeeds, work fails | Check private readiness, required workers, queue age, selected authority and provider readiness. A responding process does not establish working dependencies. |
| Connector authentication/permission failure | Check exact key expiry/revocation, stable label, machine-role grant, worker registration and active manifest digest. Additional business authority remains separate. |
| Governed proposal refuses | Read the exact refusal and selected policy/mandate/grant/route requirements, cooling window and approver eligibility. Repair the reviewed authority chain through its supported seam. |
| Semantic/hybrid search refuses | Check configured mode, qualified profile, runtime admission/canary, active generation, index and coverage. Use explicit FTS where appropriate; do not substitute stub vectors. |
| Integration page fires | Open the console’s /integrations/{integration_key} page first; it reads the same kernel check as the page. Pause if wrong facts could follow, or propose a quarantine if facts already written are suspect. Open a dated remediation case to acknowledge; a blocked or refusing release is fixed by a new signed release. A stale-attention page is a kernel fault: check readiness and the kernel log. |
| OAuth connector gets no token | Read the provider connection on the integration page. oauth_connection_missing needs a connection; oauth_reconnect_required needs new consent for the active release. For oauth_refresh_failing, act on its failure code: invalid_grant and scope_mismatch need reconnection, client_secret_unavailable a restored secret file and a broker restart, integrity_error the right sealing key or a reconnect, and tenant_mismatch handling as a security event. A 503 integration_oauth_not_configured means the kernel lacks the delivery key. |
| Task or external receipt is stuck/unknown | Compare retained release/task/intent/idempotency key with the provider’s actual state. Complete verification or reconciliation before a new effect. |
| Projection is stale | Check its worker, queue, input availability and declared freshness. Rebuild from authoritative inputs after the cause is resolved. |
| Backup/WAL/day-root check fails | Inspect that job’s status and bounded evidence; verify media and integrity. Escalate failed recovery or history-integrity evidence instead of clearing the alarm. |
| Release activation or smoke fails | Verify automatic binary rollback, retained snapshot and migration state. Follow the recorded forward-fix/restore decision. |
Use Monitoring and routine operations for host/readiness checks, Upgrading and managing releases for deploy boundaries and Backups, restoration and recovery for media and restore acceptance.
Prepare a reviewable support bundle
The kit supplies infra/support/support-bundle.py. The customer previews the exact allowlisted
collection contract, provides attributed consent for its hash and case, then initiates collection.
The collector uses a dedicated customer-custodied diagnostics login with narrow column-level
reads; keep that login disabled outside the collection window. It is not a runtime or vendor DSN.
For example, choose a real case reference and a UTC expiry within the next 14 days, then preview:
python3 /opt/lagstyr/current/install-kit/infra/support/support-bundle.py preview \
--case-reference INC-example-0001 --expires-at "$case_expiry" \
--output /secure/evidence/preview.json
Define case_expiry from the incident’s agreed expiry before running that command. Review the
preview and create consent binding its case and contract hash. The collect phase requires that
consent, the case recipient, collector attribution, encrypted output and receipt paths; use
docs/SUPPORT-AND-INCIDENT-RESPONSE.md in the kit for the exact flow and secret bindings.
The bundle includes release/schema aggregates, queue and worker freshness aggregates and failed-unit names/states. It constructs an allowlisted object and does not collect raw logs, environment, credentials, task payloads or authoritative record tables. It encrypts immediately with age and retains a custody/expiry receipt. Your organisation reviews and controls transport; the product provides no automatic vendor export or vendor-pulled diagnostics endpoint.
Recover, verify and close
Use your agreed severity and escalation channel. The single-node operating tier carries no
availability-percentage promise; the support policy defines coverage and targets. Record the
communication owner, next update and any security/privacy assessment with the appropriate owners.
Vulnerability reports follow SECURITY.md; keep sensitive details in the agreed private channel.
Before resuming work, verify the cause is contained, compromised credentials are replaced and old ones refused, selected authority is effective, uncertain effects are reconciled and required services are ready. Test the affected business path and recovery/alert controls, then explicitly release the contained work. Retain the recovery point, any lost interval and residual risk.
Close with the incident timeline, evidence references, root cause, repair, validation and named follow-ups. Update the installation configuration, monitoring, connector tests and operating instructions when the incident exposed a gap. The handbook’s source-review gate requires a chapter review when its coupled implementation changes; use the repository’s documented review workflow when maintaining these instructions.