Documentation · The Lagstyr handbook

Monitoring and routine operations

Operate the installation against its declared configuration and work budgets. Distinguish process liveness, dependency readiness, delegated authority and the freshness of projections. A dashboard with no error is not proof that business work is progressing.

Check the right surface

The default kernel operational listener is literal loopback 127.0.0.1:8090; /readyz and /metrics are credential-free there because the listener is private. Keep it out of reverse-proxy public routes. Use your configured addresses if the installation changes the defaults.

These checks run on the host without displaying environment files:

curl --fail --silent --show-error http://127.0.0.1:8080/healthz
curl --fail --silent --show-error http://127.0.0.1:8090/readyz
systemctl --failed 'lagstyr-*'
systemctl list-timers --all 'lagstyr-*'
cat /opt/lagstyr/current/.tag

Read the preflight JSON and failed-unit status alongside these responses. After an authorised change, run the installed release’s infra/deploy/post-deploy-smoke.sh in its documented deploy environment; it verifies exact versions, console health, bundle versions, executable paths for enabled units, and that no installed integration is blocked. That last check reads the kernel’s own attention check and needs no credential. It refuses an attention check older than ten minutes rather than trusting it, because a measurement that stopped being refreshed keeps its last value, and it reports that nothing was checked when the installation has no integration. It does not replace a live business acceptance test.

Monitor progress and uncertainty

Track queue depth and oldest age by task kind, dead-letter growth, retry patterns, worker freshness and receipt/read-back deadlines. A task awaiting approval can be a correct stop. An aging claimed task or unknown external effect needs diagnosis and reconciliation before replay.

A provider send that has started is not repeated automatically. The only exception is email_outbox, send_push_alert, repo_action, journal_poster or stripe_actuator proving non-acceptance: the connection was refused or the provider answered HTTP 429. A definitive provider rejection fails without retry, and a response lost after the request was written stops for adjudication. intents_sending_stale on /v1/integration-health counts started sends whose worker stopped and that no later claim has flagged. Nothing re-sends them; investigate each against the provider.

Compare the running fleet with services in the customer JSON and the required-worker/budget configuration. Do not infer success from an empty queue if the producer or required worker is missing. Treat “unavailable” as an unsuccessful read, not an empty result.

Monitor lagstyr_governed_skills_selected_broken and investigate any value above zero. Intentionally unselected catalogue skills are outside the reviewed authority profile; catalogue growth does not silently add them. Repair selected authority through its governed policy, mandate, grant and route seams instead of creating broad catch-all routes.

Watch database connections, free disk, WAL/archive age, document/chunk/event growth, CPU and memory against the configured capacity thresholds. Day-root verification and volume tripwires can fail lagstyr-day-root.service; retain and alert on the unit failure. A projection can be rebuilt; authoritative event history must be preserved and investigated if integrity checks fail.

Keep telemetry and paging inside the approved boundary

The kernel exposes Prometheus metrics on the operational listener. Rust and Python workers can push minimised logs/traces/metrics over OTLP to the configured loopback collector. Without a collector they retain local logging; collection/export is an explicit installation choice. An external sink requires the approved endpoint, credentials, egress, access and retention.

If Alloy is selected, provision the exact configured collector and sinks. Use the kit’s infra/grafana dashboards and rules as versioned assets, then prove that the rules query the installed metrics and reach your pager. The managed-heartbeat capability needs its protected systemd credential and required-worker/queue/monitor configuration. Verify failure and absence-of-signal delivery through the independent transport, including while the primary host is unavailable. A declared alert destination is not delivery evidence.

Installed integrations have their own signals. A kernel sweep publishes, every minute, lagstyr_integration_condition for each blocked release, refusing capture or overdue remediation that nothing covers, alongside covered conditions, readiness, the oldest overdue remediation age, lagstyr_capture_break_refusing for built-in adapters and a refresh timestamp; connector handlers count capture, refusal and effect-send outcomes. The kit’s rules page on each condition and on an attention refresh older than 10 minutes, which otherwise silences the rest, and the launch-critical dashboard carries three integration panels. These series are labelled with the installed integration_key, bounded to 200 keys, so the selected sink sees integration names but never records. Without a Grafana sink, page through the operational-assurance pack instead: one active integration monitor per key, the integration_probe unit with LAGSTYR_INTEGRATION_PROBE_MONITORS_JSON naming its monitors, and a routed alert channel. An installation with neither gets no integration page, only the console’s attention section.

Customer-lane OAuth adds one counter and no gauge. lagstyr_integration_access_tokens_total counts the connector’s token requests as served, unavailable or refused. A steady unavailable rate with no served means the token broker is not refreshing; routine rotation writes no event, so this counter is where it shows. Broker failures page through the existing release_blocked rule as the oauth_* findings. Its logs carry field-name markers and never token material: integration_oauth_exchange_reported, integration_oauth_refreshed, integration_oauth_refresh_failing, integration_oauth_revocation_reported and integration_oauth_integrity_refused. Include the broker in the required-worker comparison when it is enabled.

Governed monitors, thresholds and channels are commissioned installation data. Operational telemetry does not itself create a business fact, an outcome measurement contract or an approval.

Keep a maintenance record

Use this starting cadence and tighten it to your operating contract:

Retain sanitised release/configuration identities, measured results, owner and follow-ups. Review retention and erasure status; primary deletion does not mean backups or immutable anchors have expired. Use the kit’s lifecycle reports for those declared homes. Follow Upgrading and managing releases for changes and Responding to incidents for an unsafe or uncertain state.