Documentation · The Lagstyr handbook
Upgrading and managing releases
An operator chooses each release. CI publishes immutable artifacts and deploys nothing; the host pulls, verifies and applies the selected tag. There is no deploy timer or silent self-update. Emergency security updates use the same verified path.
Review the change before the window
Read the intended release’s api/support-policy.json and docs/SUPPORTED-VERSIONS.md. Confirm that
it names an installable currentRelease, that the existing installation is an admitted upgrade
source, and that the complete signed artifacts and recovery evidence exist. At this review the
policy’s currentRelease is none; the standard production upgrade channel is therefore inert.
A complete candidate release is not a supported upgrade destination.
For an admitted change, record the source/target release, configuration digest, migration delta, security changes, API/task compatibility, selected services and rollback/recovery decision. Review customer integration manifests’ Lagstyr ranges and tested provider versions, agent qualifications, embedding generations and any enabled overlay. An overlay needs its release-bound compatibility manifest, untouched core digest, disjoint migration band and customer task contracts; preflight blocks uncertainty.
Test the same source→target path in a separate representative installation. Include live service startup, governed work, documents, queues, external receipt handling and the recovery branch your plan depends on. Stub provider tests do not establish customer-account acceptance. Check new configuration against the tagged schema and agree how in-flight external work will be drained or reconciled during the window.
When a release changes the integration effect send protocol, pause new connector claims before the migration. Drain or reconcile every disclosed provider handoff, stop the old connector runtimes and prove they hold no cached provider-ready payload. If that proof is unavailable, rotate or revoke the provider credential. The compatibility schema blocks an N-1 binary from creating a fresh legacy handoff and lets historical legacy rows complete or reconcile only; it cannot recall payload already disclosed outside the database.
When a release moves the built-in effect workers onto a changed send fence, stop email_outbox,
send_push_alert, repo_action, journal_poster and stripe_actuator before starting the deploy,
and wait until no queue lease they held is still live. Let calendar_worker settle every calendar
effect it is dispatching, then stop it. The migrations refuse to run while either kind of work is in
flight. The deploy’s reapplied role matrix removes the workers’ table privileges on the intent
ledger, so an older effect worker holds no privilege to claim a send: effects stop rather than being
sent twice.
When a release adds a worker to a pack, the pack’s declared contents change, so every worker of that
pack refuses to start until an operator re-consents, not only the worker that joined. Apply the
migrations, then propose and approve revise_pack for that pack before restarting the units, and
confirm the recorded contents afterwards. An installation that never activated the pack must stop
running the added worker instead: it refuses to start, and preflight fails while it remains in the
configured service list, so remove it there and let the module disable its unit. From this release
github_projector belongs to software_delivery_governance, whose other worker is repo_action;
docs/OPERATOR-RUNBOOK.md §2 carries the procedure.
Verify offsite backup destination, available space, recent WAL/base backup and escrow access before approval. Every deployment requires an encrypted pre-migration snapshot. Allocate the maintenance window from measured restore/rebuild time, and notify the accountable work owners.
Apply the exact tag
For a release-only update on the standard host, start the installed oneshot unit:
sudo systemctl start [email protected]
sudo systemctl status [email protected]
sudo journalctl -u [email protected]
The installed kit also provides infra/deploy/pull-release.sh TAG for the documented interactive
deploy environment. Use the installer/module workflow when installation configuration changes;
keep the customer JSON, module tag/service selection and effective unit wiring in agreement.
Editing JSON alone does not regenerate systemd configuration.
The pull path serialises deployment, resolves tag/commit through GitHub, verifies the manifest, publisher attestations, digests, provenance/SBOM subjects and versions, then takes the snapshot and stages a complete immutable release. It applies forward migrations with the owner DSN, reapplies the confined role matrix and supported installation policy, and authenticates as every actual runtime login to prove hostile catalogue, trigger, ownership and privilege-escalation attempts are refused. It then performs activation preflight, seals the release and switches the runtime atomically. Restarts and version-aware smoke follow.
Do not install bare binaries, rewrite sealed tags or add an unverified hotfix updater. Release
creation follows the repository’s reviewed release pipeline; operators consume its results.
For multiple installations, customer-operated infra/fleet/fleetctl.sh rolls out the same path by
ring and stops at the first failure. Retain each install’s sanitised report and result.
Know what rollback can restore
| Failure | Recovery boundary |
|---|---|
| Download, verification or staging | Resolve the failed check and retry the same approved operation. Atomic runtime activation keeps an incomplete tree from becoming current. Retain evidence of any earlier completed steps. |
| Restart or smoke after activation | The deploy attempts automatic rollback of the immutable binary tree and runtime bindings. Verify the result; this does not reverse committed migrations or external effects. |
| Defective schema/data migration | Use an approved forward fix or restore the mandatory pre-migration snapshot, then rebuild and verify. Migrations are forward-only. |
| External effect during the window | Reconcile the provider’s actual outcome and retained intent/receipt. Restoring a database cannot undo an action performed in another system. |
When no migration shipped, a code-only rollback may be sufficient. If migrations shipped, their
expand/contract compatibility is required for old binaries, but a defective schema or data change
still needs the recorded forward-fix/restore decision. Pack re-consent is not schema compatibility
and expand/contract does not cover it: once revise_pack has recorded a pack’s new contents, an N-1
binary declares the old ones and refuses to start, every worker of that pack included. Roll the code
back before re-consenting, or re-consent again on the N-1 contents. Never hand-delete SQLx migration
rows, edit previously applied migration files or assume current switching restored the database.
Close the change with evidence
Verify the active tag, configured-versus-active release, passing preflight, kernel/console version, required-unit state, queue and selected-authority readiness, retrieval status and representative governed work. Confirm backup timers/WAL and pager delivery still function. Check integrations’ retained release pins and unresolved effects before allowing new work.
An evaluation-procedure change intentionally invalidates earlier qualification. The migration suspends affected current-contract agents, cancels pending work and requests cancellation of running work; it does not relabel old evidence as current. Re-run the attended evaluator against the new procedure and reactivate through the ordinary governed decision. Apply the same rule to custom workflow releases whose case or procedure pin changed.
Retain snapshot identity, deploy logs, preflight/smoke results, acceptance evidence and follow-ups with the change record. Update the as-built configuration and second-operator instructions. If acceptance fails, use recovery or incident response rather than declaring success from liveness alone.