Documentation · The Lagstyr handbook

Backups, restoration and recovery

Backups become a recovery capability when a second operator can retrieve the keys, rebuild a supported target and restore verifiable data without the original host. A successful upload or a green timer is only one step. Use the installed kit’s infra/backup/README.md for exact environment inputs and drill commands.

Protect the independent data paths

When backup.required=true, the NixOS module wires physical backup, WAL checking, encrypted dumps, document copy and retention timers. PostgreSQL’s configured archive timeout is 300 seconds; the WAL check runs every five minutes. These settings do not establish a measured recovery point.

Verify that backup and document mirror destinations are independent of the primary failure domain. The current acceptance contract requires the mirror’s approved AU residence and retention. Scope upload, retention-delete and lifecycle-administration credentials separately. Keep the pgBackRest configuration private: it normally includes repository credentials and a cipher passphrase as well as paths.

Recover the keys before declaring recovery operational

Generate the age identity offline and install only its public recipient. Keep the private identity in at least two durable independent locations under two-person custody. Losing it makes the encrypted logical backups unrecoverable.

At each drill, retrieve the identity into a private short-lived location and use infra/backup/record-key-recovery.sh to bind its checksum, custodian, witness, source and retrieval time. The restore drill rejects a record older than 24 hours, a checksum mismatch or the same person in both roles. Remove the temporary identity after use according to your secure handling procedure; retain the checksum/ceremony evidence without the private key.

PITR additionally needs a fresh media attestation for the recovered pgBackRest configuration, distinct custodians and primary/repository failure domains. Protect and escrow that configuration and cipher material independently. An age key alone cannot decrypt a pgBackRest repository.

Where customer-lane OAuth is enabled, backups hold its refresh credentials, access tokens and in-flight exchange material only as ciphertext. Escrow integration-token-sealing-key and integration-token-delivery-key beside the age identity. Losing either costs no record data. Without the sealing key, or on a restore to a host with a different one, every refresh credential fails with integrity_error and every OAuth integration must be reconnected; revoke the old grants in each provider’s console. Losing the delivery key means installing one new key for both the kernel and the token broker and restarting them together: refresh credentials survive, stored access tokens are replaced, connectors retry through brief 503 answers, and an in-flight consent must begin again.

Exercise the restoration paths

Perform drills on isolated disposable targets, with no production ingress or enabled external effect workers. Use the exact release/PostgreSQL versions and configuration required by the media. Keep the primary and its credentials unavailable for the host-loss exercise.

1. Logical restore: pg_restore_drill.sh retrieves and decrypts the offsite dump, restores strictly into scratch, verifies representative record floors, migration head and a known recent event, then runs the released rebuild binary and database reindex. Retain timings and cleanup proof.
2. PITR: pitr_restore_drill.sh restores offsite physical media to an exact target time and verifies before/after markers, measured WAL recovery point, migration head, rebuild and reindex.
3. Host loss: host_loss_drill.sh requires a clean tag-built target attestation with the primary unavailable, then retains the combined recovery point and recovery time.
4. Documents: document_recovery_drill.sh retrieves a known mirrored object and verifies its source-recorded size and SHA-256 plus the required object lifecycle.
5. Alerting and retention: retain real success, failure and absence-of-signal evidence for logical backup, physical backup and WAL monitors, and an applied retention result.

The kit’s scripts/prod-backup-restore-rehearsal.sh derives the expected migration head from the activated release and orchestrates the logical production-media rehearsal; the backup README defines the other required reports. Follow each script’s documented required environment rather than substituting lower record floors or skipping rebuild.

Retain exact release/media identities, database/object sizes, restored recent-event evidence, versions, failures and download/restore/rebuild/reindex/total times. RPO is the measured recoverable data gap; RTO includes rebuilding dependencies and verifying service, not just importing a dump. Retain current-head representative independent-offsite evidence before cutover and quarterly thereafter. single-host-emulation validates mechanics and cannot satisfy independent-offsite acceptance or justify an availability promise.

Restore service deliberately

For a real incident, agree the target recovery point with the incident owner and record the data loss boundary. Rebuild the approved target, restore database and document media, verify schema and record integrity, reapply confined roles and reconcile configuration before activation. Rebuild disposable projections from their authoritative inputs.

Use infra/lifecycle/replay-customer-configuration.sh to verify and recover the encrypted installation contract and setup manifest. It restores files; reapplication remains the ordinary lagstyr-setup-bootstrap ceremony. Restore access and authority through supported procedures.

Before enabling outward work, reconcile external intents/receipts against providers for the lost interval. A provider may have performed an action after the selected database recovery point. Do not blindly replay restored queues. Verify identity, required workers, preflight, smoke, retrieval, backups and paging, then release business work in a reviewed order.

Keep retention and erasure truthful

The dump policy retains 14 daily, 8 weekly and 12 monthly points. Production deletion requires reviewed RETENTION_APPLY=1; dry-run status is not applied-retention evidence. The document copy requires an expiring lifecycle on docs-mirror/ with the contract’s 365-day bound; physical backup retention follows the approved PITR window.

Aggregate backups and immutable anchors can outlive primary deletion. Use lifecycle/subject reports to retain the remaining homes, holds and expiry evidence. Do not shorten policy by deleting media ad hoc or report global erasure from a missing primary row. Use Responding to incidents for the recovery decision and communication record.