ops(backup): missed-run detection + Forgejo code/DB backup to S3 #3025

Merged

lekss361 merged 2 commits from feat/2203-backup-staleness-and-forgejo into main

2026-08-21 17:12:04 +00:00

bot-backend commented

2026-08-21 12:41:05 +00:00

Collaborator

Summary

Two narrow deliverables for #2203, refs #2989.

1. Missed-run detection for backups. ops/backup.sh and
tradein-mvp/deploy/backup-tradein-db.sh now write a sentinel file
(.last_success, UTC timestamp) on every run that passes all existing
guards -- nothing about their own hardening (min-size floor, gzip -t +
trailer integrity check, globals dump, retention) is touched. A new,
separate ops/check-backup-staleness.sh (own cron entry, suggested hourly)
checks a sentinel's age against a threshold (default 26h for a daily cron)
and alerts if it is stale or missing. Shared sentinel/notify/state logic
lives in ops/lib-backup.sh so it is not copy-pasted a third time.

Alert channel: reuses the existing Telegram bot from
ops/uptime-healthcheck.sh -- same TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID
variable names, read from /etc/default/gendesign-backup (deliberately a
different env file than uptime's /etc/default/gendesign-uptime, so backup
alerting does not depend on the uptime watchdog's config existing -- point
both at the same bot/chat if you want one destination). Alerts only on a
state transition (fresh->stale, stale->fresh), same discipline as
uptime-healthcheck.sh's prev_status/set_status, so an hourly cron
does not spam. No channel configured -> logs only, no new alerting system
invented. notify() in ops/lib-backup.sh is the single extension point
if a different channel is wanted later.

2. Forgejo backup -- did not exist at all. New ops/backup-forgejo.sh:

DB dump: pg_dump -U forgejo forgejo via docker exec against the
shared gendesign-postgres-1 container (Forgejo has no dedicated DB
container -- separate user+db in the main app's Postgres, per
infra/Forgejo_Migration_BotServer_To_Beget_2026-05-16.md in the vault).
Repo "bundle": tar -czf of the bare-repo tree under
FORGEJO_REPOS_DIR (default
/home/gendesign/forgejo/data/forgejo/git/repositories, matching the
documented Forgejo layout). Bare repos already ARE just refs+objects --
tar-ing the whole tree in one shot is a superset of what a per-repo
git bundle loop would capture, and needs zero changes when repos are
added/removed. LFS objects/avatars/attachments elsewhere under
FORGEJO_DATA_DIR are out of scope for this pass -- flagging as a known
gap, not silently dropped.
Same guards as ops/backup.sh: non-empty + min-size floor, integrity
check (gzip -t + trailer for the DB dump, tar -tzf for the bundle),
sentinel on success, non-zero exit on any failure.
Uploads to s3://gendsgn-backups/forgejo/ under a separate, narrower S3
service user -- NOT the existing gendsgn-backup-writer (which has
root-of-bucket write access and is used by the other two backups). Off-box
upload is mandatory here (unlike ops/backup.sh, where it is optional) --
the whole point of this script is getting Forgejo's code off the VM it
runs on.
The key does not exist yet. Without FORGEJO_S3_ENDPOINT /
FORGEJO_S3_BUCKET / FORGEJO_S3_ACCESS_KEY / FORGEJO_S3_SECRET_KEY
set (via /etc/default/gendesign-backup-forgejo, see the new
ops/gendesign-backup-forgejo.default.example), the script logs a loud
"NOT CONFIGURED" error and exits 1 -- it does not silently skip the
upload or fall back to local-only.

Bucket policy JSON for the new service user

For a human to create in the Selectel panel (bucket gendsgn-backups,
mirrors the existing gendsgn-backup-writer policy pattern documented in
vault meta/00_credentials.md section "Selectel S3 -- бэкап-бакет
gendsgn-backups" -- same Sid names/shape, scoped to the forgejo/ prefix
only). Replace <FORGEJO_WRITER_USER_ID> with the new service user's ID
once created, same way the existing writer/reader policy references their
own user IDs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ForgejoWriterPutOnly",
      "Effect": "Allow",
      "Principal": { "AWS": ["<FORGEJO_WRITER_USER_ID>"] },
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::gendsgn-backups/forgejo/*"
    },
    {
      "Sid": "ForgejoWriterNeverReadsOrDeletes",
      "Effect": "Deny",
      "Principal": { "AWS": ["<FORGEJO_WRITER_USER_ID>"] },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:ListBucket",
        "s3:ListBucketVersions",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion",
        "s3:DeleteBucket",
        "s3:PutBucketVersioning",
        "s3:PutBucketObjectLockConfiguration",
        "s3:BypassGovernanceRetention"
      ],
      "Resource": [
        "arn:aws:s3:::gendsgn-backups",
        "arn:aws:s3:::gendsgn-backups/*"
      ]
    }
  ]
}

Notes on the shape (matching the existing writer's documented rationale):

PutObject scoped strictly to .../forgejo/* -- this key cannot write
anywhere else in the bucket (so it cannot clobber the main/tradein
backups living at bucket root).
Multipart actions included because a repo tar.gz can exceed 5 GB.
ListBucket is explicitly denied even though it seems harmless -- a
compromised prod-adjacent host should not be able to enumerate what other
backups exist in the bucket either.
Deny statements are formally redundant under a default-deny policy but
guard against a future overly-broad Allow being added by mistake -- same
reasoning as the existing writer/reader policy.
This is s3.bucket.user-role-shaped, per the existing writer/reader
pattern (that role fails closed with no policy; s3.user would fail
open) -- create the new service user with that same role.

Effect on running stacks

None. Nothing here touches docker-compose.prod.yml, .forgejo/workflows/*,
or any running container config -- purely new/extended cron scripts. New
ops/*.sh files already match the deploy trigger glob ("ops/*.sh" in
.forgejo/workflows/deploy.yml) and its chmod +x ops/*.sh step, so no
workflow changes were needed; tradein-mvp/deploy/backup-tradein-db.sh is
already covered by the existing tradein-mvp/deploy/** trigger.

What a human needs to do after merge (not done here -- no prod SSH, no new creds created)

On the prod VM, add to /etc/default/gendesign-backup (chmod 600,
already exists): TELEGRAM_BOT_TOKEN=... / TELEGRAM_CHAT_ID=...
(reuse the uptime bot's values, or a separate bot -- either works).
Add cron entries for ops/check-backup-staleness.sh (see its header for
the exact 3 lines -- one per sentinel: main, tradein, forgejo).
In the Selectel panel: create the gendsgn-backup-forgejo-writer
s3.bucket.user service user, apply the policy JSON above (swap in the
real user ID), then create /etc/default/gendesign-backup-forgejo
(chmod 600) from ops/gendesign-backup-forgejo.default.example with the
real access/secret key.
Add a cron entry for ops/backup-forgejo.sh (see its header for the
suggested line) once step 3 is done -- until then it will run (if
cron'd early) and correctly fail loudly every time, which is expected.

Tests

Behavioral bash checks run locally (not wired into CI -- .forgejo/workflows/*
is out of scope per the issue, and there is no ops/tests/ convention in
this repo yet to hook into): stale sentinel -> nonzero + alert attempt;
fresh sentinel -> silent, exit 0; missing sentinel -> nonzero;
fresh->stale->stale->fresh transition sequence -> alerts only on the two
transitions, not the repeat; backup-forgejo.sh with no S3 vars -> loud
"NOT CONFIGURED" + nonzero, no docker/tar touched; backup-forgejo.sh with
S3 vars but a missing repos dir -> loud nonzero. All 11 assertions passed.

bash -n clean on all changed/new scripts (matches the existing CI syntax
gate in .forgejo/workflows/ci.yml, which globs ops/*.sh/ops/**/*.sh --
untouched, no workflow changes needed).

## Summary Two narrow deliverables for #2203, refs #2989. **1. Missed-run detection for backups.** `ops/backup.sh` and `tradein-mvp/deploy/backup-tradein-db.sh` now write a sentinel file (`.last_success`, UTC timestamp) on every run that passes all existing guards -- nothing about their own hardening (min-size floor, `gzip -t` + trailer integrity check, globals dump, retention) is touched. A new, separate `ops/check-backup-staleness.sh` (own cron entry, suggested hourly) checks a sentinel's age against a threshold (default 26h for a daily cron) and alerts if it is stale or missing. Shared sentinel/notify/state logic lives in `ops/lib-backup.sh` so it is not copy-pasted a third time. Alert channel: reuses the existing Telegram bot from `ops/uptime-healthcheck.sh` -- same `TELEGRAM_BOT_TOKEN`/`TELEGRAM_CHAT_ID` variable names, read from `/etc/default/gendesign-backup` (deliberately a different env file than uptime's `/etc/default/gendesign-uptime`, so backup alerting does not depend on the uptime watchdog's config existing -- point both at the same bot/chat if you want one destination). Alerts only on a state transition (fresh->stale, stale->fresh), same discipline as `uptime-healthcheck.sh`'s `prev_status`/`set_status`, so an hourly cron does not spam. No channel configured -> logs only, no new alerting system invented. `notify()` in `ops/lib-backup.sh` is the single extension point if a different channel is wanted later. **2. Forgejo backup -- did not exist at all.** New `ops/backup-forgejo.sh`: - DB dump: `pg_dump -U forgejo forgejo` via `docker exec` against the shared `gendesign-postgres-1` container (Forgejo has no dedicated DB container -- separate user+db in the main app's Postgres, per `infra/Forgejo_Migration_BotServer_To_Beget_2026-05-16.md` in the vault). - Repo "bundle": `tar -czf` of the bare-repo tree under `FORGEJO_REPOS_DIR` (default `/home/gendesign/forgejo/data/forgejo/git/repositories`, matching the documented Forgejo layout). Bare repos already ARE just refs+objects -- tar-ing the whole tree in one shot is a superset of what a per-repo `git bundle` loop would capture, and needs zero changes when repos are added/removed. LFS objects/avatars/attachments elsewhere under `FORGEJO_DATA_DIR` are out of scope for this pass -- flagging as a known gap, not silently dropped. - Same guards as `ops/backup.sh`: non-empty + min-size floor, integrity check (`gzip -t` + trailer for the DB dump, `tar -tzf` for the bundle), sentinel on success, non-zero exit on any failure. - Uploads to `s3://gendsgn-backups/forgejo/` under a separate, narrower S3 service user -- NOT the existing `gendsgn-backup-writer` (which has root-of-bucket write access and is used by the other two backups). Off-box upload is mandatory here (unlike `ops/backup.sh`, where it is optional) -- the whole point of this script is getting Forgejo's code off the VM it runs on. - The key does not exist yet. Without `FORGEJO_S3_ENDPOINT` / `FORGEJO_S3_BUCKET` / `FORGEJO_S3_ACCESS_KEY` / `FORGEJO_S3_SECRET_KEY` set (via `/etc/default/gendesign-backup-forgejo`, see the new `ops/gendesign-backup-forgejo.default.example`), the script logs a loud "NOT CONFIGURED" error and exits 1 -- it does not silently skip the upload or fall back to local-only. ## Bucket policy JSON for the new service user For a human to create in the Selectel panel (bucket `gendsgn-backups`, mirrors the existing `gendsgn-backup-writer` policy pattern documented in vault `meta/00_credentials.md` section "Selectel S3 -- бэкап-бакет gendsgn-backups" -- same Sid names/shape, scoped to the `forgejo/` prefix only). Replace `<FORGEJO_WRITER_USER_ID>` with the new service user's ID once created, same way the existing writer/reader policy references their own user IDs: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ForgejoWriterPutOnly", "Effect": "Allow", "Principal": { "AWS": ["<FORGEJO_WRITER_USER_ID>"] }, "Action": [ "s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts", "s3:ListBucketMultipartUploads" ], "Resource": "arn:aws:s3:::gendsgn-backups/forgejo/*" }, { "Sid": "ForgejoWriterNeverReadsOrDeletes", "Effect": "Deny", "Principal": { "AWS": ["<FORGEJO_WRITER_USER_ID>"] }, "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket", "s3:ListBucketVersions", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:DeleteBucket", "s3:PutBucketVersioning", "s3:PutBucketObjectLockConfiguration", "s3:BypassGovernanceRetention" ], "Resource": [ "arn:aws:s3:::gendsgn-backups", "arn:aws:s3:::gendsgn-backups/*" ] } ] } ``` Notes on the shape (matching the existing writer's documented rationale): - `PutObject` scoped strictly to `.../forgejo/*` -- this key cannot write anywhere else in the bucket (so it cannot clobber the main/tradein backups living at bucket root). - Multipart actions included because a repo tar.gz can exceed 5 GB. - `ListBucket` is explicitly denied even though it seems harmless -- a compromised prod-adjacent host should not be able to enumerate what other backups exist in the bucket either. - Deny statements are formally redundant under a default-deny policy but guard against a future overly-broad `Allow` being added by mistake -- same reasoning as the existing writer/reader policy. - This is `s3.bucket.user`-role-shaped, per the existing writer/reader pattern (that role fails closed with no policy; `s3.user` would fail open) -- create the new service user with that same role. ## Effect on running stacks None. Nothing here touches `docker-compose.prod.yml`, `.forgejo/workflows/*`, or any running container config -- purely new/extended cron scripts. New `ops/*.sh` files already match the deploy trigger glob (`"ops/*.sh"` in `.forgejo/workflows/deploy.yml`) and its `chmod +x ops/*.sh` step, so no workflow changes were needed; `tradein-mvp/deploy/backup-tradein-db.sh` is already covered by the existing `tradein-mvp/deploy/**` trigger. ## What a human needs to do after merge (not done here -- no prod SSH, no new creds created) 1. On the prod VM, add to `/etc/default/gendesign-backup` (chmod 600, already exists): `TELEGRAM_BOT_TOKEN=...` / `TELEGRAM_CHAT_ID=...` (reuse the uptime bot's values, or a separate bot -- either works). 2. Add cron entries for `ops/check-backup-staleness.sh` (see its header for the exact 3 lines -- one per sentinel: main, tradein, forgejo). 3. In the Selectel panel: create the `gendsgn-backup-forgejo-writer` `s3.bucket.user` service user, apply the policy JSON above (swap in the real user ID), then create `/etc/default/gendesign-backup-forgejo` (chmod 600) from `ops/gendesign-backup-forgejo.default.example` with the real access/secret key. 4. Add a cron entry for `ops/backup-forgejo.sh` (see its header for the suggested line) once step 3 is done -- until then it will run (if cron'd early) and correctly fail loudly every time, which is expected. ## Tests Behavioral bash checks run locally (not wired into CI -- `.forgejo/workflows/*` is out of scope per the issue, and there is no `ops/tests/` convention in this repo yet to hook into): stale sentinel -> nonzero + alert attempt; fresh sentinel -> silent, exit 0; missing sentinel -> nonzero; fresh->stale->stale->fresh transition sequence -> alerts only on the two transitions, not the repeat; `backup-forgejo.sh` with no S3 vars -> loud "NOT CONFIGURED" + nonzero, no docker/tar touched; `backup-forgejo.sh` with S3 vars but a missing repos dir -> loud nonzero. All 11 assertions passed. `bash -n` clean on all changed/new scripts (matches the existing CI syntax gate in `.forgejo/workflows/ci.yml`, which globs `ops/*.sh`/`ops/**/*.sh` -- untouched, no workflow changes needed).

bot-backend added 2 commits 2026-08-21 12:41:06 +00:00

feat(ops): missed-run detection for backups + Forgejo code/DB backup to S3 83be088263

Deliverable 1: ops/backup.sh and tradein-mvp/deploy/backup-tradein-db.sh now
write a sentinel file on every verified-good run. A new ops/check-backup-
staleness.sh (separate cron entry, hourly) alerts via the existing Telegram
channel (same TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID idiom as ops/uptime-
healthcheck.sh, transition-tracked so it doesn't spam) if a sentinel goes
stale. Shared logic (notify/sentinel/state) factored into ops/lib-backup.sh
so it isn't triplicated; the two existing scripts' own hardening (integrity
checks, retention, etc.) is untouched.

Deliverable 2: ops/backup-forgejo.sh — Forgejo (git.gendsgn.ru) has no backup
today. Dumps the shared-postgres `forgejo` DB + tars the bare-repo tree, both
off-box to s3://gendsgn-backups/forgejo/ under a SEPARATE, narrower S3 key
(root-of-bucket writer key stays out of this). The key doesn't exist yet —
the script refuses to run and exits non-zero, loudly, until the four
FORGEJO_S3_* vars are filled in (see the example env file and PR description
for the exact bucket policy JSON to create it with).

Refs #2203, #2989

fix(ops): restore +x on new ops/*.sh (exec bit lost in prior commit, Windows core.filemode=false)

CI Trade-In / changes (pull_request) Successful in 11s

Details

CI Trade-In / backend-tests (pull_request) Has been skipped

Details

CI / changes (pull_request) Successful in 13s

Details

CI Trade-In / browser-tests (pull_request) Has been skipped

Details

CI Trade-In / frontend-checks (pull_request) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped