fix(deploy): resolve FDW bootstrap race + extract bootstrap DO block to .sql file #499

Merged

lekss361 merged 1 commit from fix/deploy-fdw-ordering into main

2026-05-24 09:46:00 +00:00

lekss361 commented

2026-05-24 09:39:10 +00:00

Owner

Summary

Follow-up к PR #493 — три фикса из incident 2026-05-24 (deploy/1156 failed at bootstrap; tradein-backend USER MAPPING не создалась; main postgres не попал в gendesign_shared network). Все три проблемы починены вручную, этот PR делает workflow self-healing.

Файлы (3)

Файл	Что
ops/db-bootstrap/set_tradein_fdw_password.sql (new, 27 lines)	DO block extracted из inline psql heredoc; принимает password через psql -v pw=...; idempotent (IF EXISTS → ALTER, ELSE → NOTICE)
.forgejo/workflows/deploy.yml (+19/-10)	Bootstrap step → psql -v pw=$ENV -f ops/...; после compose up -d — verify postgres в gendesign_shared, force-recreate --no-deps postgres если нет
.forgejo/workflows/deploy-tradein.yml (+11)	docker restart tradein-backend после SQL migrations loop → lifespan hook retry

Что фиксим (root causes)

Bug A — deploy-tradein.yml ordering: compose up -d (line 157) запускает backend, который вызывает ensure_fdw_user_mapping → пробует CREATE USER MAPPING FOR ... SERVER gendesign_remote. Но FDW SERVER создаётся только потом, в 060_postgres_fdw_extension.sql (line 163). Hook падает psycopg.errors.UndefinedObject: server "gendesign_remote" does not exist и больше не дёргается. Result: USER MAPPING absent, все FDW queries fail.

Fix: docker restart tradein-backend после migrations loop → lifespan startup retry, server теперь существует, mapping создаётся.

Bug B — deploy.yml bootstrap DO block: inline psql -c с DO ... EXECUTE format(...) — multi-line backslash continuations внутри double-quoted bash string + psql var substitution = fragile escaping. deploy/1156 упал между migration apply (09:01:03) и compose up -d. Подозреваю DO block.

Fix: extract в ops/db-bootstrap/set_tradein_fdw_password.sql, выполнять через psql -v pw="$GENDESIGN_FDW_PASSWORD" -f . Clean, psql-native, никакого bash escape voodoo. Сам SQL остался idempotent (DO block с IF EXISTS).

Bug C — silent network drift: docker-compose.prod.yml уже имеет правильный networks: default + shared (alias gendesign-postgres), но compose up -d иногда не пересоздаёт postgres при изменении networks:. И в deploy/1156 up -d вообще не дошёл (упал на B). Result: gendesign-postgres-1 остался в gendesign_default only → tradein-postgres не мог резолвить DNS gendesign-postgres.

Fix: после compose up -d проверять docker inspect ... | grep gendesign_shared; если нет — compose up -d --force-recreate --no-deps postgres.

Test plan

Manual verify в production after deploy: docker inspect gendesign-postgres-1 показывает gendesign_shared=[gendesign-postgres] alias
docker exec tradein-postgres psql -U tradein -d tradein -c "SELECT count(*) FROM gendesign_cad_buildings" → ~36k (FDW работает)
tradein-backend logs: created FDW user mapping for gendesign_remote или refreshed FDW user mapping password
gendesign-postgres logs: tradein_fdw_reader password set NOTICE из bootstrap step
Smoke test estimate с адресом Хохрякова 81 → n_analogs > 0, confidence != address_not_geocoded

Runbook

Vault entry runbooks/deploy_fdw_cross_stack_network.md с healthy-state checks, симптомами и recovery procedures (network, USER MAPPING, password rotation).

Closes (по сути) incident 2026-05-24 deploy/1156.

## Summary Follow-up к PR #493 — три фикса из incident 2026-05-24 (deploy/1156 failed at bootstrap; tradein-backend USER MAPPING не создалась; main postgres не попал в gendesign_shared network). Все три проблемы починены вручную, этот PR делает workflow self-healing. ## Файлы (3) | Файл | Что | |---|---| | ops/db-bootstrap/set_tradein_fdw_password.sql (new, 27 lines) | DO block extracted из inline psql heredoc; принимает password через psql -v pw=...; idempotent (IF EXISTS → ALTER, ELSE → NOTICE) | | .forgejo/workflows/deploy.yml (+19/-10) | Bootstrap step → psql -v pw=$ENV -f ops/...; после compose up -d — verify postgres в gendesign_shared, force-recreate --no-deps postgres если нет | | .forgejo/workflows/deploy-tradein.yml (+11) | docker restart tradein-backend после SQL migrations loop → lifespan hook retry | ## Что фиксим (root causes) **Bug A — deploy-tradein.yml ordering**: compose up -d (line 157) запускает backend, который вызывает ensure_fdw_user_mapping → пробует CREATE USER MAPPING FOR ... SERVER gendesign_remote. Но FDW SERVER создаётся только потом, в 060_postgres_fdw_extension.sql (line 163). Hook падает psycopg.errors.UndefinedObject: server "gendesign_remote" does not exist и больше не дёргается. Result: USER MAPPING absent, все FDW queries fail. **Fix**: docker restart tradein-backend после migrations loop → lifespan startup retry, server теперь существует, mapping создаётся. **Bug B — deploy.yml bootstrap DO block**: inline psql -c с DO $$ ... EXECUTE format(...) — multi-line backslash continuations внутри double-quoted bash string + psql var substitution = fragile escaping. deploy/1156 упал между migration apply (09:01:03) и compose up -d. Подозреваю DO block. **Fix**: extract в ops/db-bootstrap/set_tradein_fdw_password.sql, выполнять через psql -v pw="$GENDESIGN_FDW_PASSWORD" -f <file>. Clean, psql-native, никакого bash escape voodoo. Сам SQL остался idempotent (DO block с IF EXISTS). **Bug C — silent network drift**: docker-compose.prod.yml уже имеет правильный networks: default + shared (alias gendesign-postgres), но compose up -d иногда не пересоздаёт postgres при изменении networks:. И в deploy/1156 up -d вообще не дошёл (упал на B). Result: gendesign-postgres-1 остался в gendesign_default only → tradein-postgres не мог резолвить DNS gendesign-postgres. **Fix**: после compose up -d проверять docker inspect ... | grep gendesign_shared; если нет — compose up -d --force-recreate --no-deps postgres. ## Test plan - [ ] Manual verify в production after deploy: docker inspect gendesign-postgres-1 показывает gendesign_shared=[gendesign-postgres] alias - [ ] docker exec tradein-postgres psql -U tradein -d tradein -c "SELECT count(*) FROM gendesign_cad_buildings" → ~36k (FDW работает) - [ ] tradein-backend logs: created FDW user mapping for gendesign_remote или refreshed FDW user mapping password - [ ] gendesign-postgres logs: tradein_fdw_reader password set NOTICE из bootstrap step - [ ] Smoke test estimate с адресом Хохрякова 81 → n_analogs > 0, confidence != address_not_geocoded ## Runbook Vault entry runbooks/deploy_fdw_cross_stack_network.md с healthy-state checks, симптомами и recovery procedures (network, USER MAPPING, password rotation). Closes (по сути) incident 2026-05-24 deploy/1156.

lekss361 added 1 commit 2026-05-24 09:39:10 +00:00

fix(deploy): resolve FDW bootstrap race + extract bootstrap DO block to .sql file 58655b2971

Three related fixes from PR #493 deploy incident (deploy/1156 failed at
bootstrap; tradein-backend USER MAPPING never created):

1. deploy-tradein.yml — restart tradein-backend AFTER SQL migrations apply.
   Root cause: `compose up -d` runs at line 157 before migrations at line 163,
   so tradein-backend lifespan hook (ensure_fdw_user_mapping) tries to
   CREATE USER MAPPING against FOREIGN SERVER gendesign_remote that doesn't
   exist yet (created by 060_postgres_fdw_extension.sql later). Hook fails
   permanently with `psycopg.errors.UndefinedObject: server "gendesign_remote"
   does not exist`; never retried. Restart after migrations triggers retry.

2. deploy.yml — extract inline psql DO block into ops/db-bootstrap/
   set_tradein_fdw_password.sql, run via `psql -v pw=$ENV -f <file>`.
   Original block used multi-line backslash continuations inside double-quoted
   bash string with $$-dollar-quoting + psql var substitution — fragile
   escaping likely caused deploy/1156 failure between migration apply (09:01:03)
   and compose up -d. Clean .sql file with `-v pw=` substitution is safe and
   psql-native.

3. deploy.yml — verify postgres in gendesign_shared after `compose up -d`,
   force-recreate if not. Compose sometimes skips recreate on network membership
   change (PR #493 introduced `networks: shared (alias gendesign-postgres)` to
   postgres service). Without postgres in shared, tradein-postgres can't resolve
   DNS `gendesign-postgres` -> FDW queries fail with "could not translate host
   name". Manual fix on prod: `docker network connect --alias gendesign-postgres
   gendesign_shared gendesign-postgres-1`.

No code changes — only deploy workflow and one new SQL file.

lekss361 merged commit ed49f6b953 into main

2026-05-24 09:46:00 +00:00

lekss361 referenced this pull request from a commit

2026-05-24 09:46:01 +00:00

fix(deploy): resolve FDW bootstrap race + extract bootstrap DO block to .sql file (#499)

lekss361 commented

2026-05-24 09:46:52 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE (squash → ed49f6b).

Bug A / B / C analysis:

Bug A (deploy-tradein.yml restart ordering): correct placement after migrations loop, before Caddy reload + health check. sleep 5 buffer + 30-iter healthcheck loop handles startup time. Lifespan exception swallowed → degraded but not crashed (acceptable per existing design).
Bug B (extracted SQL DO block): :'pw' + format(... %L ...) is double-safe. Backend _PASSWORD_RE = ^[A-Za-z0-9_\-]{32,256}$ whitelist is additional defense-in-depth. Bash guard if [ -n "${GENDESIGN_FDW_PASSWORD:-}" ] prevents unset-var psql error. Stdin redirect < file.sql works identically to -f for -v substitution (psql preprocesses on input stream).
Bug C (network drift verify + force-recreate): docker inspect ... 2>/dev/null | grep -q gendesign_shared self-heals fresh deploys (empty stdout → grep returns 1 → force-recreate fires). After force-recreate, aliases: [gendesign-postgres] from docker-compose.prod.yml:36-40 is restored. SQLAlchemy pool_pre_ping=True handles connection drop on app side; postgres_fdw opens fresh per-query connection on tradein side.

Follow-ups (non-blocking, NEW PR optional):

ops/db-bootstrap/set_tradein_fdw_password.sql header comment says -f file.sql but deploy.yml uses < file.sql (stdin). Functionally identical, but doc drift. Update comment to match (or switch deploy.yml to -f).
deploy.yml hardcodes gendesign-postgres-1 container name. Safer: pg_cid=$(docker compose -p gendesign -f docker-compose.prod.yml ps -q postgres). Current code correct for compose v2 default naming.

Post-merge actions: deploy.yml will auto-trigger. Monitor in Forgejo Actions UI. Verify:

docker inspect gendesign-postgres-1 --format '{{json .NetworkSettings.Networks}}' shows gendesign_shared alias
docker exec tradein-postgres psql -U tradein -d tradein -c "SELECT count(*) FROM gendesign_cad_buildings" → ~36k
tradein-backend logs: "created/refreshed FDW user mapping"
Smoke estimate Хохрякова 81 → confidence != address_not_geocoded

Merged via deep-code-reviewer — verdict APPROVE (squash → `ed49f6b`). **Bug A / B / C analysis**: - Bug A (deploy-tradein.yml restart ordering): correct placement after migrations loop, before Caddy reload + health check. `sleep 5` buffer + 30-iter healthcheck loop handles startup time. Lifespan exception swallowed → degraded but not crashed (acceptable per existing design). - Bug B (extracted SQL DO block): `:'pw'` + `format(... %L ...)` is double-safe. Backend `_PASSWORD_RE = ^[A-Za-z0-9_\-]{32,256}$` whitelist is additional defense-in-depth. Bash guard `if [ -n "${GENDESIGN_FDW_PASSWORD:-}" ]` prevents unset-var psql error. Stdin redirect `< file.sql` works identically to `-f` for `-v` substitution (psql preprocesses on input stream). - Bug C (network drift verify + force-recreate): `docker inspect ... 2>/dev/null | grep -q gendesign_shared` self-heals fresh deploys (empty stdout → grep returns 1 → force-recreate fires). After force-recreate, `aliases: [gendesign-postgres]` from docker-compose.prod.yml:36-40 is restored. SQLAlchemy `pool_pre_ping=True` handles connection drop on app side; postgres_fdw opens fresh per-query connection on tradein side. **Follow-ups (non-blocking, NEW PR optional)**: 1. `ops/db-bootstrap/set_tradein_fdw_password.sql` header comment says `-f file.sql` but `deploy.yml` uses `< file.sql` (stdin). Functionally identical, but doc drift. Update comment to match (or switch deploy.yml to `-f`). 2. `deploy.yml` hardcodes `gendesign-postgres-1` container name. Safer: `pg_cid=$(docker compose -p gendesign -f docker-compose.prod.yml ps -q postgres)`. Current code correct for compose v2 default naming. **Post-merge actions**: deploy.yml will auto-trigger. Monitor in Forgejo Actions UI. Verify: - `docker inspect gendesign-postgres-1 --format '{{json .NetworkSettings.Networks}}'` shows `gendesign_shared` alias - `docker exec tradein-postgres psql -U tradein -d tradein -c "SELECT count(*) FROM gendesign_cad_buildings"` → ~36k - tradein-backend logs: "created/refreshed FDW user mapping" - Smoke estimate Хохрякова 81 → confidence != address_not_geocoded