gendesign/.github/workflows/deploy.yml
lekss361 7c43bc5496
feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150) (#151)
* fix(frontend): apiFetchWithStatus body stream double-read crash

Per user report 2026-05-14: 'Failed to execute text on Response: body stream
already read' при analyze с non-JSON error response.

apiFetchWithStatus делал .json() (consumes stream), потом .text() в catch
(FAILS — stream already consumed). Fetch API: body — ReadableStream,
consumable один раз.

Fix: read text() once → JSON.parse on string. На fail (HTML / non-JSON)
оборачиваем в {detail: rawText}. Никаких double-reads.

Site-finder cad search больше не крашится на error pages.

* feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150)

Закрывает root cause production 500 на /api/v1/admin/site-finder/weight-profiles
(а также 2 unapplied migrations: #89/#91).

## Why это нужно

После audit обнаружили что:
- backend/alembic/versions/ пустой (Alembic configured но не используется)
- deploy.yml не имеет migration step
- 47+ raw SQL files накопилось без auto-apply
- 3 файла unapplied: 87 (engineering), 89 (drop brin), 90 (weight profiles)

User report: GET weight-profiles → 500 потому что user_weight_profiles
table не существует на prod.

## Changes

### deploy.yml +39 lines
- Path trigger расширен на data/sql/*.sql
- New step 'Apply DB migrations' между compose pull и compose up -d:
  - Idempotent CREATE TABLE _schema_migrations
  - Loop по data/sql/*.sql sorted → skip applied → psql with ON_ERROR_STOP=on
  - Record applied filename в tracking table
  - Exit 1 на failure → containers НЕ обновляются (no partial state)

### NN collision fix
- 87_engineering_networks_191fz.sql → 91_engineering_networks_191fz.sql
  (collision с 87_on_demand_indexes.sql, мой renumber 85→87 был неудачным)

### CLAUDE.md +32 lines
- Subsection 'Auto-apply' под Migration pattern
- Bootstrap procedure docs
- Idempotency requirement
- Failure recovery

### scripts/bootstrap_schema_migrations.sh +100 lines
- One-time seed _schema_migrations с 48 already-applied files
- 3 файла intentionally NOT seeded: 89/90/91 (auto-apply на first deploy)
- Usage docs + verification query

## Vault

domains/infra/Runbook_Auto_Apply_Migrations.md NEW
infra-MOC.md updated

## Required manual steps ДО merge

1. SSH gendesign (tunnel)
2. POSTGRES creds export
3. bash scripts/bootstrap_schema_migrations.sh — seed 48 already-applied files
4. Verify: SELECT COUNT(*) FROM _schema_migrations → 48
5. После merge → first deploy auto-apply #89/#90/#91 → 500 closes

Closes #150

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:06:54 +03:00

281 lines
11 KiB
YAML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

name: Deploy
# Деплоится только при изменениях основного стека.
# Obsidian-стек (CouchDB) — отдельный workflow `deploy-obsidian.yml`.
on:
push:
branches: [main]
paths:
- "backend/**"
- "frontend/**"
- "docker-compose.prod.yml"
- "Caddyfile"
- ".github/workflows/deploy.yml"
- "data/sql/*.sql"
# NB: shared compose-fragments (network) — тоже триггерят main
# потому что Caddy шарит сеть с obsidian-stack
workflow_dispatch:
concurrency:
group: deploy-prod
cancel-in-progress: false
env:
IMAGE_BACKEND: ghcr.io/lekss361/gendesign-backend
IMAGE_WORKER: ghcr.io/lekss361/gendesign-worker
IMAGE_FRONTEND: ghcr.io/lekss361/gendesign-frontend
jobs:
changes:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
outputs:
backend: ${{ steps.filter.outputs.backend }}
frontend: ${{ steps.filter.outputs.frontend }}
infra: ${{ steps.filter.outputs.infra }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
backend:
- 'backend/**'
frontend:
- 'frontend/**'
infra:
- 'docker-compose.prod.yml'
- 'Caddyfile'
- '.github/workflows/deploy.yml'
build-backend:
runs-on: ubuntu-latest
needs: changes
permissions:
contents: read
packages: write
if: |
needs.changes.outputs.backend == 'true' ||
needs.changes.outputs.infra == 'true' ||
github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v4
- name: Login to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build & push backend (lean — без Chromium)
uses: docker/build-push-action@v6
with:
context: ./backend
target: runner
push: true
cache-from: type=gha,scope=backend-lean
cache-to: type=gha,mode=max,scope=backend-lean
tags: |
${{ env.IMAGE_BACKEND }}:latest
${{ env.IMAGE_BACKEND }}:${{ github.sha }}
build-worker:
runs-on: ubuntu-latest
needs: changes
permissions:
contents: read
packages: write
if: |
needs.changes.outputs.backend == 'true' ||
needs.changes.outputs.infra == 'true' ||
github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v4
- name: Login to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build & push worker (с Chromium для Playwright)
uses: docker/build-push-action@v6
with:
context: ./backend
target: runner-with-chromium
push: true
cache-from: type=gha,scope=worker-chromium
cache-to: type=gha,mode=max,scope=worker-chromium
tags: |
${{ env.IMAGE_WORKER }}:latest
${{ env.IMAGE_WORKER }}:${{ github.sha }}
build-frontend:
runs-on: ubuntu-latest
needs: changes
permissions:
contents: read
packages: write
if: |
needs.changes.outputs.frontend == 'true' ||
needs.changes.outputs.infra == 'true' ||
github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v4
- name: Login to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build & push frontend
uses: docker/build-push-action@v6
with:
context: ./frontend
push: true
cache-from: type=gha,scope=frontend
cache-to: type=gha,mode=max,scope=frontend
tags: |
${{ env.IMAGE_FRONTEND }}:latest
${{ env.IMAGE_FRONTEND }}:${{ github.sha }}
deploy:
runs-on: ubuntu-latest
needs: [changes, build-backend, build-worker, build-frontend]
permissions:
contents: read
# Запускается если ни один build не завершился failure.
# `always()` позволяет выполнить job даже когда зависимые jobs были skipped.
if: |
always() &&
!cancelled() &&
needs.build-backend.result != 'failure' &&
needs.build-worker.result != 'failure' &&
needs.build-frontend.result != 'failure'
steps:
- name: Deploy to VM via SSH
uses: appleboy/ssh-action@v1.0.3
env:
IMAGE_TAG: latest
SENTRY_RELEASE_VAL: ${{ github.sha }}
with:
host: ${{ secrets.DEPLOY_HOST }}
username: ${{ secrets.DEPLOY_USER }}
key: ${{ secrets.DEPLOY_SSH_KEY }}
port: ${{ secrets.DEPLOY_PORT || 22 }}
envs: IMAGE_TAG,SENTRY_RELEASE_VAL
script: |
set -euo pipefail
cd /opt/gendesign
# Sync compose / Caddyfile / init scripts from the repo.
# --ff-only refuses if there are local commits on the VM (we don't expect any).
git fetch origin main
git reset --hard origin/main
# Sentry release tracking — записываем git-sha в .env.runtime.
# ВАЖНО: НЕ перезаписываем файл целиком (там могут быть
# COUCHDB_PASSWORD и др. user-managed secrets). Заменяем только
# строку SENTRY_RELEASE или добавляем если её нет.
# IMAGE_TAG=latest для docker pull; SENTRY_RELEASE_VAL = git sha для трекинга.
mkdir -p backend
touch backend/.env.runtime
if grep -q '^SENTRY_RELEASE=' backend/.env.runtime; then
sed -i "s|^SENTRY_RELEASE=.*|SENTRY_RELEASE=$SENTRY_RELEASE_VAL|" backend/.env.runtime
else
printf 'SENTRY_RELEASE=%s\n' "$SENTRY_RELEASE_VAL" >> backend/.env.runtime
fi
chmod 600 backend/.env.runtime
# Создать external network если её нет (нужна Caddy для маршрута
# obsidian.gendsgn.ru → couchdb из отдельного obsidian-stack).
docker network inspect gendesign_shared >/dev/null 2>&1 \
|| docker network create gendesign_shared
export IMAGE_TAG="$IMAGE_TAG"
# Project name явно — `gendesign` (по имени папки auto), но
# фиксируем для consistency с obsidian-stack (-p gendesign-obsidian).
docker compose -p gendesign -f docker-compose.prod.yml pull
# ── Apply pending SQL migrations ──────────────────────────────────
# Tracking table: _schema_migrations (filename TEXT PRIMARY KEY).
# Each NN_*.sql file is applied exactly once, in sort order.
# ON_ERROR_STOP=on ensures a failing migration aborts the deploy.
set -a; source backend/.env; set +a
# Ensure tracking table exists (idempotent).
docker compose -p gendesign -f docker-compose.prod.yml exec -T postgres \
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=on -c "
CREATE TABLE IF NOT EXISTS _schema_migrations (
filename TEXT PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
"
# Apply each pending .sql file in NN order.
for sql_file in $(ls -1 data/sql/*.sql 2>/dev/null | sort); do
fname=$(basename "$sql_file")
applied=$(docker compose -p gendesign -f docker-compose.prod.yml exec -T postgres \
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc \
"SELECT COUNT(*) FROM _schema_migrations WHERE filename='$fname'")
if [ "$applied" = "0" ]; then
echo "→ Applying migration: $fname"
docker compose -p gendesign -f docker-compose.prod.yml exec -T postgres \
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=on \
< "$sql_file" \
|| { echo "FAILED on migration: $fname"; exit 1; }
docker compose -p gendesign -f docker-compose.prod.yml exec -T postgres \
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
"INSERT INTO _schema_migrations (filename) VALUES ('$fname') ON CONFLICT DO NOTHING;"
else
echo "✓ Already applied: $fname"
fi
done
echo "All migrations applied."
# ─────────────────────────────────────────────────────────────────
docker compose -p gendesign -f docker-compose.prod.yml up -d
# Caddyfile is bind-mounted; up -d won't re-read it. Reload explicitly.
# `|| true` so a temporary Caddy hiccup doesn't fail the whole deploy.
docker compose -p gendesign -f docker-compose.prod.yml exec -T caddy \
caddy reload --config /etc/caddy/Caddyfile || true
# Clean up disk — aggressive чтобы избежать накопления.
# 1. Для каждого gendesign-* repo оставляем 2 самых свежих SHA-тега
# + :latest. Docker images сортирует по Created DESC.
for repo in ghcr.io/lekss361/gendesign-backend \
ghcr.io/lekss361/gendesign-worker \
ghcr.io/lekss361/gendesign-frontend; do
docker images "$repo" --format '{{.Repository}}:{{.Tag}}' \
| grep -v ':latest$' \
| tail -n +3 \
| xargs -r docker rmi 2>/dev/null || true
done
# 2. ВСЕ unused images (даже свежие). Running контейнеры безопасны
# — их images не unused. Раньше был filter until=72h, но при
# частых деплоях оставались "fresh dangling" layers, накапливая
# до 50GB за неделю.
docker image prune -af || true
docker builder prune -af || true
# Wait for backend to be healthy (up to 30 s).
for i in $(seq 1 30); do
curl -fsS http://localhost:8000/health && break
sleep 1
done