gendesign/.forgejo/workflows/deploy-tradein.yml
bot-backend 5bcb7ab909
All checks were successful
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Successful in 1m12s
CI Trade-In / backend-tests (pull_request) Successful in 1m43s
CI Trade-In / changes (pull_request) Successful in 10s
CI / changes (pull_request) Successful in 9s
CI / openapi-codegen-check (pull_request) Has been skipped
ci(tradein): pre-merge гейт — backend pytest + frontend type-check/lint (#2208)
Тradein-PR мержились на пусто-зелёных чеках: ci.yml фильтрует только
backend/**+frontend/** главного стека, а pytest tradein жил лишь в
post-merge deploy-tradein.yml. Новый ci-tradein.yml гейтит pull_request
до мержа; deploy test-job синхронизирован.

- уборка устаревшего deselect: test_cian_valuation::test_cache_hit_returns_cached
  проходит в полном прогоне (2947 passed); остаётся только test_search_cache_hit
- pytest-timeout=120 per-test (страховка от зависших тестов в CI)
- uv sync --frozen в обоих test-job (workspace-лок tracked, зеркало Dockerfile)
- eslint flat config для tradein-frontend (гейт lint был невозможен: next lint
  без конфига уходит в интерактивный prompt и падает в CI)
- 2 apostrophe lint-фикса в scrapers/page.tsx

Refs #2208
2026-07-02 22:48:18 +03:00

468 lines
24 KiB
YAML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

name: Deploy Trade-In
# Forgejo Actions — отдельный pipeline для подпроекта tradein-mvp/.
# Триггерится только на изменения внутри tradein-mvp/ (или этого workflow),
# не пересекается с основным deploy.yml.
on:
push:
branches: [main]
paths:
- "tradein-mvp/**"
- ".forgejo/workflows/deploy-tradein.yml"
workflow_dispatch:
concurrency:
group: deploy-tradein-prod
cancel-in-progress: false
env:
IMAGE_BACKEND: ghcr.io/lekss361/gendesign-tradein-backend
IMAGE_FRONTEND: ghcr.io/lekss361/gendesign-tradein-frontend
IMAGE_BROWSER: ghcr.io/lekss361/gendesign-tradein-browser
jobs:
changes:
runs-on: ubuntu-latest
outputs:
backend: ${{ steps.set-all.outputs.backend || steps.filter.outputs.backend }}
frontend: ${{ steps.set-all.outputs.frontend || steps.filter.outputs.frontend }}
browser: ${{ steps.set-all.outputs.browser || steps.filter.outputs.browser }}
infra: ${{ steps.set-all.outputs.infra || steps.filter.outputs.infra }}
scraper: ${{ steps.set-all.outputs.scraper || steps.filter.outputs.scraper }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
# Resolve base SHA: read last-successfully-deployed SHA from the VPS host file.
# The file is written by the deploy job on every successful deploy.
# Fail-safe: if we cannot read the file, or the SHA is not an ancestor of HEAD,
# we leave DEPLOYED_SHA empty — the next step will then build everything.
- name: Resolve deployed base SHA
id: resolve-base
env:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_PORT: ${{ secrets.DEPLOY_PORT }}
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
run: |
# Write SSH key to a temp file
SSH_KEY_FILE=$(mktemp)
echo "$DEPLOY_SSH_KEY" > "$SSH_KEY_FILE"
chmod 600 "$SSH_KEY_FILE"
# Try to read the marker file from the VPS. Suppress errors — if host is
# unreachable or file missing, RAW_SHA will be empty.
RAW_SHA=$(ssh -i "$SSH_KEY_FILE" \
-o StrictHostKeyChecking=no \
-o ConnectTimeout=10 \
-p "${DEPLOY_PORT:-22}" \
"${DEPLOY_USER}@${DEPLOY_HOST}" \
"cat /opt/gendesign/.tradein-deployed-sha 2>/dev/null || true" \
2>/dev/null || true)
RAW_SHA=$(echo "$RAW_SHA" | tr -d '[:space:]')
rm -f "$SSH_KEY_FILE"
# Validate: non-empty, looks like a git SHA, and is an ancestor of HEAD.
DEPLOYED_SHA=""
if [ -n "$RAW_SHA" ] && echo "$RAW_SHA" | grep -qE '^[0-9a-f]{40}$'; then
if git merge-base --is-ancestor "$RAW_SHA" HEAD 2>/dev/null; then
DEPLOYED_SHA="$RAW_SHA"
echo "Resolved deployed base: $DEPLOYED_SHA"
else
echo "WARNING: stored SHA $RAW_SHA is not an ancestor of HEAD — falling back to build-all"
fi
else
echo "No valid deployed SHA found — falling back to build-all"
fi
echo "deployed_sha=$DEPLOYED_SHA" >> "$GITHUB_OUTPUT"
# FAIL-SAFE: if no valid base SHA, emit all=true and skip paths-filter.
# This covers: first run, post-force-push, VPS unreachable, corrupt marker.
# A spurious full build is always safer than a missed build.
- name: Build-all fallback (no base SHA)
id: set-all
if: steps.resolve-base.outputs.deployed_sha == ''
run: |
echo "No base SHA — enabling build-all"
echo "backend=true" >> "$GITHUB_OUTPUT"
echo "frontend=true" >> "$GITHUB_OUTPUT"
echo "browser=true" >> "$GITHUB_OUTPUT"
echo "infra=true" >> "$GITHUB_OUTPUT"
echo "scraper=true" >> "$GITHUB_OUTPUT"
# Cumulative diff: compare deployed SHA → HEAD so that a fast chain of merges
# (e.g. backend #1829 then frontend #1830) doesn't lose earlier changes.
- uses: dorny/paths-filter@v3
id: filter
if: steps.resolve-base.outputs.deployed_sha != ''
with:
base: ${{ steps.resolve-base.outputs.deployed_sha }}
filters: |
backend:
- 'tradein-mvp/backend/**'
# scraper-kit вкомпилирован в backend-образ (build context tradein-mvp/,
# scheduler_main импортирует пакет) — kit-only изменение обязано
# пересобрать образ, иначе деплой рестартует контейнеры на старом.
- 'tradein-mvp/packages/scraper-kit/**'
frontend:
- 'tradein-mvp/frontend/**'
browser:
- 'tradein-mvp/browser/**'
infra:
- 'tradein-mvp/docker-compose.prod.yml'
- 'tradein-mvp/deploy/**'
- '.forgejo/workflows/deploy-tradein.yml'
scraper:
- 'tradein-mvp/backend/app/services/scrapers/**'
- 'tradein-mvp/backend/app/services/scrape_pipeline.py'
- 'tradein-mvp/backend/app/services/scheduler.py'
- 'tradein-mvp/backend/app/scheduler_main.py'
- 'tradein-mvp/backend/app/tasks/**'
# #2188: scheduler исполняет matching/dedup при каждом scrape-тике —
# без этих путей scraper-контейнер оставался на старом коде
# (2026-07-02: fias-dedup доехал до tradein-backend, но не до
# tradein-scraper). После USE_KIT_SCHEDULER=true kit-код и есть
# scheduler — его правки тоже обязаны пересоздавать контейнер.
- 'tradein-mvp/backend/app/services/matching/**'
- 'tradein-mvp/backend/app/services/house_dedup_merge.py'
- 'tradein-mvp/packages/scraper-kit/**'
# Quality gate: pytest MUST pass before any image is built/deployed (#666).
# Runs the tradein-mvp/backend suite; a red test blocks build + deploy.
# Tests use mocks + a stub DATABASE_URL — no real Postgres/Redis needed.
# 1 pre-existing order-dependent test is deselected (see DESELECT note below).
test:
runs-on: ubuntu-latest
needs: changes
if: |
needs.changes.outputs.backend == 'true' ||
needs.changes.outputs.infra == 'true' ||
github.event_name == 'workflow_dispatch'
defaults:
run:
working-directory: ./tradein-mvp/backend
env:
# psycopg v3 requires a parseable URL at import time; never connected to.
DATABASE_URL: postgresql+psycopg://test:test@localhost:5432/test
steps:
- uses: actions/checkout@v4
- name: Install uv
# Официальный standalone-инсталлер: системный `pip install uv` на
# ubuntu-runner падает с PEP 668 externally-managed-environment (#666 CI).
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Sync deps (incl. dev group — pytest)
# Workspace-лок tradein-mvp/uv.lock TRACKED (с воркспейса #2137; gitignored
# только старый backend/uv.lock) → --frozen детерминирован и зеркалит
# Dockerfile (uv sync --frozen --no-dev). Актуализировано в #2208.
run: uv sync --frozen
- name: Run pytest (tradein-mvp/backend)
# DESELECT (актуализировано 2026-07-02, #2208): test_search_cache_hit падает
# ТОЛЬКО в whole-suite ordering (401 vs 200; в изоляции проходит) — global-state
# leak из другого test-модуля, pre-existing. Второй исторический deselect
# (test_cian_valuation::test_cache_hit_returns_cached) убран — проходит в полном
# прогоне (проверено 2026-07-02: 2947 passed / 1 failed). Список обязан
# совпадать с backend-tests в ci-tradein.yml (pre-merge гейт).
run: |
uv run pytest -q \
--deselect "tests/test_search_api.py::test_search_cache_hit"
build-backend:
runs-on: ubuntu-latest
needs: [changes, test]
if: |
needs.changes.outputs.backend == 'true' ||
needs.changes.outputs.infra == 'true' ||
github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v4
- name: Login to GHCR (shell-based — docker/login-action@v3 unreliable под Forgejo Actions)
env:
GHCR_PAT: ${{ secrets.GHCR_PAT }}
run: |
echo "$GHCR_PAT" | docker login ghcr.io -u lekss361 --password-stdin
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build & push tradein-backend
uses: docker/build-push-action@v6
with:
# Context = tradein-mvp/ (uv workspace root): образу нужен packages/scraper-kit
# для editable install (#2137). Dockerfile — в backend/.
context: ./tradein-mvp
file: ./tradein-mvp/backend/Dockerfile
push: true
cache-from: type=registry,ref=${{ env.IMAGE_BACKEND }}:buildcache
cache-to: type=registry,ref=${{ env.IMAGE_BACKEND }}:buildcache,mode=max
tags: |
${{ env.IMAGE_BACKEND }}:latest
${{ env.IMAGE_BACKEND }}:${{ github.sha }}
build-frontend:
runs-on: ubuntu-latest
needs: changes
if: |
needs.changes.outputs.frontend == 'true' ||
needs.changes.outputs.infra == 'true' ||
github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v4
- name: Login to GHCR (shell-based — docker/login-action@v3 unreliable под Forgejo Actions)
env:
GHCR_PAT: ${{ secrets.GHCR_PAT }}
run: |
echo "$GHCR_PAT" | docker login ghcr.io -u lekss361 --password-stdin
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build & push tradein-frontend
uses: docker/build-push-action@v6
with:
context: ./tradein-mvp/frontend
push: true
# basePath=/trade-in baked-in во время build (Next.js)
build-args: |
NEXT_PUBLIC_BASE_PATH=/trade-in
NEXT_PUBLIC_API_BASE_URL=/trade-in
NEXT_PUBLIC_ENABLE_PREVIEW=1
cache-from: type=registry,ref=${{ env.IMAGE_FRONTEND }}:buildcache
cache-to: type=registry,ref=${{ env.IMAGE_FRONTEND }}:buildcache,mode=max
tags: |
${{ env.IMAGE_FRONTEND }}:latest
${{ env.IMAGE_FRONTEND }}:${{ github.sha }}
build-browser:
runs-on: ubuntu-latest
needs: changes
# tradein-browser несёт camoufox + Firefox-build (#905). Триггерится на
# изменения browser/ или infra (compose ссылается на образ) или вручную.
if: |
needs.changes.outputs.browser == 'true' ||
needs.changes.outputs.infra == 'true' ||
github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v4
- name: Login to GHCR (shell-based — docker/login-action@v3 unreliable под Forgejo Actions)
env:
GHCR_PAT: ${{ secrets.GHCR_PAT }}
run: |
echo "$GHCR_PAT" | docker login ghcr.io -u lekss361 --password-stdin
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build & push tradein-browser
uses: docker/build-push-action@v6
with:
context: ./tradein-mvp/browser
push: true
cache-from: type=registry,ref=${{ env.IMAGE_BROWSER }}:buildcache
cache-to: type=registry,ref=${{ env.IMAGE_BROWSER }}:buildcache,mode=max
tags: |
${{ env.IMAGE_BROWSER }}:latest
${{ env.IMAGE_BROWSER }}:${{ github.sha }}
deploy:
runs-on: ubuntu-latest
needs: [changes, test, build-backend, build-frontend, build-browser]
# NB: a failed `test` skips build-backend (result='skipped', not 'failure'),
# so we must block deploy on test failure explicitly (#666 quality gate).
if: |
always() &&
!cancelled() &&
needs.test.result != 'failure' &&
needs.build-backend.result != 'failure' &&
needs.build-frontend.result != 'failure' &&
needs.build-browser.result != 'failure'
steps:
- name: Deploy via SSH
uses: appleboy/ssh-action@v1.0.3
env:
IMAGE_TAG: latest
GHCR_PAT: ${{ secrets.GHCR_PAT }}
# Phase 0: generic infra edits (compose / workflow / deploy/**) must NOT
# recreate the scraper and SIGKILL a running multi-hour job. Only genuine
# scraper-code paths (the `scraper` paths-filter already covers
# app/services/scrapers/**, scrape_pipeline.py, scheduler.py,
# scheduler_main.py, app/tasks/**) — or a manual workflow_dispatch —
# should trigger a scraper recreate. (infra term intentionally dropped.)
SCRAPER_CHANGED: ${{ needs.changes.outputs.scraper == 'true' || github.event_name == 'workflow_dispatch' }}
GITHUB_SHA: ${{ github.sha }}
with:
host: ${{ secrets.DEPLOY_HOST }}
username: ${{ secrets.DEPLOY_USER }}
key: ${{ secrets.DEPLOY_SSH_KEY }}
port: ${{ secrets.DEPLOY_PORT }}
envs: IMAGE_TAG,GHCR_PAT,SCRAPER_CHANGED,GITHUB_SHA
script: |
set -euo pipefail
cd /opt/gendesign
# repo уже clone'ен — origin = Forgejo. Подтягиваем последний main.
git fetch origin main
git reset --hard origin/main
cd tradein-mvp
# .env.runtime создаётся вручную при первом запуске (см. README-АДМИНУ.md).
# Здесь только подгружаем переменные для docker compose (POSTGRES_PASSWORD).
if [ ! -f .env.runtime ]; then
echo "ERROR: /opt/gendesign/tradein-mvp/.env.runtime отсутствует."
echo "Создай его вручную (см. tradein-mvp/README.md или DEPLOY.md)."
exit 1
fi
chmod 600 .env.runtime
set -a; source .env.runtime; set +a
# External network для Caddy (он в основном gendesign-стеке)
docker network inspect gendesign_shared >/dev/null 2>&1 \
|| docker network create gendesign_shared
# Re-login to GHCR (PAT может быть rotated)
echo "$GHCR_PAT" | docker login ghcr.io -u lekss361 --password-stdin
export IMAGE_TAG="$IMAGE_TAG"
docker compose -p gendesign-tradein -f docker-compose.prod.yml pull
# Селективный up: scraper НЕ пересоздаём на каждый backend-деплой (#1182) —
# один image :latest на backend+scraper, блочный up -d рестартовал бы scraper
# и убивал бегущий sweep. Scraper поднимается только при scraper/infra изменениях.
docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --no-deps postgres browser backend frontend
if [ "${SCRAPER_CHANGED:-true}" = "true" ]; then
echo "→ scraper paths changed — recreating tradein-scraper"
docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --no-deps scraper
else
echo "→ scraper unchanged — tradein-scraper left running (подхватит новый image при следующем своём рестарте)"
fi
# Применяем SQL миграции (если есть backend/data/sql/*.sql)
# Postgres init load *.sql из /docker-entrypoint-initdb.d ТОЛЬКО при первом
# старте volume. Здесь — для повторных миграций после первого запуска.
# Tracking через _schema_migrations (порт паттерна из deploy.yml):
# каждый .sql применяется РОВНО один раз, failed migration → exit 1
# (никаких swallowed errors). cwd = /opt/gendesign/tradein-mvp.
sleep 5
# Pre-existence detection ДО CREATE TABLE: если таблицы ещё нет, это
# первый deploy после внедрения tracking на уже-наполненной prod-БД
# (миграции 001-076 живут в схеме). 7 из них (002/003/052/053/054/063/072)
# содержат INSERT-seed БЕЗ ON CONFLICT — повторный прогон под строгим
# ON_ERROR_STOP упал бы на PK violation. Поэтому: baseline (см. ниже).
migrations_table_existed=$(docker compose -p gendesign-tradein -f docker-compose.prod.yml exec -T postgres \
psql -U "${TRADEIN_POSTGRES_USER:-tradein}" -d tradein -tAc \
"SELECT to_regclass('public._schema_migrations') IS NOT NULL;" | tr -d '[:space:]')
docker compose -p gendesign-tradein -f docker-compose.prod.yml exec -T postgres \
psql -U "${TRADEIN_POSTGRES_USER:-tradein}" -d tradein -v ON_ERROR_STOP=on -c "
CREATE TABLE IF NOT EXISTS _schema_migrations (
filename TEXT PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
"
if [ "$migrations_table_existed" != "t" ]; then
# BASELINE: таблицы не было → seed ВСЕ текущие миграции как applied
# БЕЗ их прогона. prod уже работает на этой схеме; помечаем её
# текущим состоянием, чтобы под строгий gate попадали только НОВЫЕ
# (077+) миграции. INSERT ... ON CONFLICT DO NOTHING — идемпотентно.
echo "→ _schema_migrations отсутствовала — baseline существующих миграций (без прогона)"
for sql_file in $(ls -1 backend/data/sql/*.sql 2>/dev/null | sort); do
fname=$(basename "$sql_file")
echo " baseline: $fname"
docker compose -p gendesign-tradein -f docker-compose.prod.yml exec -T postgres \
psql -U "${TRADEIN_POSTGRES_USER:-tradein}" -d tradein -v ON_ERROR_STOP=on -c \
"INSERT INTO _schema_migrations (filename) VALUES ('$fname') ON CONFLICT DO NOTHING;"
done
echo "Baseline complete — existing schema marked as applied."
fi
for sql_file in $(ls -1 backend/data/sql/*.sql 2>/dev/null | sort); do
fname=$(basename "$sql_file")
applied=$(docker compose -p gendesign-tradein -f docker-compose.prod.yml exec -T postgres \
psql -U "${TRADEIN_POSTGRES_USER:-tradein}" -d tradein -tAc \
"SELECT COUNT(*) FROM _schema_migrations WHERE filename='$fname'" | tr -d '[:space:]')
if [ "$applied" = "0" ]; then
echo "→ Applying migration: $fname"
docker compose -p gendesign-tradein -f docker-compose.prod.yml exec -T postgres \
psql -U "${TRADEIN_POSTGRES_USER:-tradein}" -d tradein -v ON_ERROR_STOP=on \
< "$sql_file" \
|| { echo "FAILED on migration: $fname"; exit 1; }
docker compose -p gendesign-tradein -f docker-compose.prod.yml exec -T postgres \
psql -U "${TRADEIN_POSTGRES_USER:-tradein}" -d tradein -c \
"INSERT INTO _schema_migrations (filename) VALUES ('$fname') ON CONFLICT DO NOTHING;"
else
echo "✓ Already applied: $fname"
fi
done
echo "All migrations applied."
# Bootstrap gendesign_reader password from env (post-migration, #976).
# SQL migration 101_gendesign_reader_role.sql creates role passwordless;
# password lives only in /opt/gendesign/tradein-mvp/.env.runtime.
# .env.runtime already sourced above (set -a; source .env.runtime).
#
# ⚠️ psql variable substitution (:'pw') НЕ работает внутри -c
# (переменная доходит до сервера as literal → syntax error at ':').
# Решение: передаём SQL через stdin (< file), psql интерполирует
# :'pw' ВНЕ dollar-quoted блока. Детали: ops/db-bootstrap/set_gendesign_reader_password.sql.
if [ -n "${TRADEIN_READER_PASSWORD:-}" ]; then
echo "→ Applying gendesign_reader password from env"
docker compose -p gendesign-tradein -f docker-compose.prod.yml exec -T postgres \
psql -U "${TRADEIN_POSTGRES_USER:-tradein}" -d tradein -v ON_ERROR_STOP=on \
-v "pw=${TRADEIN_READER_PASSWORD}" \
< ops/db-bootstrap/set_gendesign_reader_password.sql
else
echo "WARNING: TRADEIN_READER_PASSWORD not set in .env.runtime — gendesign_reader без пароля, ETL #976 не сможет подключиться"
fi
# Retry backend lifespan hook AFTER migrations applied.
# tradein-backend startup runs ensure_fdw_user_mapping which needs
# FOREIGN SERVER gendesign_remote (created by 060_postgres_fdw_extension.sql).
# Without restart, the first compose-up's startup hook failed with
# "server gendesign_remote does not exist" because migrations hadn't run yet.
# See PR #493 deploy/1156 for the incident details.
echo "→ Restarting tradein-backend so lifespan hook retries USER MAPPING setup"
docker restart tradein-backend
# Give backend time to come up before Caddy reload + health check below
sleep 5
# Caddy reload — основной Caddyfile содержит inline tradein routes
# (см. Caddyfile в корне репы). Reload, чтобы Caddy перечитал DNS
# tradein-backend / tradein-frontend (они в gendesign_shared network).
cd /opt/gendesign
docker compose -p gendesign -f docker-compose.prod.yml exec -T caddy \
caddy reload --config /etc/caddy/Caddyfile || true
# Health check
for i in $(seq 1 30); do
docker compose -p gendesign-tradein -f /opt/gendesign/tradein-mvp/docker-compose.prod.yml \
exec -T backend curl -fsS http://localhost:8000/health && break
sleep 1
done
# Cleanup старых образов
for repo in ghcr.io/lekss361/gendesign-tradein-backend \
ghcr.io/lekss361/gendesign-tradein-frontend; do
docker images "$repo" --format '{{.Repository}}:{{.Tag}}' \
| grep -v ':latest$' | tail -n +3 \
| xargs -r docker rmi 2>/dev/null || true
done
docker image prune -af || true
# Mark this SHA as successfully deployed.
# Written LAST — only after all of the above completed without error.
# The changes job reads this file on the next run to compute cumulative diff.
echo "$GITHUB_SHA" > /opt/gendesign/.tradein-deployed-sha
echo "→ Deployed SHA marker updated: $GITHUB_SHA"