Commit graph

24 commits

Author SHA1 Message Date
98ecdecb5b chore(deploy): wire OPENAI_API_KEY + LLM_ENABLED into prod env for LLM chat
All checks were successful
CI / changes (push) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 28s
Deploy / build-backend (push) Successful in 30s
Deploy / build-frontend (push) Successful in 28s
Deploy / deploy (push) Successful in 1m1s
Forward OPENAI_API_KEY (Forgejo secret) and LLM_ENABLED (Forgejo actions
variable) into the SSH deploy and write them to backend/.env.runtime ONLY
when non-empty, so the LLM chat (#960/#957) stays dormant
(llm_enabled=False, openai_api_key=None) until BOTH are set in Forgejo.
Mirrors the OBJECTIVE_API_KEY wiring (env block + envs list + idempotent
.env.runtime grep-sed-or-printf), but guarded by [ -n ] so an unset
secret/var writes nothing. No literal key (refs only). Compose header
documents both as optional. .env.runtime is shared by backend+worker+beat.

Refs #960
2026-06-09 13:51:41 +05:00
795661557f ci(frontend): add vitest job to Forgejo Actions gate
Run the merged frontend vitest suite on every frontend-touching PR so the
tests don't rot. Mirrors the #1032 backend gate: per-job dorny/paths-filter
(new `frontend` output, `frontend/**`), node:20 + npm cache on the lockfile,
`npm ci --legacy-peer-deps --no-audit --no-fund` (exact frontend/Dockerfile
flags), then `npm run test`. Additive — backend-tests job unchanged; no
top-level paths filter so frontend-only PRs already trigger the workflow.
Making frontend-tests a required check is a follow-up in the Forgejo UI.

Refs #999.
2026-06-07 11:52:10 +00:00
37837e72f4 ci(forgejo): add pytest gate for the main backend suite (CI-rehab 3/3)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
Forgejo runs only .forgejo/workflows/* — the .github pytest gate never
executed here, so backend changes merged + deployed untested (live bug
#994 district 500 shipped uncaught). Add a real gate: ruff check + the full
backend pytest suite (1687 passed) on PRs to main and feature-branch pushes,
scoped to backend/** + data/sql/** via paths-filter. Mock-only lane (no
postgres service): the one real-DB test self-skips via a connectivity probe;
WeasyPrint PDF tests RUN thanks to installed libpango. uv via official
installer (not setup-uv, PEP 668), uv sync --frozen against the committed lock,
TESTING=1 to activate the test-mode RBAC bypass.

NOTE: making this a HARD required check + having the auto-merge bot consult
check status needs Forgejo branch-protection config (human action) — until
then the gate is visible but advisory.

Refs #944.
2026-06-06 15:04:18 +00:00
0566fed5a2 ci(tradein): build+push tradein-browser image in deploy workflow (#905)
The new browser service references ghcr.io/lekss361/gendesign-tradein-browser;
without a build job the deploy's compose-up would fail pulling a missing image.
Adds build-browser job (mirrors build-backend, context ./tradein-mvp/browser),
a browser path-filter (tradein-mvp/browser/**), IMAGE_BROWSER env, and wires
build-browser into deploy needs + the no-failure gate.

Refs #905
2026-05-31 17:29:13 +03:00
ae1a35140c ci(tradein): pass NEXT_PUBLIC_ENABLE_PREVIEW=1 to frontend build (#801) 2026-05-30 18:40:22 +00:00
c30b3c714e fix(ci): drop --frozen in tradein test job (uv.lock is gitignored) (#678)
All checks were successful
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / test (push) Successful in 23s
Deploy Trade-In / build-frontend (push) Successful in 24s
Deploy Trade-In / deploy (push) Successful in 34s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-29 21:23:56 +00:00
22f1a9c1fc fix(ci): install uv via official installer in tradein test job (PEP 668) (#676)
Some checks failed
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / test (push) Failing after 6s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-29 20:45:12 +00:00
be63b249b5 chore(tradein): close pre-commit ruff gap + add CI pytest gate (#666) (#675)
Some checks failed
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / test (push) Failing after 4s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / build-frontend (push) Successful in 25s
Deploy Trade-In / deploy (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-29 19:18:48 +00:00
40002cfcd2 fix(deploy-tradein): track SQL migrations with _schema_migrations + baseline (#637)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 26s
Deploy Trade-In / build-frontend (push) Successful in 31s
Deploy Trade-In / deploy (push) Successful in 44s
2026-05-29 06:34:01 +00:00
lekss361
8c9d1344cc fix(stale-claims): avoid SIGPIPE 141 — pass file path не pipe
Root cause: `cat $ISSUES_TMP | python3 <<EOF ... sys.exit(0)` — Python exits
0 early, cat получает SIGPIPE, set -o pipefail → script exits 141 = failure.

Fix: передача file path через env var вместо stdin pipe. Python читает файл
напрямую. No pipe → no SIGPIPE.

Verified локально на VPS с bot-qa token: exit 0, '[OK] 0 wip issues —
nothing to do'.

Also remove diagnostic steps from workflow (added в PR #611, нужно для
поиска root cause, теперь убираем).

Refs: PR #611 (diagnostic), runs #1514-1520 failures.
2026-05-28 01:05:56 +03:00
lekss361
b86688bc0f chore(workflow): add diagnostic output to stale-claims.yml
Adds runner env + token sanity check + bash -x trace перед запуском cleanup
script. Цель — увидеть actual failure cause в Forgejo Actions logs (manual
runs #1514-1517 все упали ~4s, но script локально работает с тем же PAT).

Diag steps:
- runner environment (python3/curl/mktemp)
- token validity (curl /api/v1/user)
- bash -x для script trace

После merge — trigger manual dispatch + читать logs. Diagnostic строки уберём
fixup-commit'ом после диагноза.

Refs: PR #610 (stale-claims initial), runs #1514-1517 failures.
2026-05-28 01:01:29 +03:00
lekss361
e4dc866e20 fixup(claude): address review-bot blockers on PR #610
🔴 Critical (3):

1. cleanup-stale-claims.sh: export CUTOFF + STALE_HOURS
   Без `export` Python heredoc child process получал "0" вместо реального
   cutoff → cron всегда видел все issues как fresh → ничего не освобождал.
   Plus added abort при CUTOFF=0 в Python heredoc для defense-in-depth.

2. cleanup-stale-claims.sh: datetime.UTC → datetime.timezone.utc
   datetime.UTC требует Python 3.11+. Forgejo runner ubuntu-latest имеет 3.10.
   datetime.timezone.utc works в 3.8+.

3. cleanup-stale-claims.sh: invert label order — DELETE wip FIRST, потом
   POST ready. Если step 2 (add ready) fails — issue в neutral state без
   status/*, менее опасно чем оба status/ready+status/wip одновременно.

🟠 + 🟡 + 🟢 (5):

4. Added pagination loop (defensive cap 10 pages × 50 = 500 issues), но
   реализована через temp file + Python parsing (не bash JSON concat).

5. Removed dead code: iso_to_epoch() shell helper не использовался.

6. _autonomous_pickup.md: fix broken `Out-Null | if (-not $?)` pattern.
   GetEnvironmentVariable НЕ throws при missing — возвращает $null. $? у
   Get* всегда $true. Заменено на прямую проверку результата.

7. stale-claims.yml: добавлен concurrency group чтобы overlapping runs
   не stomp'ились (хотя 5-min timeout делает overlap unlikely, defensive).

8. Documented updated_at != heartbeat assumption в header script'а.

Не сделано (low/optional):

- Setup script tokens в memory (UX trade-off — не critical)
- Token suffix в console output (минор info leak в shell history)

Refs: PR #610 review by review-bot (sha=297eb29, verdict=changes)
2026-05-28 00:33:27 +03:00
lekss361
297eb29d65 feat(claude): env-vars refactor + stale-claim cron (Phase 2 finalization)
#11: slash-commands и _autonomous_pickup.md теперь читают persistent User-scope
env vars напрямую (FORGEJO_TOKEN_<ROLE>) вместо file-based из ~/.claude/secrets/.

Один раз: scripts/setup-bot-env.ps1 (запускается user'ом локально, кладёт
токены из vault в Windows User registry). После этого окна grab credentials
через [System.Environment]::GetEnvironmentVariable() — никаких файлов.

Все 5 work-as-*.md обновлены (analyst/backend/frontend/reviewer/qa).

#12: stale-claim cleanup — cron каждые 30 минут.

- scripts/cleanup-stale-claims.sh — bash, parses Forgejo API + Python для JSON
- .forgejo/workflows/stale-claims.yml — cron schedule `*/30 * * * *`

Освобождает issues застрявшие в status/wip >4h:
  - assignees=[]
  - +status/ready -status/wip
  - comment "stale claim released, worker likely crashed"

Использует secret FORGEJO_BOT_QA_TOKEN (минимум прав = write:issue).

Setup: secret нужно завести в Forgejo Actions UI после merge —
git.gendsgn.ru/lekss361/gendesign/-/settings/actions/secrets.

Refs: PRs #608, #609. Closes TODO с PR #608 (stale-claim cron).
2026-05-28 00:24:45 +03:00
ed49f6b953 fix(deploy): resolve FDW bootstrap race + extract bootstrap DO block to .sql file (#499)
Some checks failed
Deploy Trade-In / changes (push) Successful in 6s
Deploy / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 24s
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy / build-backend (push) Successful in 26s
Deploy / build-worker (push) Successful in 25s
Deploy / build-frontend (push) Successful in 25s
Deploy Trade-In / deploy (push) Successful in 33s
Deploy / deploy (push) Failing after 22s
2026-05-24 09:46:00 +00:00
94cf199276 feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493)
Some checks failed
Deploy Trade-In / changes (push) Successful in 5s
Deploy / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Successful in 26s
Deploy Trade-In / build-backend (push) Successful in 47s
Deploy / build-frontend (push) Successful in 29s
Deploy / build-backend (push) Successful in 1m24s
Deploy Trade-In / deploy (push) Successful in 40s
Deploy / build-worker (push) Successful in 2m57s
Deploy / deploy (push) Failing after 37s
Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81).

Security (deep-review fixes):
- 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe).
- core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit.
- 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback.
- geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread.
- 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings.
- pg_user_mappings query handles PUBLIC mapping (usename IS NULL).

Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.
2026-05-24 08:57:30 +00:00
3295808b41 fix(security): caddy/** в deploy paths + username из 401 попытки (#434)
All checks were successful
Deploy / build-frontend (push) Successful in 24s
Deploy / build-worker (push) Successful in 25s
Deploy / build-backend (push) Successful in 26s
Deploy / changes (push) Successful in 5s
Deploy / deploy (push) Successful in 54s
2026-05-23 09:25:41 +00:00
f4d43d1cfc fix(security): forwarder PermissionError — убрать USER nobody + deploy automation (#432)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / deploy (push) Successful in 59s
Deploy / build-backend (push) Successful in 27s
Deploy / build-worker (push) Successful in 27s
Deploy / build-frontend (push) Successful in 25s
Hotfix для GlitchTip Issue #71 (PermissionError при чтении /var/log/caddy/gendsgn.ru.log).

Changes:
- ops/glitchtip-auth-forwarder/Dockerfile — убран USER nobody, форвардер под root
- ops/glitchtip-auth-forwarder/forwarder.py — throttle capture_exception (300s)
- .forgejo/workflows/deploy.yml — paths-filter + build + force-recreate sidecar
- docs/runbooks/basic_auth_management.md — Deploy section

Security: caddy_logs mount read-only, нет network listener, нет shell,
auth_forwarder_state dedicated volume. Attack surface минимален.

После merge deploy.yml автоматически rebuild+recreate sidecar на VPS.
2026-05-23 09:04:00 +00:00
TradeIn Deploy
ee43248b11 fix(tradein): HOSTNAME=0.0.0.0 + NEXT_PUBLIC_API_BASE_URL build-arg
502 fix: Next.js standalone server.js биндился на HOSTNAME=<container-id>
вместо 0.0.0.0 → Caddy не мог достучаться до tradein-frontend:3000.
Теперь HOSTNAME=0.0.0.0 и в Dockerfile (runner ENV), и в compose.

API routing fix: фронт собирался без NEXT_PUBLIC_API_BASE_URL, fetch шёл
на same-origin /api/* (главный gendesign-backend) вместо /trade-in/api/*.
Добавлен build-arg в Dockerfile + workflow.
2026-05-21 10:21:32 +03:00
lekss361
02267d0306 feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in)
- FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex)
- Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in
- WeasyPrint PDF (Брусника-style 4-page report)
- Address autocomplete с typo-tolerance + 6 EKB presets
- Изолированный docker stack gendesign-tradein (отдельная postgres БД)
- Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/*
- Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login)
- Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml)
- Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest

Первый запуск на сервере (вручную, один раз):
  - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email)
  - docker network create gendesign_shared (если нет)
  - docker compose -p gendesign-tradein up -d
  - docker compose -p gendesign exec caddy caddy reload
2026-05-21 00:25:39 +03:00
d1b721a3be fix(deploy): trigger workflow on all data/sql/** changes, not only .sql (#320)
All checks were successful
Deploy / build-backend (push) Successful in 25s
Deploy / build-worker (push) Successful in 26s
Deploy / build-frontend (push) Successful in 24s
Deploy / changes (push) Successful in 4s
Deploy / deploy (push) Successful in 37s
2026-05-17 18:51:28 +00:00
a93bef4147 feat(deploy): inject OBJECTIVE_API_KEY into backend/.env.runtime (#311)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 33s
Deploy / build-worker (push) Successful in 31s
Deploy / build-frontend (push) Successful in 25s
Deploy / deploy (push) Successful in 54s
2026-05-17 16:05:13 +00:00
lekss361
11e78d738e fix(deploy): force-recreate caddy чтобы новые compose mounts применились
Раньше deploy.yml делал `docker compose exec caddy caddy reload` — это
только reload Caddyfile, но НЕ пересоздаёт container. Когда в
docker-compose.prod.yml добавлялся новый volume mount (например
./preview:/srv/preview:ro в PR #268), он не появлялся в running container
до manual `docker compose up -d --force-recreate caddy` через SSH.

Теперь deploy всегда вызывает force-recreate caddy после image pull —
идемпотентно, +~5 сек к deploy time, гарантирует что новые mounts +
Caddyfile блоки активны.
2026-05-17 11:30:45 +03:00
lekss361
e64fbd670b fix(infra): wire GlitchTip DSNs (build-arg + env_file) + reject sentry.io promote
Problem
- GlitchTip Issues = 0 за всё время, хотя backend + frontend SDK интегрированы (PR #207, #208).
- Старый Sentry.io продолжает получать события — user видит уведомления оттуда.

Root cause
- frontend/Dockerfile не имеет ARG NEXT_PUBLIC_GLITCHTIP_DSN → `npm run build` бьёт
  с пустым env var → Next.js инлайнит undefined → SDK init guard `if (dsn)` skips.
  Chrome-devtools check на prod bundle подтвердил: ни в одном из 9 chunks DSN-строка
  не запечена; `i.env.NEXT_PUBLIC_GLITCHTIP_DSN` evaluates to undefined.
- .forgejo/workflows/deploy.yml build-frontend не передавал build-args.
- На VPS backend/.env.runtime содержит legacy SENTRY_DSN=...@sentry.io/...
  config.py:_promote_legacy_sentry_dsn слепо промоутит его в glitchtip_dsn → SDK
  шлёт в чужой Sentry. GLITCHTIP_DSN там не задан.
- deploy.yml SSH-скрипт никогда не редактировал SENTRY_DSN/GLITCHTIP_DSN в .env.runtime.

Solution
1. frontend/Dockerfile: ARG NEXT_PUBLIC_GLITCHTIP_DSN + NEXT_PUBLIC_ENVIRONMENT
   с пустыми defaults, ENV-mirror перед `npm run build`. Локальный build без
   build-args работает по-прежнему (no-op DSN).

2. .forgejo/workflows/deploy.yml:
   - build-frontend: `build-args` передаёт `secrets.GLITCHTIP_FRONTEND_DSN`
     + NEXT_PUBLIC_ENVIRONMENT=production. Инвалидирует cache → frontend
     image пересоберётся (expected).
   - deploy step: GLITCHTIP_BACKEND_DSN через secret, в SSH-скрипте:
     a) `sed -i '/^SENTRY_DSN=/d' backend/.env.runtime` — снести legacy
     b) upsert GLITCHTIP_DSN (sed/printf) тем же паттерном что SENTRY_RELEASE
     c) `compose up -d --force-recreate --no-deps backend worker beat` —
        обычный `up -d` не перечитывает env_file без image change.

3. backend/app/core/config.py: _promote_legacy_sentry_dsn ужесточён —
   принимает SENTRY_DSN только если host == errors.gendsgn.ru. Для других
   URLs (sentry.io) выдаёт UserWarning и НЕ промоутит. Anti-regression на
   случай если SENTRY_DSN снова окажется в .env.runtime после ручного
   вмешательства.

Required Forgejo secrets (Settings → Actions → Secrets)
- GLITCHTIP_BACKEND_DSN = https://3d6e291003e142458957490c83559867@errors.gendsgn.ru/1
- GLITCHTIP_FRONTEND_DSN = https://5d7bc85e300c4e80a8554ccc818ff56d@errors.gendsgn.ru/2
DSNs публичны (видны в browser bundle) — secrets ради build-time injection,
не для конфиденциальности. Если secrets не заданы → deploy succeeds, SDK
no-op, без регрессии.

Test plan
- Verify Forgejo deploy.yml зелёный после merge
- chrome-devtools: открыть gendsgn.ru → search bundle на DSN string → должна
  быть запечена строка errors.gendsgn.ru/2
- Trigger frontend error → POST к errors.gendsgn.ru/api/2/envelope/
- Backend: curl на endpoint вызывающий 500 → событие в GlitchTip backend project
- GlitchTip dashboard https://errors.gendsgn.ru/gendesign/issues — Issues > 0

References
- vault: meta/00_credentials.md (DSNs + incident notes 2026-05-16)
- vault: decisions/Dec_GlitchTip_Frontend_Sentry_SDK.md (env contract)
- vault: fixes/fixes-MOC.md (#204 backend SDK init)
2026-05-16 21:48:28 +03:00
e97f96afa7 fix(ci): switch deploy to .forgejo/workflows + shell GHCR login (#192)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 3m44s
Deploy / build-frontend (push) Successful in 26s
Deploy / deploy (push) Successful in 35s
Deploy / build-worker (push) Successful in 4m43s
2026-05-16 05:25:00 +00:00