Commit graph

18 commits

Author SHA1 Message Date
7505dbc1c6 fix(devops): переносим deploy-obsidian.yml в .forgejo/workflows/ (#2416)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 8s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Жил в .github/workflows/ с миграции с GitHub (16.05.2026), README называл
это намеренным решением («остался на GitHub») — но живого GitHub-зеркала
с секретами не нашлось: 0 запусков за всю историю Forgejo Actions (12000+
прогонов остальных workflow), контейнер не пересоздавался с 17.05 до
ручного SSH-фикса 04.07 (couchdb healthcheck, #2413).

Anton подтвердил vault живым (CouchDB 200 + gendesign-vault 46830
документов + Obsidian REST MCP листит структуру) и дал добро на перенос.

Перенесено в единственную реально исполняемую директорию + обновлены
self-reference путь в самом workflow, README.md, docs/obsidian-livesync.md
(заодно поправлена соседняя устаревшая строка про main deploy.yml —
тоже давно .forgejo/, не .github/).

Refs #2416
2026-07-05 20:28:28 +05:00
5455d57bed chore(ci): run CI on push to feature branches (#195) 2026-05-16 08:26:04 +00:00
e97f96afa7 fix(ci): switch deploy to .forgejo/workflows + shell GHCR login (#192)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 3m44s
Deploy / build-frontend (push) Successful in 26s
Deploy / deploy (push) Successful in 35s
Deploy / build-worker (push) Successful in 4m43s
2026-05-16 05:25:00 +00:00
lekss361
d0b8eda6ad
fix(deploy): source root .env (где POSTGRES_USER) не backend/.env (#153)
PR #151 deploy упал с 'POSTGRES_USER: unbound variable'.

source backend/.env — backend/.env содержит app vars (SCRAPE_ADMIN_TOKEN,
OPENROUTESERVICE_API_KEY и пр.), НО НЕ POSTGRES_USER/PASSWORD/DB.

Postgres credentials живут в /opt/gendesign/.env (root) — оттуда их читает
compose через env_file для postgres service. Это canonical location для
DB creds на этом стеке.

Fix: source .env (root) — corresponds к compose env_file resolution.

После merge → next deploy auto-apply применит pending #89/#91 (90 уже
applied via MCP).

Refs: #150, incident 2026-05-15 deploy #25901378924

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:19:17 +03:00
lekss361
7c43bc5496
feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150) (#151)
* fix(frontend): apiFetchWithStatus body stream double-read crash

Per user report 2026-05-14: 'Failed to execute text on Response: body stream
already read' при analyze с non-JSON error response.

apiFetchWithStatus делал .json() (consumes stream), потом .text() в catch
(FAILS — stream already consumed). Fetch API: body — ReadableStream,
consumable один раз.

Fix: read text() once → JSON.parse on string. На fail (HTML / non-JSON)
оборачиваем в {detail: rawText}. Никаких double-reads.

Site-finder cad search больше не крашится на error pages.

* feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150)

Закрывает root cause production 500 на /api/v1/admin/site-finder/weight-profiles
(а также 2 unapplied migrations: #89/#91).

## Why это нужно

После audit обнаружили что:
- backend/alembic/versions/ пустой (Alembic configured но не используется)
- deploy.yml не имеет migration step
- 47+ raw SQL files накопилось без auto-apply
- 3 файла unapplied: 87 (engineering), 89 (drop brin), 90 (weight profiles)

User report: GET weight-profiles → 500 потому что user_weight_profiles
table не существует на prod.

## Changes

### deploy.yml +39 lines
- Path trigger расширен на data/sql/*.sql
- New step 'Apply DB migrations' между compose pull и compose up -d:
  - Idempotent CREATE TABLE _schema_migrations
  - Loop по data/sql/*.sql sorted → skip applied → psql with ON_ERROR_STOP=on
  - Record applied filename в tracking table
  - Exit 1 на failure → containers НЕ обновляются (no partial state)

### NN collision fix
- 87_engineering_networks_191fz.sql → 91_engineering_networks_191fz.sql
  (collision с 87_on_demand_indexes.sql, мой renumber 85→87 был неудачным)

### CLAUDE.md +32 lines
- Subsection 'Auto-apply' под Migration pattern
- Bootstrap procedure docs
- Idempotency requirement
- Failure recovery

### scripts/bootstrap_schema_migrations.sh +100 lines
- One-time seed _schema_migrations с 48 already-applied files
- 3 файла intentionally NOT seeded: 89/90/91 (auto-apply на first deploy)
- Usage docs + verification query

## Vault

domains/infra/Runbook_Auto_Apply_Migrations.md NEW
infra-MOC.md updated

## Required manual steps ДО merge

1. SSH gendesign (tunnel)
2. POSTGRES creds export
3. bash scripts/bootstrap_schema_migrations.sh — seed 48 already-applied files
4. Verify: SELECT COUNT(*) FROM _schema_migrations → 48
5. После merge → first deploy auto-apply #89/#90/#91 → 500 closes

Closes #150

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:06:54 +03:00
lekss361
59466de811 fix(deploy): aggressive docker image prune (drop until=72h filter to avoid 50GB buildup) 2026-05-11 23:26:31 +03:00
lekss361
9658baa6b2 fix(deploy): keep last 2 SHA-tagged images per repo, drop older
Previously `docker image prune -af --filter until=72h` only removed
images older than 72h. With multiple deploys/day we accumulate many
fresh (<72h) tagged images — each gendesign-backend / worker / frontend
SHA-tag is 500MB-1GB, filling /var/lib/docker.

New logic: for each ghcr.io/lekss361/gendesign-* repo, list tags sorted
by Created DESC, skip :latest, keep top 2, remove the rest. Still runs
the standard prune after for dangling layers + build cache.
2026-05-11 17:45:20 +03:00
lekss361
de21fac0a8 perf(ci): skip unchanged docker builds via paths-filter
Add `changes` job (dorny/paths-filter@v3) to detect which parts of the
codebase changed. Build jobs now run only when their source tree is
touched; deploy runs as long as no build failed (skipped builds are OK).

Split IMAGE_TAG (latest) from SENTRY_RELEASE_VAL (git sha) so Sentry
release tracking still points to the exact commit while compose pulls
the :latest tag.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 15:10:23 +03:00
lekss361
a7d4223dfa perf(ci): parallelize docker builds + add npm cache mount
Split single build-and-deploy job into 4 jobs: build-backend,
build-worker, build-frontend run in parallel (all 3 independent),
deploy job has needs:[...] on all three. Removes ~2 min of sequential
build time on warm cache runs.

Also: scope GHA cache keys per image (backend-lean / worker-chromium /
frontend) so builds don't invalidate each other's layer cache. Replace
sleep 8 health-check with a 30-iteration polling loop (fails fast on
bad deploy, doesn't over-wait on fast ones). Add --mount=type=cache to
frontend npm ci for node_modules cache across builds.
2026-05-11 15:01:10 +03:00
lekss361
b14700e583 fix deploy 2026-05-11 10:01:36 +03:00
lekss361
d50626b32a fix 2026-05-11 09:57:40 +03:00
lekss361
b0f5c38894 feat(deploy): split obsidian stack from main + path-filtered GHA
Obsidian Self-hosted LiveSync (CouchDB) выведен в отдельный docker-compose
stack и GHA workflow. Любой стек теперь редеплоится независимо.

* docker-compose.obsidian.yml: только couchdb в shared external network
* docker-compose.prod.yml: убрана couchdb-секция; caddy подключён к
  shared network для маршрута obsidian.gendsgn.ru → couchdb:5984
* .github/workflows/deploy-obsidian.yml: новый workflow с path-filters
  (триггерится только при изменениях obsidian-related файлов)
* .github/workflows/deploy.yml: + path-filters (триггерится только при
  main-related файлах) + создание shared network в bootstrap
* docs/obsidian-livesync.md: обновлены инструкции под split-архитектуру

После push:
1. ssh gendesign 'docker network create gendesign_shared'  (один раз)
2. Прописать COUCHDB_USER/PASSWORD в backend/.env.runtime
3. Добавить DNS A-record obsidian.gendsgn.ru → IP VPS
4. git push → GHA задеплоит обе части независимо

Преимущества:
- Изменения main не разрывают LiveSync клиентов
- Изменения obsidian не рестартуют main
- Падение obsidian-стека не влияет на основное приложение
2026-05-11 09:51:31 +03:00
lekss361
25b73035a1 sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases
- NSPD-skraper переехал в backend/app/services/scrapers/nspd_kn.py +
  Celery task scrape_nspd_region (beat: 20-е февраля/мая/авг/нояб).
  Redis lock 3h, WAF auto-retry, heartbeat в nspd_scrape_runs.
- Recommend_mix Tier 3: per-bucket elasticity через регрессию по
  «доминирующему bucket» каждого ЖК. Weighted-elasticity для inverse-mode.
  UI показывает разброс эластичностей и переключение regression/fallback.
- Cadastre vs market cross-check: spatial-join cad_buildings →
  ekb_districts_geom; cadastre_vs_market_pct в scope, аномалии
  (>+50% / <-30%) подсвечены в UI.
- Sentry release tracking (#4): IMAGE_TAG → backend/.env.runtime →
  sentry_sdk.init(release=...). Compose v2 env_file optional path.

Schemas: 63_schema_nspd_runs.sql (cad_buildings + nspd_scrape_runs/log
формализуют то, что уже жило в проде через 61_import_nspd_batch.py),
64_v_zk_rosreestr_velocity.sql (refresh с cad_buildings).
2026-04-30 21:51:19 +03:00
lekss361
5056438fcf add anal4 2026-04-27 19:45:47 +03:00
lekss361
522ecca5c4 ship #13-16: multi-target Dockerfile, per-object drill-in, CRM pipeline,
442-sweep instructions
#16 Dockerfile: runner (lean, 1.6 GB) + runner-with-chromium (2.9 GB).
   Compose + deploy.yml pushes 2 GHCR images. Backend saves 1.3 GB.
#15 5 backend object endpoints + /analytics/objects/[id] page (gallery,
   POI table, sale chart) + sparkline-driven PrinzipObjectsTable.
#14 prinzip_leads/deals + 2 MVs. Import script with PII-hashing &
   --inspect-only mode. 3 funnel endpoints + Sankey/Monthly charts.
#13 README section with UI/CLI/SQL for 442-object sweep.
2026-04-27 19:32:37 +03:00
lekss361
b7230f1bc8 deploy.yml: sync compose+Caddyfile from git, reload caddy after up 2026-04-26 14:26:57 +03:00
lekss361
b49cebb577 GHCR migration + env defaults + memory updates 2026-04-25 19:47:07 +03:00
lekss361
b95559eac9 init 2026-04-25 13:45:19 +03:00