Commit graph

22 commits

Author SHA1 Message Date
bot-backend
e059e41f5f fix(deps/site-finder): 53 advisory в backend/uv.lock — точечный апгрейд восьми пакетов
All checks were successful
CI Trade-In / browser-tests (pull_request) Has been skipped
CI / changes (pull_request) Successful in 9s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 8s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 2m9s
CI / backend-tests (pull_request) Successful in 17m38s
Прогон всех локов через OSV batch API (2026-08-29) показал: уязвимые версии есть
только в backend/uv.lock. Корневой лок «Меры» (tradein-mvp/uv.lock, из которого и
собирается её образ) чист полностью — ноль находок.

Было 8 пакетов / 53 advisory, стало 0. Проверено повторным прогоном OSV по
получившемуся локу.

  pillow            12.2.0 -> 12.3.0   26 advisory, среди них heap OOB write в
                                       Image.paste/crop и ImageFilter.RankFilter;
                                       Image.open у нас на внешнем входе (загрузка фото)
  starlette         1.0.0  -> 1.6.0    10 advisory, в т.ч. обход лимитов request.form()
  cryptography      47.0.0 -> 50.0.1   7 advisory; транзитив pdfminer-six
  urllib3           2.6.3  -> 2.7.0    4 advisory; обход защиты от decompression-bomb
  weasyprint        68.1   -> 69.0     CSS injection via presentational hints
  idna              3.13   -> 3.19     2 advisory
  mako              1.3.11 -> 1.4.1    path traversal в TemplateLookup (Windows-only,
                                       у нас Linux — берём попутно)
  pydantic-settings 2.14.0 -> 2.15.0   1 advisory

Апгрейд ТОЧЕЧНЫЙ (--upgrade-package на 8 имён), не общий --upgrade: спеки в
pyproject все вида ">=" без верхней границы, и общий апгрейд затянул бы мажоры
вроде redis 7->8 и numpy заодно. Диффом подтверждено: из 138 пакетов изменились
ровно эти 8, прямые спеки в pyproject.toml не тронуты.

Проверка совместимости локально: fastapi 0.136.1 поднимается на starlette 1.6.0,
TestClient отвечает. weasyprint импортировать на Windows нельзя (нет GTK), его
гоняет CI на Linux.

NB: fastapi.testclient на starlette 1.6 предупреждает, что связка с httpx
устарела в пользу httpx2 — это ворнинг, не отказ; отдельная задача.
2026-08-29 13:28:36 +03:00
bot-backend
124cfb3d5d feat(observability): /metrics в обоих бэкендах — счётчики, задержка, дашборд
All checks were successful
CI / backend-tests (pull_request) Successful in 17m30s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 9s
CI Trade-In / browser-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 2m22s
CI Trade-In / backend-tests (pull_request) Successful in 4m51s
Третья часть #3078 и единственная, трогающая прод-код.

До неё числовых рядов у приложений не было вовсе: только логи и исключения в
GlitchTip. Класс отказов «отвечает, но медленно» и «отдаёт 401 потоком» в такой
картине невидим — исключения нет, строка в логе выглядит обычной, а продукт
при этом не работает.

Метка route — ШАБЛОН маршрута, а не путь запроса. Это несущее решение, а не
деталь: кадастровый номер или идентификатор заявки в метке даёт новый временной
ряд на каждую сущность, а ряд у Prometheus стоит памяти постоянно, а не в момент
запроса. Самый известный способ уронить мониторинг тем самым мониторингом.
Незаматченные пути (404, сканеры) сведены в одну метку, иначе тот же взрыв
устроит любой бот, перебирающий адреса. Оба свойства сторожатся тестами, а не
комментарием: тест бьёт тремя разными идентификаторами и требует ОДИН ряд.

Слой регистрируется последним и потому оказывается самым внешним. Изнутри
RBAC-гварда не видно ни отказов авторизации, ни времени, которое он тратит на
резолв сессии в БД auth, — а именно этот путь уже давал инцидент с блокирующим
I/O в middleware (#1202). Упавший исключением запрос считается как 500 в
finally: без этого он просто отсутствовал бы в счётчике, то есть ровно тогда,
когда метрики нужнее всего.

Путь публичен ВНУТРИ и закрыт СНАРУЖИ — это два разных периметра. Скрейп идёт
из docker-сети, где заголовка X-Authenticated-User нет ни у кого, поэтому
/metrics внесён в _PUBLIC_PATHS обоих бэкендов; иначе агент получал бы 401 и
метрик не было бы вовсе. Наружу путь не открывается ни через gendsgn.ru, ни
через meraocenka.ru, и вдобавок закрыт явным respond 404 в обоих site-блоках —
чтобы закрытость осталась решением, а не следствием текущего порядка директив.

Ограничитель частоты и аудит «Меры» не трогались: оба смотрят только на пути
под /api/, скрейп под них не попадает. Проверено тестом, а не чтением.

Прод-поведение не меняется ничем, кроме нового публичного пути: ни один
существующий обработчик, гвард или маршрут не тронут.

Refs #3078
2026-08-26 11:30:18 +03:00
1782dae0e2 chore(tooling): pre-commit, pyproject и uv.lock — один ruff 0.15.20 (#2864) (#3021)
All checks were successful
Deploy / changes (push) Successful in 7s
Deploy / build-frontend (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 11s
Deploy / deploy-caddy (push) Has been skipped
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy / build-backend (push) Successful in 3m57s
Deploy Trade-In / test (push) Successful in 4m5s
Deploy / build-worker (push) Successful in 4m59s
Deploy Trade-In / build-backend (push) Successful in 1m53s
Deploy / deploy (push) Successful in 1m58s
Deploy / deploy-status (push) Successful in 1s
Deploy / perimeter-smoke (push) Successful in 11s
Deploy Trade-In / deploy (push) Successful in 2m27s
Deploy Trade-In / deploy-status (push) Successful in 1s
Deploy Trade-In / perimeter-smoke (push) Successful in 9s
2026-08-21 11:37:23 +00:00
0b3ec7f91b feat(exporters): серверный статик-рендер карт для PDF-отчёта (#2259 PR-C Maps) (#2286)
All checks were successful
Deploy / changes (push) Successful in 8s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 3m47s
Deploy / build-worker (push) Successful in 4m45s
Deploy / deploy (push) Successful in 1m59s
2026-07-03 10:29:23 +00:00
eb251ba7e7 ci(infra): coverage-gate (#68) + OpenAPI codegen-assert (#69) + uptime monitoring (#75)
All checks were successful
CI / changes (push) Successful in 13s
CI / changes (pull_request) Successful in 12s
CI / frontend-tests (push) Successful in 53s
CI / openapi-codegen-check (push) Successful in 1m58s
CI / frontend-tests (pull_request) Successful in 47s
CI / openapi-codegen-check (pull_request) Successful in 1m50s
CI / backend-tests (push) Successful in 8m54s
CI / backend-tests (pull_request) Successful in 8m45s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 2m50s
Deploy / build-frontend (push) Successful in 3m11s
Deploy / build-worker (push) Successful in 4m38s
Deploy / deploy (push) Successful in 2m1s
#68: pytest-cov + [tool.coverage] fail_under=65 (baseline 71%), backend-tests --cov + xml.
#69: openapi-codegen-check job (dump app.openapi() → openapi-typescript → git diff).
  Job использует node_modules-pinned openapi-typescript/prettier (НЕ npx --yes latest —
  иначе version-mismatch ложный diff). + regenerated api-types.ts из СВЕЖЕГО main openapi
  (8288 строк, включая #73 /freshness — синхронен).
#75: Uptime Kuma isolated stack + status.gendsgn.ru Caddy + external Telegram watchdog.
#77: no change — build cache достаточен.

Closes #68
Closes #69
Closes #75
Closes #77
2026-06-13 23:28:06 +05:00
5da4271235 feat(sf): OCR-пайплайн изъятия ЕКБ (Tesseract rus) → land_reservation (#1062)
All checks were successful
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 6m22s
CI / backend-tests (pull_request) Successful in 6m19s
- Dockerfile: добавлены tesseract-ocr tesseract-ocr-rus в runner apt-слой
- pyproject.toml: pymupdf>=1.24 (рендер PDF без poppler) + pytesseract>=0.3.13
- izyatie_client.py: list_izyatie_documents() + fetch_pdf() с екатеринбург.рф
- izyatie_ocr.py: ocr_pdf_text() (PyMuPDF→PIL→Tesseract rus) + extract_izyatie_records()
  с нормализацией OCR-шума (пробелы в кад-номерах, кирилл. О→0, б→6)
- izyatie_ocr_ingest.py: Celery task → land_reservation UPSERT (SAVEPOINT per-row)
- beat_schedule.py: izyatie-ocr-ingest-weekly (пятница 07:00 МСК)
- 33 теста, все зелёные; без реального Tesseract/сети в CI
2026-06-07 18:07:06 +03:00
57c6d119f5 feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites (#1126)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 2m40s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 3m22s
Deploy / deploy (push) Successful in 1m30s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 11:24:49 +00:00
59f2628e0b feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1118)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m41s
Deploy / build-worker (push) Successful in 3m25s
Deploy / deploy (push) Successful in 1m25s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 10:15:03 +00:00
40b1f3ad69 feat(export): add PPTX presentation export of §22 forecast report (#959)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
Deploy / changes (push) Successful in 7s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m58s
Deploy / build-worker (push) Successful in 3m26s
Deploy / deploy (push) Successful in 1m26s
CI / backend-tests (pull_request) Successful in 6m7s
Completes EPIC #959 export formats (md/json/tg/docx/pptx; PDF/XLSX pre-existing).
render_report_pptx (python-pptx, pure + graceful, reuses report_pdf helpers,
local import) → 6-slide deck. Endpoint format=pptx (early-return, presentationml
media-type, .pptx attachment). Add python-pptx + regenerate uv.lock (python-pptx
+ xlsxwriter; uv sync --frozen passes).

Closes #959.
2026-06-07 14:09:35 +05:00
3a89287c6b feat(export): add DOCX export of §22 forecast report (#959)
All checks were successful
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 6s
CI / backend-tests (push) Has been skipped
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m49s
Deploy / build-worker (push) Successful in 3m27s
Deploy / deploy (push) Successful in 1m13s
CI / backend-tests (pull_request) Successful in 6m4s
render_report_docx (python-docx) mirrors report_md/report_pdf section order &
content, reuses report_pdf pure helpers (DRY), graceful on thin/empty report.
Widen /forecast/export format Literal to include docx → Word attachment.
Add python-docx dep + regenerate uv.lock (uv sync --frozen passes). Part of #959.
2026-06-07 13:58:04 +05:00
88cdfd6adb feat(rbac): role-based access control via X-Authenticated-User middleware (#585)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Successful in 27s
Deploy / build-frontend (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 1m8s
Deploy Trade-In / deploy (push) Successful in 45s
Deploy / build-backend (push) Successful in 2m40s
Deploy / build-worker (push) Successful in 3m20s
Deploy / deploy (push) Successful in 1m16s
Backend RBAC + Caddy header_up. Closes part of #fixes-rbac-pra.
2026-05-26 06:18:40 +00:00
4040cba4c0 feat(sf-b7): GET /parcels/{cad}/snapshot.pdf — 1-page WeasyPrint export (#334)
Some checks failed
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Has been cancelled
Deploy / build-backend (push) Has been cancelled
2026-05-17 21:11:03 +00:00
389e141ec9 fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 3m0s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 5m56s
Deploy / deploy (push) Successful in 1m19s
2026-05-17 16:37:30 +00:00
b61f025673 feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207)
Some checks failed
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 4s
Deploy / build-backend (push) Has been cancelled
Deploy / build-worker (push) Has been cancelled
2026-05-16 15:09:16 +00:00
lekss361
b3f32ae927
chore(infra): correct frontend healthcheck + tighten depends_on (re-apply #117) (#120)
Forward-fix после incident May 14 (см. vault events/Incident_Prod_Down_May14_2026).
PR #117 был полностью reverted PR #119 (Caddy depends_on frontend: service_healthy
+ failing wget --spider /  → Caddy не стартовал → vsё прод down ~25 мин).

Этот PR делает то же что планировал #117, но корректно:

### Frontend healthcheck — TCP probe через node (НЕ HTTP)

```yaml
test: ["CMD", "node", "-e",
  "require('net').createConnection({port:3000,host:'127.0.0.1'})
   .once('connect',function(){process.exit(0)})
   .once('error',function(){process.exit(1)})"]
start_period: 60s
retries: 6
interval: 15s
```

- НЕ зависит от HTTP status (200/404/308 — все OK если порт слушает)
- `node` гарантированно в alpine image (запускает server.js)
- `127.0.0.1` — никакой DNS ambiguity (busybox alpine localhost IPv6 vs IPv4)
- 60s + 6×15s = 150s window — с запасом на Next.js standalone cold start

### Caddy depends_on — backend healthy, frontend started (НЕ healthy)

```yaml
caddy:
  depends_on:
    backend: { condition: service_healthy }    # backend имеет /health endpoint
    frontend: { condition: service_started }   # safety net — НЕ блокирует Caddy
```

Lesson from incident: frontend health flaky (Next.js cold start variable).
Strict service_healthy для frontend → одна misconfig = full outage обоих доменов
(gendsgn.ru + obsidian.gendsgn.ru через тот же Caddy на shared network).

### backend/worker/beat redis: service_started → service_healthy

Redis уже имеет `redis-cli ping` healthcheck. Tightening безопасно — был
artifact pre-healthcheck Redis.

### Pin rosreestr2coord>=5.0.0 (lock уже 5.3.3)

v5 убрала delay параметр; код адаптирован (rate limit на worker уровне).
Защита от silent downgrade при `uv lock --upgrade`.

### Что НЕ повторено из #117

- Caddy depends_on frontend: service_healthy — заменено на service_started
- wget --spider / — заменено на node TCP probe

### Vault обновлён

- `events/Incident_Prod_Down_May14_2026.md` — постмортем + timeline + lessons
- Update Changelog в `domains/infra/infra-MOC.md` (если есть)

Closes incident. Re-applies #117 goals safely.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:24:47 +03:00
lekss361
6ea2dcd46c
Revert "chore(infra): healthcheck frontend + tighten depends_on; pin rosreestr2coord>=5 (#117)" (#119)
This reverts commit bade2f7772.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:02:58 +03:00
lekss361
bade2f7772
chore(infra): healthcheck frontend + tighten depends_on; pin rosreestr2coord>=5 (#117)
- Add wget healthcheck to frontend service (alpine-compatible) — Caddy
  больше не роутит трафик до того как Next.js готов.
- Caddy depends_on backend/frontend: service_started → service_healthy.
- backend/worker/beat redis dependency: service_started → service_healthy
  (Redis уже имеет redis-cli ping healthcheck).
- Pin rosreestr2coord>=5.0.0 (v5 убрала delay параметр; код уже адаптирован
  per CLAUDE.md); uv resolved 5.3.3.

Closes Day-1 quick wins #3+4 from code review audit (May 14).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 22:42:09 +03:00
lekss361
ead0de99ed fix(worker): rosreestr2coord in lockfile + admin logs panel for NSPD/objective
Two fixes for the NSPD geo bulk-fetcher:

1. rosreestr2coord was listed in pyproject.toml but missing from uv.lock,
   so Docker's `uv sync --frozen` didn't install it. NSPD jobs failed every
   target with `No module named 'rosreestr2coord'`. Ran `uv lock` to add it
   (v5.3.3) plus requests + charset-normalizer transitive deps.

2. /admin/scrape/geo and /admin/scrape/objective lacked the log panel that
   /admin/scrape (DomRF) had. Extracted ScrapeLogsPanel component over the
   unified /api/v1/admin/scrape/all/logs endpoint with scraper_type filter
   and wired into both pages.

Tests: ruff ✓, tsc --noEmit ✓
2026-05-11 12:47:19 +03:00
lekss361
1507575698 fix alch 2026-04-27 22:03:48 +03:00
lekss361
621d56c064 fix 2026-04-27 18:12:57 +03:00
lekss361
4f034bd86b ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts
- backend/alembic/* — alembic infra (env.py, script.py.mako). versions/ empty
  for now; first migration goes in Stage 2a when models are finalized.
- backend/Dockerfile: bake alembic.ini + alembic/ into the image so
  `docker compose exec backend alembic upgrade head` works in prod.
- backend/db/init/99_drop_unused_extensions.sql + bind-mount in both compose
  files: drops postgis-image's TIGER/topology/fuzzystrmatch on fresh volumes.
- .pre-commit-config.yaml + pre-commit in dev deps: ruff/prettier on commit
  to stop CI failures like UP035 from leaking out.
- ops/backup.sh, ops/restore.sh: pg_dump cron script with optional Selectel S3
  upload. 7-day local retention. Restore guard: requires RESTORE_CONFIRM=yes.
- Makefile: new targets `make migration NAME=...`, `make pre-commit-install`.
- backend/.env.example: SENTRY_DSN comment with sentry.io reference.
2026-04-26 13:08:51 +03:00
lekss361
e22aeb9b5a Docker: multi-stage backend, npm ci, healthcheck fix, same-origin API
- backend/Dockerfile: split builder/runner, runtime libs only, non-root app user, curl for healthcheck, --frozen via new uv.lock
- frontend/Dockerfile: npm ci instead of npm install (deterministic), USER node
- docker-compose.prod.yml: working backend healthcheck (curl now in image), redis healthcheck, drop dead NEXT_PUBLIC_API_BASE_URL env (Next.js bakes NEXT_PUBLIC_* at build time, runtime override is no-op)
- docker-compose.yml: same redis healthcheck, BACKEND_URL for next.config rewrites, drop uv from CMD (not in new image)
- Caddyfile: handle (not handle_path) so /api prefix is preserved into FastAPI router
- next.config.ts: rewrite /api/* and /health to BACKEND_URL in dev (no Caddy locally)
- frontend/src/lib/api.ts: empty default = same-origin relative URLs
- Makefile: drop uv run from migrate target
- Add backend/uv.lock and frontend/package-lock.json for reproducible builds

Verified: docker build succeeds for both, backend container starts, /health responds, curl healthcheck works inside image, container runs as non-root.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 10:47:35 +03:00