Commit graph

148 commits

Author SHA1 Message Date
lekss361
f08db43145 feat(#263 sub-A): cad_* overlay schemas + cadastre territorial_zones (ПЗЗ)
Phase 1 (migration 101): создаёт 5 новых таблиц cad_territorial_zones,
cad_red_lines, cad_engineering_structures, cad_risk_zones,
cad_opportunity_parcels. Idempotent (IF NOT EXISTS), GIST-индексы.

Phase 2 partial (ПЗЗ only):
- NSPDBulkClient.get_features_in_bbox_grid — async grid-walk WMS GetFeatureInfo
- NSPDBulkClient.get_territorial_zones_in_bbox — convenience wrapper layer 875838
- harvest_quarter Phase 2.5 — вызов get_territorial_zones_in_bbox после grid-walk
- _save_territorial_zones — UPSERT loop с SAVEPOINT per-row, ON CONFLICT zone_id

9 unit tests (mock DB): insert/upsert/no-geom/no-id/error-isolation/savepoint pattern.
2026-05-17 10:28:22 +03:00
lekss361
485f489bd2 feat(#254): user_custom_pois — backend schema + CRUD + scoring integration
- SQL migration 101_user_custom_pois.sql: table with GEOGRAPHY(POINT) + GIST index
- Pydantic schemas: CustomPoiCreate / CustomPoiUpdate / CustomPoiOut
- Service custom_pois.py: CRUD + get_overlaps_for_scoring (1km radius, psycopg v3 CAST)
- API /api/v1/custom-pois: POST(201)/GET/PATCH/DELETE, X-Session-Id header auth
- Scoring in analyze_parcel: custom POI block after OSM POI loop,
  decay weight * max(0, 1 - dist/1000), custom_poi_score_items in response
- Tests: CRUD + validation (weight/lon/lat 422) + scoring integration (mock-based)
- Vault: Schema_User_Custom_Pois.md + Endpoint_Parcel_Analyze v3.9 section
2026-05-17 09:34:33 +03:00
lekss361
a6e4ff0407 feat(#114): seed 3 default weight presets + include_system API param
Add SQL migration 100_user_weight_profiles_default_seed.sql with system
presets Эконом/Комфорт/Бизнес (user_id='__system__'). Migration is
idempotent via ON CONFLICT DO UPDATE.

Backend:
- weight_profiles.py: add SYSTEM_USER_ID constant + list_profiles_with_system()
- admin_weight_profiles.py: add include_system query param to GET list endpoint

Tests: 3 new tests covering include_system flag and service sentinel behaviour.
2026-05-16 22:49:56 +03:00
lekss361
f704785428 fix(sql): escape apostrophe in migration 99 COMMENT (deploy hotfix)
PR #222 deploy failed на 99_nspd_entities_denorm.sql:
  ERROR: syntax error at or near "а"
  LINE 2: '3-сегментный квартал последнего harvest'а. Используется...

Unescaped apostrophe в COMMENT string closes literal early.
Fix: harvest'а → harvest''а (SQL escape).

Других unescaped апострофов в SQL литералах нет (другие 3 — в SQL комментариях
'--' , безопасно).
2026-05-16 21:02:00 +03:00
lekss361
db1ff5b1fc feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4)
- data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes
  (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%')
- nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump
  - SAVEPOINT per row via with db.begin_nested()
  - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326)
  - json.dumps(geom, ensure_ascii=False) for psycopg bind
  - All CAST(:x AS type) — no :type
- nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal)
- nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit)
- admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth)
- 19 tests: coerce helpers, denorm parcel/building, aggregate counts

Foundation for downstream features (МКД neighbors lookup, parcel attrs).

Part of #94
2026-05-16 20:20:03 +03:00
lekss361
fdb54834e5 fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length (#220)
- data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin
- _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of
  ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4
  tolerance error and returns correct non-zero length for LINESTRING intersections
- Rename intersection_area_sqm → intersection_length_m across schema, TS types,
  frontend component, and tests; add test_get_red_lines_db_exception_returns_empty

Addresses review-bot feedback on PR #220.
2026-05-16 19:33:14 +03:00
lekss361
ed3c128528 feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4)
- NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt)
  + QuarterDump.opportunity field
- nspd_sync: harvest_quarter accepts include_opportunity, denorm cols
  has_auction_parcels + opportunity_count in UPSERT
- quarter_dump_lookup:
  - _get_opportunity_parcels (sort by distance, early-exit on count=0)
  - _get_red_lines (query existing dump.red_lines core path, layer='red_lines')
- SQL 89_*: has_auction_parcels + opportunity_count + partial index
- Pydantic: OpportunityParcel + RedLine schemas
- /analyze: nspd_opportunity_parcels + nspd_red_lines fields
- Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration
- Tests: 28 total (11 PR1 + 17 PR2), all pass

Red lines uses existing core harvest path (layer 879243 already in dump.red_lines
from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed.

Part of #94
2026-05-16 19:06:22 +03:00
dbf31cc911 feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)
All checks were successful
Deploy / changes (push) Successful in 4s
Deploy / build-backend (push) Successful in 1m38s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m36s
Deploy / deploy (push) Successful in 38s
2026-05-16 12:47:09 +00:00
adc45f5724 feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)
All checks were successful
Deploy / changes (push) Successful in 4s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
Deploy / deploy (push) Successful in 39s
2026-05-16 11:41:21 +00:00
efa66a9f68 feat(analytics): mv_quarter_price_per_m2 MV + refresh helper (#33 D1 PR A) (#209)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m38s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m47s
Deploy / deploy (push) Successful in 54s
2026-05-16 11:12:01 +00:00
6aa22ef8ba feat(analytics): mv_layout_velocity MV + refresh helper (#113 PR B) (#194)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m50s
Deploy / deploy (push) Successful in 48s
2026-05-16 08:26:59 +00:00
lekss361
5f8df655cc
fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185)
NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4)
which crashed upsert_parcel — schema was strict POLYGON.

Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom),
DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom.

upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to
MultiPolygon. NSPD returns both types (simple + multi-contour parcels).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 18:52:38 +03:00
lekss361
37e4f854c5
fix(db): retire ALL legacy cad_buildings index names before rename (#168 hotfix2) (#174)
PR #173 fixed cad_buildings_pkey but missed 5 other indexes. Same root cause:
PG does NOT auto-rename indexes on table rename → all 6 backing indexes
remain on cad_buildings_old_apr26 with original names.

Add 5 more `ALTER INDEX IF EXISTS ... RENAME TO ..._old_apr26_*` statements
for: geom_gist, quarter_idx, objdoc_idx, complex_idx, purpose_idx.

Migration 92 still NOT in _schema_migrations (4× rollback now). Next deploy
will re-apply cleanly.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 14:09:40 +03:00
lekss361
7b2d1782e9
fix(db): retire legacy cad_buildings_pkey before rename — fix migration 92 (#168 hotfix) (#173)
Migration 92_cad_bulk_layers.sql fails on deploy with:
  ERROR: relation "cad_buildings_pkey" already exists
  CONTEXT: ALTER INDEX cad_buildings_new_pkey RENAME TO cad_buildings_pkey

Root cause: K.1 renamed cad_buildings → cad_buildings_old_apr26, but PG
does NOT auto-rename the backing PK index on table rename. The legacy
cad_buildings_pkey index is still attached to cad_buildings_old_apr26.
K.3 then tries to rename cad_buildings_new_pkey to the same name → collision.

Fix: add an explicit `ALTER INDEX IF EXISTS cad_buildings_pkey RENAME TO
cad_buildings_old_apr26_pkey` BEFORE the new rename. Idempotent (IF EXISTS).

Migration 92 was rolled back in last 3 deploy attempts (not in
_schema_migrations), so this edit will re-apply cleanly on next deploy.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:59:53 +03:00
lekss361
59f222eaf3
feat(db): bulk cadastre schema — 7 tables + cadastre_jobs (#168 PR1/5) (#169)
* feat(db): schema migration for bulk cadastre ingest layers (#168)

Add 7 cadastre layer tables + cadastre_jobs saga-state table.

Tables:
- cad_parcels (36368 ЗУ ЕГРН) replaces cad_parcels_geom
- cad_buildings v2 wide schema (replaces existing, backed up as cad_buildings_old_apr26)
- cad_constructions (36383 Сооружения + ЕГРН, flag is_egrn)
- cad_oncs (36384), cad_enk (39663), cad_zouit (4 categoryIds), cad_quarter_stats
- cadastre_jobs (resumable saga state, mirrors geo_jobs pattern)

GIST + BTREE indexes. Migration of cad_parcels_geom data via INSERT...SELECT.
Restore v_complex_full + v_complex_buildings views after table rename.

* fixup(db): address PR #169 bot review — 2 blockers + 3 lessers

Blocker #1: cad_parcels_geom DROP → back-compat VIEW
  Old DROP TABLE would break site-finder analyze fallback (per 83_*.sql COMMENT).
  Now: drop old BASE TABLE (guarded by table_type), create VIEW over cad_parcels
  with identical 4-column signature. Callers continue to work; PR2/5 will switch.

Blocker #2: K.3 redundant RENAME CONSTRAINT removed
  ALTER INDEX RENAME on PK index already renames the backing constraint in PG16.
  Second ALTER TABLE RENAME CONSTRAINT would throw 'does not exist' and abort tx.

Lesser #1: cadastre_jobs.status CHECK constraint
  Inline CHECK (status IN ('queued','running','done','failed','zombie','cancelled','paused'))
  per partial-index intent and admin UI plan.

Lesser #2: COMMENT ON TABLE for all 8 tables + view (keeps convention from 63, 83).

Lesser #3: set_updated_at() triggers on 7 tables with updated_at column.
  Reuses existing function (verified present). DO $$ guards on pg_trigger
  for idempotent re-apply. cadastre_jobs excluded (uses heartbeat_at instead).

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:05:25 +03:00
lekss361
7c43bc5496
feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150) (#151)
* fix(frontend): apiFetchWithStatus body stream double-read crash

Per user report 2026-05-14: 'Failed to execute text on Response: body stream
already read' при analyze с non-JSON error response.

apiFetchWithStatus делал .json() (consumes stream), потом .text() в catch
(FAILS — stream already consumed). Fetch API: body — ReadableStream,
consumable один раз.

Fix: read text() once → JSON.parse on string. На fail (HTML / non-JSON)
оборачиваем в {detail: rawText}. Никаких double-reads.

Site-finder cad search больше не крашится на error pages.

* feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150)

Закрывает root cause production 500 на /api/v1/admin/site-finder/weight-profiles
(а также 2 unapplied migrations: #89/#91).

## Why это нужно

После audit обнаружили что:
- backend/alembic/versions/ пустой (Alembic configured но не используется)
- deploy.yml не имеет migration step
- 47+ raw SQL files накопилось без auto-apply
- 3 файла unapplied: 87 (engineering), 89 (drop brin), 90 (weight profiles)

User report: GET weight-profiles → 500 потому что user_weight_profiles
table не существует на prod.

## Changes

### deploy.yml +39 lines
- Path trigger расширен на data/sql/*.sql
- New step 'Apply DB migrations' между compose pull и compose up -d:
  - Idempotent CREATE TABLE _schema_migrations
  - Loop по data/sql/*.sql sorted → skip applied → psql with ON_ERROR_STOP=on
  - Record applied filename в tracking table
  - Exit 1 на failure → containers НЕ обновляются (no partial state)

### NN collision fix
- 87_engineering_networks_191fz.sql → 91_engineering_networks_191fz.sql
  (collision с 87_on_demand_indexes.sql, мой renumber 85→87 был неудачным)

### CLAUDE.md +32 lines
- Subsection 'Auto-apply' под Migration pattern
- Bootstrap procedure docs
- Idempotency requirement
- Failure recovery

### scripts/bootstrap_schema_migrations.sh +100 lines
- One-time seed _schema_migrations с 48 already-applied files
- 3 файла intentionally NOT seeded: 89/90/91 (auto-apply на first deploy)
- Usage docs + verification query

## Vault

domains/infra/Runbook_Auto_Apply_Migrations.md NEW
infra-MOC.md updated

## Required manual steps ДО merge

1. SSH gendesign (tunnel)
2. POSTGRES creds export
3. bash scripts/bootstrap_schema_migrations.sh — seed 48 already-applied files
4. Verify: SELECT COUNT(*) FROM _schema_migrations → 48
5. После merge → first deploy auto-apply #89/#90/#91 → 500 closes

Closes #150

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:06:54 +03:00
lekss361
f3f1e7f6bc
feat(db): schema user_weight_profiles (#114 sub-PR 1/4 — foundation) (#136)
Per #114 Custom POI weights — Макс feedback "садики и Мегамарт должны
иметь разные веса". Sub-PR 1/4 per feedback_split_big_issues:

1. **#114 sub-PR 1 (this)**: schema migration
2. #114 sub-PR 2: backend Pydantic models + CRUD service
3. #114 sub-PR 3: POST /parcels/{cad}/analyze?profile= override + admin endpoint
4. #114 sub-PR 4: frontend 25 sliders UI + live preview

## Schema

`data/sql/90_user_weight_profiles.sql`:

- Table `user_weight_profiles` — SERIAL PK + user_id + profile_name +
  weights JSONB + is_default + timestamps
- Unique constraint (user_id, profile_name)
- B-tree index user_idx
- Partial index `WHERE is_default = TRUE` (fast default lookup)
- Trigger auto-updated_at via `set_updated_at()` function

## Compat with existing POI categories

`weights` JSONB compatible с current `_POI_WEIGHTS` const в
`backend/app/api/v1/parcels.py`:
- school 1.5, kindergarten 1.5, pharmacy 0.8, hospital 0.6
- shop_mall 1.2, shop_supermarket 1.0, shop_small 0.5
- park 1.8, bus_stop 0.3, metro_stop 1.5, tram_stop -0.5

## Idempotency

- BEGIN/COMMIT обёртка
- IF NOT EXISTS на CREATE TABLE/INDEX
- DROP TRIGGER IF EXISTS → CREATE TRIGGER (safe re-run)
- CREATE OR REPLACE FUNCTION (idempotent)

## Apply

```bash
psql postgresql://gendesign:...@localhost:15432/gendesign \
  -f data/sql/90_user_weight_profiles.sql
```

После apply таблица inert (нет backend code yet). Sub-PR 2 добавит CRUD.

## Vault

`code/schemas/Schema_User_Weight_Profiles.md` — created (status:
ready_to_apply, full sub-PR plan).

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:19:33 +03:00
lekss361
8a266502a1
fix(sql): wrap 73_complexes_master.sql in BEGIN/COMMIT (atomic apply) (#125)
Per code review audit (May 14): файл без transaction wrapper — backfill CTE
INSERTs + view recreations executed как individual auto-commit statements.
Mid-file crash → orphan rows в complexes/complex_sources с no rollback path.

## Fix

`BEGIN;` после header comments (line 31, перед section 1).
`COMMIT;` в конце файла после последнего COMMENT ON VIEW.

Wraps все 6 секций atomically:
1-2: CREATE TABLE + CREATE INDEX для complexes + complex_sources
3-5: 3 backfill CTE INSERTs с ON CONFLICT DO NOTHING
6: DROP VIEW IF EXISTS + CREATE VIEW + COMMENT для v_complex_full

## No CONCURRENTLY split needed

Проверил — все 10 indexes используют plain `CREATE INDEX IF NOT EXISTS` без
CONCURRENTLY keyword. Безопасно обернуть в одну tx.

## Idempotency preserved

Все existing guards остаются:
- CREATE TABLE/INDEX IF NOT EXISTS
- ON CONFLICT (source, source_id) DO NOTHING
- DROP VIEW IF EXISTS

Re-running file двукратно — safe.

## Vault

`fixes/Migration_73_Complexes_Atomic_May14.md` — created (status: resolved).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:28 +03:00
lekss361
7b32edc34d
fix(db): DROP dead BRIN index rosreestr_deals_period_brin (#122)
Per code review audit (May 14): BRIN index на partitioned parent table dead
weight. PG16 НЕ propagates такие индексы на child partitions; pg_stat
показывает idx_scan=0 → никогда не использовался.

## Why dead

rosreestr_deals — yearly partitioned по period_start_date. Queries делают
partition pruning через partition key. BRIN index на parent существует, но:
1. На child partitions его нет (PG16 не reflects)
2. Parent сам не имеет данных (partitioned out)
3. Queries никогда не trigger BRIN scan

Maintenance overhead (storage, autoanalyze) без value.

## Migration

`data/sql/89_drop_dead_brin_rosreestr_deals.sql` — атомарный DROP с idempotent
`IF EXISTS`. Apply order: после 88_*.

## Cleanup

`data/sql/01_schema_rosreestr_deals.sql:65-67` — закомментировал оригинальное
CREATE с note: fresh DB bootstrap НЕ создаст index заново.

## Rollback path

Если в будущем нужен range scan optimization — re-create per-partition:
```sql
CREATE INDEX rosreestr_deals_2024q1_period_brin
  ON rosreestr_deals_2024q1 USING BRIN (period_start_date);
-- per-partition × N partitions
```

Но обычно partition pruning достаточно. Измерь EXPLAIN перед re-creation.

## Vault

`fixes/Migration_Drop_Dead_Brin_May14.md` создан (status: ready_to_apply).

Apply на prod: `psql ... -f data/sql/89_drop_dead_brin_rosreestr_deals.sql`
после merge.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:01 +03:00
lekss361
d4cb56d093
fix(sql): renumber engineering_networks 85→87 + dedup classid + add SRID 4326 (#118)
* fix(sql): renumber engineering_networks 85→87, dedup classid, add SRID

Three critical fixes на untracked 85_engineering_networks_191fz.sql per
code review (May 14) — до первого применения на prod.

1. **NN collision fix**: renumber 85_engineering_networks_191fz.sql → 87_
   85_pzz_zones_ekb.sql уже занимает 85_ prefix. При alphabetic deploy
   order `85_e*` применился бы ДО `85_p*` — но 87_ ставит нас после
   86_v_bucket_success_score.sql без зависимости на 85_pzz.

2. **Duplicate classid '907015504'**:
   - Был дважды: 'Опора металлическая' + 'Опора железобетонная'.
   - Второй ON CONFLICT DO UPDATE молча перетёр бы первый → NSI код
     металлической опоры терялся бы.
   - Fix: 907015504 остаётся за металлической (lower-numbered subtype
     per Минстрой NSI source), железобетонная переезжает на 907015507
     (заполняет gap 906→508). Помечено comment-меткой 'verify with
     Минстрой NSI table' до prod deploy.

3. **Untyped geometry**:
   - Было: `geom GEOMETRY` без SRID constraint → разрешало бы insert
     mix MSK-66 (srid=100000) и WGS84 (srid=4326) → silent ST_DWithin
     corruption.
   - Fix: `geom GEOMETRY(Geometry, 4326)`. Capital G позволяет
     mixed Point/LineString/Polygon, SRID 4326 жёстко фиксирует WGS84.
     ETL loader должен делать ST_Transform(msk66_raw, 4326) перед
     insert (см. backend/app/core/crs.py для CRS utility).

Idempotency: BEGIN...COMMIT, IF NOT EXISTS, ON CONFLICT DO UPDATE
паттерн — уже корректные в файле.

Closes Day-1 critical #2 from code review audit (May 14).
Pre-deploy: verify 907015507 classid assignment against Минстрой
official NSI table; reassign and re-run migration if needed.

* fix(sql): remove provisional 907015507, fix Фонати→Фонари typo

Addresses review blocker on PR #118:

- Provisional classid 907015507 ('Опора железобетонная (столб)') убран
  из seed insert. Commented out с TODO(NSI-verify) — следующий шаг
  верифицировать против официальной NSI-таблицы Минстроя ДО восстановления
  строки. Без верификации ON CONFLICT DO UPDATE name перетёр бы корректное
  значение в eng_ref_classid.
- Опечатка 'Фонати электрические' → 'Фонари электрические' (907015602).
- Sort: 904015504/505/506/508 теперь в численном порядке (после удаления
  провизорной 507).

Live seed теперь покрывает 504 (металл столб), 505 (металл ферма),
506 (металл фермовый столб), 508 (железобет ферма). Строка 507
restored after Минстрой NSI verification (см. follow-up).

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 22:51:11 +03:00
lekss361
6fce81e10c
feat(db): add nspd_quarter_dumps table for PKK harvest persistence (#94 pt.3/4) (#110)
Sprint 1.1 item #2 из плана #94 part 2. Persistent storage для QuarterDump
snapshots (PR #109). Foundation для Celery harvest_quarter task (next PR)
и analyze_parcel integration (после).

Schema (data/sql/88_nspd_quarter_dumps.sql):
- Table `nspd_quarter_dumps`: 17 columns matching QuarterDump dataclass.
  - PK quarter_cad (natural key, one row per quarter)
  - quarter_geom (MultiPolygon, 4326) + bbox_3857 (Polygon, 3857) — dual-index
    стратегия: bbox для scheduler queries без transform, geom для joins с
    PZZ/district в 4326
  - 7 per-layer count columns + total_features (computed denorm для быстрых
    статистических queries без распаковки JSON)
  - features_json JSONB — array of ~117 NSPDFeature objects (~120KB per row,
    TOAST handles transparently). Geometry в raw EPSG:3857 — caller transforms.
  - layers_fetched TEXT[] mirrors QuarterDump.layers_fetched tuple
  - harvest_duration_ms (nullable — NULL = "not measured"), harvest_error TEXT
  - region_code SMALLINT NOT NULL (no DEFAULT — explicit per insert)
- 6 indexes: PK, GIST(bbox_3857), GIST(quarter_geom), B-tree(fetched_at_utc DESC),
  B-tree(region_code), partial B-tree(fetched_at_utc WHERE harvest_error IS NULL)
- View v_quarter_dumps_freshness — admin monitoring (AGE + is_failed flag)

Конвенции:
- BEGIN/COMMIT atomic
- IF NOT EXISTS на каждом CREATE — idempotent
- COMMENT ON TABLE + 7 COMMENT ON COLUMN + comments на каждом index
- psycopg v3 compatible (DDL only, no %s)

Применено к prod в рамках database-expert verification (DDL only, idempotent).

Vault entry: code/schemas/Schema_Nspd_Quarter_Dumps.md (upsert template +
EXPLAIN plans + cross-refs).

Code review (code-reviewer pre-push): APPROVE, 2 minor non-blocking:
- MINOR-1 quarter_geom MultiPolygon vs Polygon — resolved в Celery PR via
  ST_Multi(ST_Transform(...)) на ingest contract
- MINOR-2 layers_fetched default '{}' — documented в comment как
  partial-failure path для error rows

Next: Celery harvest_quarter task + beat schedule.

Part of #94 Sprint 1.1.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 18:22:50 +03:00
lekss361
49fac806fc
feat(site-finder): auto-fetch cadastre geometry on-demand (#93) (#95)
* feat(site-finder): auto-fetch cadastre geometry on-demand (#93)

Когда пользователь вводит cad-номер которого нет в БД (cad_quarters_geom /
cad_buildings / cad_parcels_geom), вместо 404 «Загрузи через NSPD geo» (dead-
end для non-admin) — теперь backend автоматически инициирует NSPD fetch,
frontend показывает loading state с polling /fetch-status каждые 2с.

Backend:
- Новый модуль app/services/site_finder/cadastre_fetch.py с helpers:
  find_or_enqueue_fetch (atomic check + enqueue с дедупликацией по cad),
  fetch_status (smart polling endpoint — отличает not_in_nspd от failed),
  detect_thematic_id (3-сегм quarter / 4-сегм parcel / 5-сегм building),
  validate_cad_format.
- Reuse: enqueue_geo_job + process_nspd_geo_job (workers/tasks/nspd_geo).
  source_kind='auto_on_demand' отличает от bulk; rate_ms=200, priority=9.
- POST /parcels/{cad}/analyze graceful fallback: inline await до 25s
  (fast path), затем 202 + job_id + eta_seconds для polling, либо
  400/404/503 в зависимости от статуса (с Retry-After 60s на 503).
- GET /parcels/{cad}/fetch-status новый endpoint для polling.

Frontend:
- useSiteAnalysis расширен: fetchingState + cancel(). POST analyze + при 202
  polling каждые 2с (max 60 итераций = 2 мин cap). status=ready → re-trigger
  analyze; not_in_nspd/failed/invalid_format → typed errors.
- apiFetchWithStatus<T> + HTTPError в lib/api.ts — status-aware variant
  для 200 vs 202.
- FetchingState.tsx: animated spinner, progress bar (linear до etaSeconds,
  asymptote после), elapsed counter, cancel button. Светло-голубая
  scheme отличается от обычного pending skeleton.
- site-finder/page.tsx: FetchingState когда fetchingState активен; обычный
  pending skeleton — только при первичном analyze без 202.

Edge cases (per #93 acceptance):
- cad валидный в НСПД, fetch <25s → inline 200 OK
- cad валидный, fetch >25s → 202 + frontend polls → ready → analyze
- cad валидный, не в НСПД → 404 с понятным сообщением + формат hint
- cad invalid format → 400 + формат hint
- NSPD rate-limited / failed → 503 + Retry-After 60s
- Параллельные запросы на тот же cad → один job, оба клиента poll'ят (дедуп
  через find_active_on_demand_job).

Closes #93.

* fix(site-finder): address PR #95 auto-review minor feedback

Backend (cadastre_fetch.py):
1. (race condition) Advisory lock `pg_try_advisory_xact_lock(hashtext(cad_num))`
   обёрнут вокруг шагов "check active job → enqueue" в find_or_enqueue_fetch.
   Lock transaction-scoped, released at COMMIT. Параллельные запросы на тот
   же cad: первый получает lock и enqueue; второй lock=false → re-check
   active job (уже виден после первого COMMIT) → возвращает тот же job_id.
   Docstring обновлён, упоминание SELECT FOR UPDATE удалено.

Backend (parcels.py):
3. (threadpool exhaustion) _INLINE_FETCH_WAIT_S снижен 25 → 15s с подробным
   комментарием: tradeoff про default Starlette anyio threadpool (40 slots)
   и concurrent burst saturation. 15s баланс: НСПД avg 5-15s для quarter,
   ~70% fast path; остальные 30% получают 202 + polling без блока.

Data (87_on_demand_indexes.sql):
2. (missing index) New migration:
   - `nspd_geo_targets_cad_num_idx ON nspd_geo_targets(cad_num)` — для
     find_active/recent_completed_job (existing UNIQUE composite не покрывает
     WHERE cad_num=:c).
   - `nspd_geo_jobs_source_status_idx ON nspd_geo_jobs(source_kind, status)`
     composite для filter auto_on_demand + queued/running.
   Idempotent (CREATE INDEX IF NOT EXISTS), не блокер при текущем размере,
   критично при росте on-demand traffic.

Frontend (useSiteAnalysis.ts):
4. (UI flicker) setFetchingState(null) перемещён ПОСЛЕ `await second
   apiFetch`. Иначе между clear и resolve есть момент когда mutation
   isPending=true + fetchingState=null → пустой экран ~1 RTT.

NOT addressed (rebuttal):
5. (Tailwind convention) — проверил: в проекте нет ни globals.css, ни
   @tailwind directives. ВСЕ существующие site-finder components используют
   inline styles (ConfidenceBadge, GeotechRiskBlock, ScoreBreakdownPanel etc).
   Tailwind в deps но не wired up. Keep inline styles для consistency.
6. (animate-spin) — требует Tailwind globals (см. #5). `<style jsx>`
   keyframes — built-in Next.js, работает.

Per auto-review on 2252236.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 09:02:17 +03:00
lekss361
369b5a4706 feat(db): v_bucket_success_score view (success-driven mix #25) 2026-05-11 22:04:22 +03:00
lekss361
8be539ddc6 feat(site-finder): ПЗЗ territorial zones from Rosreestr PKK6 + zoning in analyze 2026-05-11 21:51:51 +03:00
lekss361
a2e8a49e6d feat(db): osm_noise_sources_ekb for site-finder noise scoring 2026-05-11 20:35:27 +03:00
lekss361
81cd7499f6 feat(site-finder): cad_parcels_geom migration + POI markers on map + bulk-job UX 2026-05-11 19:54:56 +03:00
lekss361
688e094844 feat(db): osm_poi_ekb table for site-finder POI analysis 2026-05-11 18:06:28 +03:00
lekss361
cbe1253461 feat(db): backfill complexes nulls + add job_settings (migrations 80, 81)
80_complexes_backfill_nulls.sql:
- cad_quarter: +722 rows via ST_Contains spatial join on cad_quarters_geom
- developer_id: +247 rows via fuzzy name match on domrf_developers
- obj_class: +2 rows via objective_lots.class mode
- cad_building_num TEXT dropped (100% NULL; relation lives in
  cad_buildings.complex_id)
- v_complex_full rebuilt: cad_building_nums TEXT[] replaces dead column
- v_complex_buildings created: complex_id → cad_building_nums[], buildings_count

81_job_settings.sql:
- New table job_settings (job_type PK, enabled, queue_name, cron_schedule,
  rate_ms, max_retries, max_concurrency, extra_config JSONB)
- Seeded 4 rows: scrape_kn (queue=scrape_kn, '15 4 * * mon'), nspd_geo
  (queue=geo, rate_ms=600), objective_etl, objective_sync (cron from prod
  objective_sync_config)
- objective_sync_config marked DEPRECATED via COMMENT — kept for backward
  compat until backend refactor

Both already applied on prod via postgres MCP.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 15:30:44 +03:00
lekss361
3919a40c49 refactor(nspd-geo): unify on rosreestr2coord v5, drop legacy column
Объединяю несколько связанных изменений вокруг NSPD geo bulk-fetcher:

Adapt to rosreestr2coord v5 API:
- nspd_lite.fetch_via_rosreestr2coord: drop `delay` kwarg from Area()
  (removed upstream in v5); keep it in our function signature for
  backward-compat, comment why.
- nspd_geo worker: add explicit time.sleep(rate_ms/1000) after lib-branch
  fetch — in v4 the library throttled internally via delay, in v5
  rate limiting is the caller's job. Без этого получали Area.__init__()
  unexpected kwarg `delay` на каждом target.

Drop use_rosreestr2coord switch:
- Removed urllib-vs-lib choice everywhere. We always use community
  rosreestr2coord library — авторы регулярно обновляют WAF-tricks,
  наш urllib-fetcher (fetch_geoportal) уже неактуален.
- admin_scrape.py: Pydantic schema, INSERT, SELECT, API response
  cleaned of `use_rosreestr2coord`.
- nspd_geo.enqueue_geo_job: param dropped, INSERT shrunk.
- worker process loop: dropped `if use_lib:` branch + import of
  fetch_geoportal.
- frontend/geo/page.tsx: removed checkbox + GeoJob.use_rosreestr2coord
  field + POST body field.

DB column drop:
- data/sql/78_drop_use_rosreestr2coord.sql (NEW):
  DROP COLUMN nspd_geo_jobs.use_rosreestr2coord + CREATE OR REPLACE
  VIEW v_scrape_runs_unified (которая depended on the column).
- data/sql/77_nspd_geo_jobs.sql: cleaned historical DDL for fresh setups.
- Migration applied to prod (in-conversation via postgres MCP).

Frontend polish:
- Thematic ID changed from free-form number input to labeled select
  (1=parcel / 2=quarter / 4=admin / 5=building / 7=zone / 15=complex).
- Auto-sync thematic_id from Job kind on change (override possible).
- ScrapeLogsPanel: extended union type with "nspd_geo" + fixed
  /admin/scrape/geo to pass scraperType="nspd_geo" (was "nspd",
  filtering empty legacy nspd_scrape_log table; real logs live in
  nspd_geo_log via v_scrape_log_unified).

Verified: ruff ✓, tsc --noEmit ✓, migration ran (BEGIN..COMMIT clean).

Deploy order safe: prod column уже удалена → новый backend код, который
не INSERT'ит use_rosreestr2coord, совпадёт со схемой после deploy.
2026-05-11 14:13:33 +03:00
lekss361
79e2e94e09 refactor(nspd): remove Playwright-based scraper, rosreestr2coord = default
УДАЛЕНО (~1700 строк):
* backend/app/services/scrapers/nspd_kn.py (Playwright + WAF-bypass)
* backend/app/workers/tasks/scrape_nspd.py
* frontend/src/app/admin/scrape/nspd/page.tsx
* 5 NSPD admin endpoints (POST /nspd, /nspd/release-lock,
  GET /nspd/runs, /logs, /coverage)
* NSPD beat schedule + worker_ready resume hook + nav tab + link
* _nspd_default_regions() helper

СОХРАНЕНО (история):
* nspd_scrape_runs / nspd_scrape_log таблицы (3 row + log) для аудита
* UNION ALL в v_scrape_runs_unified / v_scrape_log_unified —
  старые runs видны под фильтром scraper_type='nspd'
* settings scrape_nspd_* помечены DEPRECATED

CHANGED:
* nspd_geo_jobs.use_rosreestr2coord DEFAULT FALSE → TRUE
* UI checkbox "rosreestr2coord lib" по умолчанию = checked
* /admin/scrape/all + /admin/scrape/geo — единственный путь к NSPD-данным

NSPD-доступ теперь только через rosreestr2coord (community lib) —
upstream поддерживает обновления headers/WAF-tricks. Локальный
nspd_lite.py (urllib) остаётся как fallback (use_rosreestr2coord=FALSE).
2026-05-11 09:21:12 +03:00
lekss361
54dbc77348 feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI
ОТКРЫТИЕ: stdlib urllib проходит WAF nspd.gov.ru (TLS-fingerprint stdlib
отличается от mainstream HTTP-clients). Это позволяет полностью убрать
Playwright + Chromium для NSPD-задач.

* nspd_lite.py: urllib + ssl._create_unverified_context(). 4 публичных
  fetcher'а (geoportal/quarter/parcel/building) + fetch_via_rosreestr2coord
  fallback на community-lib

* schema 77: nspd_geo_jobs (журнал) + nspd_geo_targets (cad-номера со
  статусом pending/done/failed). Resume-state в БД.

* tasks/nspd_geo.py: Celery task process_nspd_geo_job(job_id). Heartbeat
  каждые 5 items, WAF backoff 30s × 2^N (max 8 → paused). UPSERT в
  cad_quarters_geom / cad_buildings → идемпотентно.

* worker_ready hook: stale jobs (>10min без heartbeat) автоматически
  re-enqueue после redeploy. Никаких потерь прогресса.

* 4 admin endpoint + UI /admin/scrape/geo с формой запуска (manual_list /
  rosreestr_pending для авто-наполнения из ДДУ-кварталов), progress-bars
  и cancel/resume кнопками per job.

* settings: use_nspd_lite=True, nspd_lite_rate_ms=600.
* dep: rosreestr2coord>=4.0.0 (community lib для fallback).

TODO следующих шагов:
- smoke-test на проде через SSH (validation RU-IP не банит urllib)
- feature toggle USE_NSPD_LITE в scrape_nspd.py для переключения с старого
  Playwright-пути
- docker lean image (-300 MB после убирания Chromium)
- расширение SCRAPE_NSPD_DEFAULT_REGIONS на 66,74,72,59 (Челябинск/Тюмень/Пермь)
2026-05-11 08:53:28 +03:00
lekss361
52bcf9d30c feat(db): canonical complexes + unified views + naming consistency
* Task 2+3: complexes.obj_class и district_name из Objective
  (350 ЖК с классом, 1525 с district; objective приоритетнее)

* Task 1 (fuzzy match orphan Objective ↔ domrf_kn):
  - pg_trgm extension
  - auto-merge 14 high-confidence (similarity ≥ 0.85)
  - v_complex_match_review для остальных 82 кандидатов
  - complexes 1752 → 1740 · cross-source 114 → 124

* Task 4 (naming unification без RENAME COLUMN):
  - GENERATED ALWAYS AS (...) STORED для синонимов
  - region_id в 15 таблиц + 9 partitions
  - developer_id в 7, developer_name в 5
  - COMMENT 'DEPRECATED' на region_cd/region_code/dev_id/dev_name

* Task 5 (unified scrape dashboard):
  - GET /api/v1/admin/scrape/all/runs + /all/logs (поверх v_scrape_runs_unified)
  - new page /admin/scrape/all с filter табами + 4 stat-плитки
  - +tab «📊 Все скраперы» в admin layout

Файлы: data/sql/75_unification_naming.sql · backend/app/api/v1/admin_scrape.py
       · frontend/src/app/admin/scrape/all/page.tsx · admin/layout.tsx
2026-05-10 21:11:20 +03:00
lekss361
7c05d0a0d8 feat(objective): full sync pipeline + dynamic admin config
Objective API (api.objctv.ru) интегрирован как новый source of truth для
per-flat данных по новостройкам УрФО. Заменяет промежуточный Anton-SQLite
(legacy bootstrap-ETL остался как fallback в свёрнутом блоке UI).

Schema (data/sql/68_v2 — applied на проде, 6 таблиц):
- objective_lots          (UPSERT по lot_id; 303 677 rows)
- objective_corpus_room_month (long-формат месяц×корпус×room_bucket; 19 738 rows)
- objective_lots_history  (append-only weekly snapshots для elasticity)
- objective_complex_mapping (Objective ComplexName ↔ domrf_kn_objects.obj_id)
- objective_raw_reports   (jsonb страховка на смену схемы API)
- objective_scrape_runs   (журнал прогонов)
+ data/sql/72_objective_sync_config (single-row динамический конфиг)

Backend:
- services/scrapers/objective.py: ObjectiveClient — Bearer-токен (Redis +
  in-memory fallback), retry на 401/429/5xx, Retry-After header support
- services/objective_etl.py: ETL SQLite Антона → PG (legacy)
- services/objective_sync_config.py: read/update single-row config
- workers/tasks/scrape_objective.py:
  * sync_objective_group: 2 рабочих отчёта (corp_sum, lots_pf), inline-парсинг
  * sync_all_groups: wrapper, перебирает группы из БД-конфига с
    inter-group паузой; PATCH-merge explicit args > DB config
- workers/tasks/objective_etl.py: Celery task для legacy bootstrap
- workers/celery_app.py: beat читает cron из БД при старте (fallback на env)
- api/v1/admin_scrape.py: 5 новых endpoints для /objective/*

Frontend (frontend/src/app/admin/scrape/objective/page.tsx):
- PRIMARY blue блок «🌐 Наш sync» с input для override групп
- Collapsible «⚙️ Настройки» с формой (cron + 8 параметров) → PUT в БД
- Coverage-панель с PG counts + строка про SQLite Антона как legacy
- Collapsible «🛠 Bootstrap ETL» — legacy-инструмент

Beat schedule: вторник 06:00 МСК, ~10-15 мин на 4 группы (Свердл.обл +
Челябинск + Тюмень + Пермь = ~700K квартир УрФО). Расписание и параметры
меняются через админку без редеплоя (cron требует restart beat).

Эмпирические находки об API (probe 2026-05-10):
- 13 из 21 проверенных group_name доступны на тарифе (включая «Свердловская
  область», «Челябинск», «Тюмень», «Пермь», но НЕ «Свердловская обл» —
  формат имени строгий)
- ComplexName требует БЕЗ префикса «ЖК» и БЕЗ кавычек
- Поле «Банк» для всех 303k = NULL (тариф не отдаёт), но «Тип обременения»
  работает (36% строк = ипотека) → ipoteka_share возможен, банковская
  атрибуция — нет

Docker:
- bind-mount /opt/gendesign/site-finder:/data/anton-sqlite:ro в worker

GitHub backlog: добавлены #22-25 (recommend_mix v3 — 4 сабтаска по
улучшению алгоритма на основе фидбэка про POI / границы районов /
конкурентов / окно данных / success-driven mix).

Knowledge graph (memory/memory-gendesign.jsonl): обновлены entities
Objective_Integration_May07_2026, Schema_Objective_v2_May07,
Objective_API_Findings_May07, Module_Objective_Client + новый
Session_End_May07_2026.

TODO для прода: прописать OBJECTIVE_API_KEY=<key> в backend/.env +
docker compose restart worker beat.
2026-05-10 19:54:15 +03:00
lekss361
12732949b9 ignore 2026-05-07 21:04:38 +03:00
lekss361
aaa5d44e77 feat(objective): integrate api.objctv.ru — auth, schema, weekly sync
backend/app/services/scrapers/objective.py:
  ObjectiveClient — GetToken (Bearer cached в Redis 25 min TTL) +
  GetReport v2 с UseDdu/UseDkp. 4 высокоуровневых метода:
  report_corpuses_summary, report_lots_summary, report_corpuses_per_flat,
  report_lots_per_flat. Retry 401/429/5xx, rate-limit 500ms, brotli.
backend/app/workers/tasks/scrape_objective.py:
  Celery task sync_objective_group — еженедельно тянет 4 канон. отчёта по
  группе, raw payload в objective_raw_reports.
celery_app.py: +include + beat «0 5 * * mon».
data/sql/68_schema_objective.sql: 6 таблиц — runs, raw_reports (jsonb),
oks (готовые ТЭП + escrow/debt), lots, lots_history (per-flat per-day),
complex_mapping (Objective ↔ domrf_kn_objects, is_reviewed workflow).
data/sql/69_objective_smoke.py: stand-alone GetToken + 4 отчёта →
data/raw/objective_smoke/<ts>/. Используется один раз чтобы понять
реальную схему payload перед написанием parser-слоя.
config: OBJECTIVE_API_KEY, OBJECTIVE_DEFAULT_GROUP=Екатеринбург,
OBJECTIVE_SYNC_CRON='0 5 * * mon'.
2026-05-07 21:04:00 +03:00
lekss361
f80a2c1d05 fix(scrape): SQLAlchemy text() парсил ':00:' regex как named-param
В nspd_kn.py:get_pending_cads и admin_scrape.py:nspd_coverage regex-фильтры
quarter_cad_number !~ '^00:00:' и !~ ':0000000$' падали с psycopg.errors.
DataError — двоеточие в text() трактуется как ':param'. Заменил на NOT LIKE
с bindparams (быстрее regex и без issue с парсингом).

Эффект на проде:
- /api/v1/admin/scrape/nspd/coverage возвращал 500 → теперь 200
- Кнопка «Запустить sweep» больше не дизейбленa (pending=964 теперь видно)
- Сам scraper тоже бы валился на этом же баге при beat-запуске
2026-04-30 23:16:11 +03:00
lekss361
e22f86bc06 audit-fix: 27mo окно, MODE bucket, индексы recommend_mix
B1: дефолт months_window 12→24, расширение fallback 24→27 (max архив).
2024Q1+Q2 теперь реально влияют на recommend_mix (raw 784→3815 сделок
для Академического, ×4.9).

B2: per-bucket elasticity через MODE по семантике (is_studio+rooms),
а не PERCENTILE_CONT(0.5) по total_area. ЖК с mix studios+3к больше не
попадают в bucket "1-к" как среднее. Для Академического: 1-к получает
208 точек регрессии (vs 118), slope -1.17.

B3: share_window_months и elasticity_window_months в API scope —
прозрачность для UI tooltip про рассинхрон окон (24mo rosreestr vs
36mo sale_graph).

B4: cad_buildings.floors при INSERT больше НЕ приводится к INT —
NSPD отдаёт диапазоны "1-2", "2-3" которые теряются как NULL. Хранится
как TEXT, аналитика делает CAST через regex.

R1: data/sql/66_indexes_recommend.sql — 4 индекса для горячего пути
recommend_mix (sale_graph composite, kn_objects partial, kn_flats partial,
cad_buildings partial). Применены в проде через MCP.

R3: ON CONFLICT в insert_buildings теперь обновляет ВСЕ изменяемые
поля (purpose, address, floors, geom, registration_date) — раньше при
re-scrape апдейтились только cost/area/year, остальное игнорилось.
2026-04-30 22:41:43 +03:00
lekss361
db6711446a fix 2026-04-30 22:16:44 +03:00
lekss361
25b73035a1 sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases
- NSPD-skraper переехал в backend/app/services/scrapers/nspd_kn.py +
  Celery task scrape_nspd_region (beat: 20-е февраля/мая/авг/нояб).
  Redis lock 3h, WAF auto-retry, heartbeat в nspd_scrape_runs.
- Recommend_mix Tier 3: per-bucket elasticity через регрессию по
  «доминирующему bucket» каждого ЖК. Weighted-elasticity для inverse-mode.
  UI показывает разброс эластичностей и переключение regression/fallback.
- Cadastre vs market cross-check: spatial-join cad_buildings →
  ekb_districts_geom; cadastre_vs_market_pct в scope, аномалии
  (>+50% / <-30%) подсвечены в UI.
- Sentry release tracking (#4): IMAGE_TAG → backend/.env.runtime →
  sentry_sdk.init(release=...). Compose v2 env_file optional path.

Schemas: 63_schema_nspd_runs.sql (cad_buildings + nspd_scrape_runs/log
формализуют то, что уже жило в проде через 61_import_nspd_batch.py),
64_v_zk_rosreestr_velocity.sql (refresh с cad_buildings).
2026-04-30 21:51:19 +03:00
lekss361
17ad0e18f5 feat(geo): ЖК → district mapping через PostGIS-полигоны OSM (Limit_District_Geometry closed)
- 56_fetch_ekb_districts.py: Overpass query (admin_level=9 в bbox ЕКБ),
  shapely-ассемблинг рваных OSM-веев в MultiPolygon, кэш в data/raw/.
- 56_schema_ekb_districts_geom.sql: CREATE TABLE с GIST.
- 57_kn_objects_district.sql: ALTER + backfill UPDATE через ST_Contains.
  3032 ЖК Свердл проставили district_name (208 в Академическом, 158 в Чкаловском,
  и т.д.), 454 за bbox ЕКБ остались NULL.
- domrf_kn.py: _backfill_district_names() после Phase A — каждый sweep
  автоматически проставляет district_name свежезаписанным ЖК.
- analytics_queries.py: 4 места addr ILIKE → district_name = :dn
  (_velocity_baseline, _elasticity_coef, _active_competitors_count, comparable).
- analytics_queries.py: graceful obj_class-degrade — если в районе все
  obj_class=NULL (kn-API quirk), фильтр по target_class опускается на уровне
  geo-запросов с warning. class_multiplier на ценах продолжает работать.

Smoke (Академический, Comfort, 50K м²): velocity_source=sale_graph (104 ЖК /
1872 точек), competitors=208 в районе, headline «3400 млн ₽ · 112 мес ·
ликвидность 21/100», 5 comparable ЖК района — все реалистичные.
2026-04-29 08:03:38 +03:00
lekss361
8dda3903e5 add log scrapper 2026-04-27 21:50:44 +03:00
lekss361
12b1eb8169 add logs 2026-04-27 20:51:44 +03:00
lekss361
b3c15bb8ea fix ui lag 2026-04-27 20:26:55 +03:00
lekss361
522ecca5c4 ship #13-16: multi-target Dockerfile, per-object drill-in, CRM pipeline,
442-sweep instructions
#16 Dockerfile: runner (lean, 1.6 GB) + runner-with-chromium (2.9 GB).
   Compose + deploy.yml pushes 2 GHCR images. Backend saves 1.3 GB.
#15 5 backend object endpoints + /analytics/objects/[id] page (gallery,
   POI table, sale chart) + sparkline-driven PrinzipObjectsTable.
#14 prinzip_leads/deals + 2 MVs. Import script with PII-hashing &
   --inspect-only mode. 3 funnel endpoints + Sankey/Monthly charts.
#13 README section with UI/CLI/SQL for 442-object sweep.
2026-04-27 19:32:37 +03:00
lekss361
6d3fa8cbd2 add domrf extras parser (sale_graph + sales_agg + infra + photos) +
fix Docker build for playwright

- Regenerate uv.lock with playwright + tenacity (was missing → uv sync
  --frozen skipped them, hence playwright not in /app/.venv/bin/).
- Dockerfile: explicit Chromium runtime libs in apt (libnss3 + 14 more)
  + `playwright install chromium` BEFORE apt-lists cleanup.
- Schema 51_schema_kn_extras.sql: 5 new tables for time-series sales,
  current aggregates, POI infrastructure, photo metadata, failure log.
- Scraper: 4 new fetch helpers + upserts; photo binary download via
  Playwright APIRequest; failure-logging with full_url to kn_scrape_failures.
- CLI: --no-extras / --download-photos / --no-flats / --probe URL.
- Admin UI: extras checkboxes + failures table with copy-URL button.
- PRINZIP smoke: 28 objs, 5409 POI, 5150 photos metadata, sales_agg
  matches probe (Парк Победы 145/291 = 49%).
2026-04-27 18:35:46 +03:00
lekss361
79070bfe99 add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright
- Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object
  (offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table.
  Server-side filter ignored — local filter by developer.companyGroup.
- Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp).
- Saved fingerprint state in data/playwright_state.json for server reuse.
- Celery beat (configurable cron + jitter) + admin trigger endpoint
  + /admin/scrape UI page.
- Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots,
  kn_scrape_runs journal.
- Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.
2026-04-27 17:58:40 +03:00
lekss361
16481868a6 add anal 2 2026-04-27 13:05:36 +03:00
lekss361
c13cbaef2e add data 2026-04-26 22:30:52 +03:00