KILLER feature от Макса — точки подключения к инженерным сетям + охранные
зоны как proxy для подбора инвест-участков.
Phase 1 — публичные NSPD данные (cat 36328 + 37578).
- ConnectionPointsResponse pydantic + 3 nested models
- get_connection_points() + helpers (parcel WKT, structures by boundary,
zouit overlaps) с ST_Distance(::geography) для метров
- GET endpoint с Query(radius_m, ge=50, le=2000, default=500)
- 7 tests
Closes part of #115 (Phase 1 backend).
Three distinct SQL syntax errors caught from worker logs:
1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326
in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'.
Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc).
Likely introduced by ruff-format auto-comma on closing paren.
2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz`
psycopg double-colon trap → 'syntax error at or near ":"'.
Crashed legacy on-demand harvest_quarter called from analyze.
3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb`
same double-colon trap in admin settings UPDATE.
All converted to CAST(:x AS type) pattern (psycopg v3 safe).
40 tests still passing.
Co-authored-by: lekss361 <claudestars@proton.me>
cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD
occasionally returns Point geometry for parcels without detailed
boundary, causing INSERT failure:
psycopg.errors.InvalidParameterValue:
Geometry type (Point) does not match column type (Polygon)
This killed pilot v8 at quarter 66:41:0104002 — worker autoretry
exhausted, job hung at 25/50 (heartbeat stale).
Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit
(same pattern as upsert_quarter_geom_from_feature). Non-Polygon
geometry → geom=NULL, raw_props preserved.
Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions,
cad_enk, cad_oncs) unaffected — they accept any geometry type.
Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit).
Co-authored-by: lekss361 <claudestars@proton.me>
cadastre_jobs.phase_state is shared between ALL parallel Celery workers
of the same job. The early-exit triggered as soon as the FIRST worker
wrote phase=done via progress_cb — all subsequent workers (reading same
shared row) bailed without work.
Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter).
Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers
early-exited without doing snapshot phase.
Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped
because most workers never reached Phase 1/1.5.
Solution: delete the early-exit. Idempotency is guaranteed by
ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks
write same rows without duplicates.
Closes part of #168 (root-cause fix for race condition revealed by #182 metrics).
Co-authored-by: lekss361 <claudestars@proton.me>
scrape_cadastre.py previously incremented requests_count only by
grid_walk_requests, so snapshot-only quarters (and the entire ekb_full
job after PR #179 skips grid-walk on broken geom) reported 0 NSPD
requests — misleading for observability.
Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5
per-cat probes) and a total_requests property. Update Celery task to
write total_requests into cadastre_jobs.requests_count.
Tests added for the new property + Phase 1 snapshot_requests=1 assertion.
Co-authored-by: lekss361 <claudestars@proton.me>
72% of EKB quarters (1735/2408) have broken micro-precision geom in
cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS
(36381) feature but Phase 4 was only consuming stats.
Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms
3857 polygon to 4326 and writes only when existing geom is NULL or broken
(bbox width outside 100-10000m), verified via SQL-side sanity check.
Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON
(self-intersecting ring) rollbacks only geom UPDATE, preserves
upsert_quarter_stats.
Closes part of #168 (follow-up data quality fix for #179).
Co-authored-by: lekss361 <claudestars@proton.me>
Switch base search_by_quarter from thematicSearchId=1 to thematic=1
(includes ZOUIT in features). Add optional category_id param that routes
ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5,
ENK (39663) -> thematicSearchId=15.
Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS
category present in meta_counts (meta_total>0), run dedicated
per-category search call. Skips if category in GRID_WALK_CATS and
bbox is valid (grid-walk gives better coverage).
Curl-probe confirmed NSPD API caps at 20/category with no pagination;
categoryId param is server-ignored; thematic=1 strictly better than
thematicSearchId=1 (includes ZOUIT).
Closes part of #168 (ZOUIT/ENK coverage gap).
Co-authored-by: lekss361 <claudestars@proton.me>
SQLAlchemy text() parser confuses ':window_interval' (named param) followed
by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb.
Result: psycopg видит literal ':window_interval' в SQL → syntax error →
exception caught by velocity catch → return None → UI shows null.
## Fix
':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест:
sales_rows query + _get_ekb_median percentile).
Pre-push code-reviewer должен был catch — добавим в feedback что для каждого
text() SQL grep ':[a-z]*::' before push.
Refs: PR #158 deploy verify, e2e velocity = null root cause
Co-authored-by: lekss361 <claudestars@proton.me>
* feat(site-finder): auto-fetch cadastre geometry on-demand (#93)
Когда пользователь вводит cad-номер которого нет в БД (cad_quarters_geom /
cad_buildings / cad_parcels_geom), вместо 404 «Загрузи через NSPD geo» (dead-
end для non-admin) — теперь backend автоматически инициирует NSPD fetch,
frontend показывает loading state с polling /fetch-status каждые 2с.
Backend:
- Новый модуль app/services/site_finder/cadastre_fetch.py с helpers:
find_or_enqueue_fetch (atomic check + enqueue с дедупликацией по cad),
fetch_status (smart polling endpoint — отличает not_in_nspd от failed),
detect_thematic_id (3-сегм quarter / 4-сегм parcel / 5-сегм building),
validate_cad_format.
- Reuse: enqueue_geo_job + process_nspd_geo_job (workers/tasks/nspd_geo).
source_kind='auto_on_demand' отличает от bulk; rate_ms=200, priority=9.
- POST /parcels/{cad}/analyze graceful fallback: inline await до 25s
(fast path), затем 202 + job_id + eta_seconds для polling, либо
400/404/503 в зависимости от статуса (с Retry-After 60s на 503).
- GET /parcels/{cad}/fetch-status новый endpoint для polling.
Frontend:
- useSiteAnalysis расширен: fetchingState + cancel(). POST analyze + при 202
polling каждые 2с (max 60 итераций = 2 мин cap). status=ready → re-trigger
analyze; not_in_nspd/failed/invalid_format → typed errors.
- apiFetchWithStatus<T> + HTTPError в lib/api.ts — status-aware variant
для 200 vs 202.
- FetchingState.tsx: animated spinner, progress bar (linear до etaSeconds,
asymptote после), elapsed counter, cancel button. Светло-голубая
scheme отличается от обычного pending skeleton.
- site-finder/page.tsx: FetchingState когда fetchingState активен; обычный
pending skeleton — только при первичном analyze без 202.
Edge cases (per #93 acceptance):
- cad валидный в НСПД, fetch <25s → inline 200 OK
- cad валидный, fetch >25s → 202 + frontend polls → ready → analyze
- cad валидный, не в НСПД → 404 с понятным сообщением + формат hint
- cad invalid format → 400 + формат hint
- NSPD rate-limited / failed → 503 + Retry-After 60s
- Параллельные запросы на тот же cad → один job, оба клиента poll'ят (дедуп
через find_active_on_demand_job).
Closes#93.
* fix(site-finder): address PR #95 auto-review minor feedback
Backend (cadastre_fetch.py):
1. (race condition) Advisory lock `pg_try_advisory_xact_lock(hashtext(cad_num))`
обёрнут вокруг шагов "check active job → enqueue" в find_or_enqueue_fetch.
Lock transaction-scoped, released at COMMIT. Параллельные запросы на тот
же cad: первый получает lock и enqueue; второй lock=false → re-check
active job (уже виден после первого COMMIT) → возвращает тот же job_id.
Docstring обновлён, упоминание SELECT FOR UPDATE удалено.
Backend (parcels.py):
3. (threadpool exhaustion) _INLINE_FETCH_WAIT_S снижен 25 → 15s с подробным
комментарием: tradeoff про default Starlette anyio threadpool (40 slots)
и concurrent burst saturation. 15s баланс: НСПД avg 5-15s для quarter,
~70% fast path; остальные 30% получают 202 + polling без блока.
Data (87_on_demand_indexes.sql):
2. (missing index) New migration:
- `nspd_geo_targets_cad_num_idx ON nspd_geo_targets(cad_num)` — для
find_active/recent_completed_job (existing UNIQUE composite не покрывает
WHERE cad_num=:c).
- `nspd_geo_jobs_source_status_idx ON nspd_geo_jobs(source_kind, status)`
composite для filter auto_on_demand + queued/running.
Idempotent (CREATE INDEX IF NOT EXISTS), не блокер при текущем размере,
критично при росте on-demand traffic.
Frontend (useSiteAnalysis.ts):
4. (UI flicker) setFetchingState(null) перемещён ПОСЛЕ `await second
apiFetch`. Иначе между clear и resolve есть момент когда mutation
isPending=true + fetchingState=null → пустой экран ~1 RTT.
NOT addressed (rebuttal):
5. (Tailwind convention) — проверил: в проекте нет ни globals.css, ни
@tailwind directives. ВСЕ существующие site-finder components используют
inline styles (ConfidenceBadge, GeotechRiskBlock, ScoreBreakdownPanel etc).
Tailwind в deps но не wired up. Keep inline styles для consistency.
6. (animate-spin) — требует Tailwind globals (см. #5). `<style jsx>`
keyframes — built-in Next.js, работает.
Per auto-review on 2252236.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
- nspd_geo: add _save_parcel() for thematic_id=1 → cad_parcels_geom (UPSERT,
ST_Transform from Web Mercator); _persist_target now handles 1/2/5
- parcels.py: analyze endpoint geom lookup extended with cad_parcels_geom as
3rd fallback source (after cad_quarters_geom, cad_buildings); both SELECT
and WKT subqueries updated
- parcels.py: POI score_breakdown items now include lat/lon for map markers
- poi_loader: OSM_CATEGORIES expanded — college+university→school,
hypermarket→shop_supermarket; coverage +3 tag pairs
The comparables block on /analytics/recommend was showing the same ЖК
multiple times (ЖК СТАРТ × 3, ЭХО ЛЕСА × 2 etc) because domrf_kn_objects
has ~3 historical snapshots per obj_id. The previous query joined all
rows and LIMIT 5 cherry-picked duplicates by flat_count.
Fix: wrap base in CTE `latest_obj` with `DISTINCT ON (obj_id) ORDER BY
obj_id, snapshot_date DESC` to pick only the latest snapshot per object
before sorting by flat_count.
- POST /api/v1/admin/scrape/geo/bulk — splits pending Sverdlovsk cad-nums
into N chunks (parallelism 1..10, default 5), creates N jobs and enqueues
each in queue=geo. source_kind='rosreestr_pending_chunk' for tracking.
- analytics_queries.complex_buildings(db, obj_id) — returns list of buildings
from cad_buildings (cad_num, floors, area, purpose, name, address, geom).
- object_detail: LEFT JOIN v_complex_buildings, adds buildings_count.
- top_developers: adds complexes_count via correlated subquery.
- GET /api/v1/analytics/object/{obj_id}/buildings → list[ComplexBuilding].
- JobSetting ORM model (JSONB extra_config) + Pydantic schemas
- GET/PUT /api/v1/admin/jobs/settings + /{job_type} (X-Admin-Token auth)
- celery_app.py builds beat_schedule from job_settings DB (env fallback
retained for safety on first deploy / DB unreachable)
- nspd_geo task reads rate_ms from job_settings when per-job row has
no override
- enqueue/resume geo jobs route to queue_name from job_settings
- Worker container: --queues=celery,scrape_kn,geo (one container,
three named queues — kn sweep no longer blocks nspd_geo)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three incremental bugs surfaced while wiring rosreestr2coord v5 into our
worker. All in backend/app/services/scrapers/nspd_lite.py.
1. `Area.__init__()` removed `delay` kwarg in v5 (was throttling). Drop
it from our call. Rate limit now happens in worker via
time.sleep(rate_ms/1000) after each fetch.
2. `Area(..., use_cache=True)` default writes media cache to ./tmp/
relative to CWD. In our docker image CWD=/app (owned by root), worker
runs as non-root → PermissionError on every target. Fix:
use_cache=False, media_path="/tmp/rosreestr2coord".
3. `Area.to_geojson_poly(dumps=True)` default returns JSON-serialized
STRING, not dict (changed in v5). Worker's _persist_target expects
dict with .get("properties") → AttributeError. Fix: dumps=False.
Vault entries:
- fixes/Bug_Rosreestr2coord_Delay_Arg_v5_Fixed.md
- fixes/Bug_Nspd_Geo_Tmp_Permission_Fixed.md
- fixes/Bug_Nspd_Geo_Str_Object_No_Get_Fixed.md
rosreestr2coord.Area() by default creates ./tmp/ media cache relative
to CWD. In our worker container CWD=/app (owned by root), runs as
non-root user `app` → PermissionError: '/app/tmp' on every target.
Fix: pass explicit kwargs to Area():
- use_cache=False (we don't need cache — each cad_num is unique)
- media_path='/tmp/rosreestr2coord' (world-writable in container)
Vault: fixes/Bug_Nspd_Geo_Tmp_Permission_Fixed.md
Объединяю несколько связанных изменений вокруг NSPD geo bulk-fetcher:
Adapt to rosreestr2coord v5 API:
- nspd_lite.fetch_via_rosreestr2coord: drop `delay` kwarg from Area()
(removed upstream in v5); keep it in our function signature for
backward-compat, comment why.
- nspd_geo worker: add explicit time.sleep(rate_ms/1000) after lib-branch
fetch — in v4 the library throttled internally via delay, in v5
rate limiting is the caller's job. Без этого получали Area.__init__()
unexpected kwarg `delay` на каждом target.
Drop use_rosreestr2coord switch:
- Removed urllib-vs-lib choice everywhere. We always use community
rosreestr2coord library — авторы регулярно обновляют WAF-tricks,
наш urllib-fetcher (fetch_geoportal) уже неактуален.
- admin_scrape.py: Pydantic schema, INSERT, SELECT, API response
cleaned of `use_rosreestr2coord`.
- nspd_geo.enqueue_geo_job: param dropped, INSERT shrunk.
- worker process loop: dropped `if use_lib:` branch + import of
fetch_geoportal.
- frontend/geo/page.tsx: removed checkbox + GeoJob.use_rosreestr2coord
field + POST body field.
DB column drop:
- data/sql/78_drop_use_rosreestr2coord.sql (NEW):
DROP COLUMN nspd_geo_jobs.use_rosreestr2coord + CREATE OR REPLACE
VIEW v_scrape_runs_unified (которая depended on the column).
- data/sql/77_nspd_geo_jobs.sql: cleaned historical DDL for fresh setups.
- Migration applied to prod (in-conversation via postgres MCP).
Frontend polish:
- Thematic ID changed from free-form number input to labeled select
(1=parcel / 2=quarter / 4=admin / 5=building / 7=zone / 15=complex).
- Auto-sync thematic_id from Job kind on change (override possible).
- ScrapeLogsPanel: extended union type with "nspd_geo" + fixed
/admin/scrape/geo to pass scraperType="nspd_geo" (was "nspd",
filtering empty legacy nspd_scrape_log table; real logs live in
nspd_geo_log via v_scrape_log_unified).
Verified: ruff ✓, tsc --noEmit ✓, migration ran (BEGIN..COMMIT clean).
Deploy order safe: prod column уже удалена → новый backend код, который
не INSERT'ит use_rosreestr2coord, совпадёт со схемой после deploy.
Worker крашился на старте с `ModuleNotFoundError: No module named 'psycopg2'`
при импорте app/workers/tasks/objective_etl, потому что app/services/objective_etl
писал на psycopg2 API, а в pyproject.toml есть только psycopg[binary]>=3.2.0.
Изменения:
- psycopg2 → psycopg (v3)
- psycopg2.extras.execute_values → _bulk_upsert helper, эмулирующий тот же
behavior через cur.executemany (в psycopg v3 это pipeline-mode, overhead
на сетевые round-trips минимален)
- psycopg.connect совместим с тем же connection string
Тесты: AST parse ✓, ruff ✓, import ✓
ОТКРЫТИЕ: stdlib urllib проходит WAF nspd.gov.ru (TLS-fingerprint stdlib
отличается от mainstream HTTP-clients). Это позволяет полностью убрать
Playwright + Chromium для NSPD-задач.
* nspd_lite.py: urllib + ssl._create_unverified_context(). 4 публичных
fetcher'а (geoportal/quarter/parcel/building) + fetch_via_rosreestr2coord
fallback на community-lib
* schema 77: nspd_geo_jobs (журнал) + nspd_geo_targets (cad-номера со
статусом pending/done/failed). Resume-state в БД.
* tasks/nspd_geo.py: Celery task process_nspd_geo_job(job_id). Heartbeat
каждые 5 items, WAF backoff 30s × 2^N (max 8 → paused). UPSERT в
cad_quarters_geom / cad_buildings → идемпотентно.
* worker_ready hook: stale jobs (>10min без heartbeat) автоматически
re-enqueue после redeploy. Никаких потерь прогресса.
* 4 admin endpoint + UI /admin/scrape/geo с формой запуска (manual_list /
rosreestr_pending для авто-наполнения из ДДУ-кварталов), progress-bars
и cancel/resume кнопками per job.
* settings: use_nspd_lite=True, nspd_lite_rate_ms=600.
* dep: rosreestr2coord>=4.0.0 (community lib для fallback).
TODO следующих шагов:
- smoke-test на проде через SSH (validation RU-IP не банит urllib)
- feature toggle USE_NSPD_LITE в scrape_nspd.py для переключения с старого
Playwright-пути
- docker lean image (-300 MB после убирания Chromium)
- расширение SCRAPE_NSPD_DEFAULT_REGIONS на 66,74,72,59 (Челябинск/Тюмень/Пермь)
Objective API (api.objctv.ru) интегрирован как новый source of truth для
per-flat данных по новостройкам УрФО. Заменяет промежуточный Anton-SQLite
(legacy bootstrap-ETL остался как fallback в свёрнутом блоке UI).
Schema (data/sql/68_v2 — applied на проде, 6 таблиц):
- objective_lots (UPSERT по lot_id; 303 677 rows)
- objective_corpus_room_month (long-формат месяц×корпус×room_bucket; 19 738 rows)
- objective_lots_history (append-only weekly snapshots для elasticity)
- objective_complex_mapping (Objective ComplexName ↔ domrf_kn_objects.obj_id)
- objective_raw_reports (jsonb страховка на смену схемы API)
- objective_scrape_runs (журнал прогонов)
+ data/sql/72_objective_sync_config (single-row динамический конфиг)
Backend:
- services/scrapers/objective.py: ObjectiveClient — Bearer-токен (Redis +
in-memory fallback), retry на 401/429/5xx, Retry-After header support
- services/objective_etl.py: ETL SQLite Антона → PG (legacy)
- services/objective_sync_config.py: read/update single-row config
- workers/tasks/scrape_objective.py:
* sync_objective_group: 2 рабочих отчёта (corp_sum, lots_pf), inline-парсинг
* sync_all_groups: wrapper, перебирает группы из БД-конфига с
inter-group паузой; PATCH-merge explicit args > DB config
- workers/tasks/objective_etl.py: Celery task для legacy bootstrap
- workers/celery_app.py: beat читает cron из БД при старте (fallback на env)
- api/v1/admin_scrape.py: 5 новых endpoints для /objective/*
Frontend (frontend/src/app/admin/scrape/objective/page.tsx):
- PRIMARY blue блок «🌐 Наш sync» с input для override групп
- Collapsible «⚙️ Настройки» с формой (cron + 8 параметров) → PUT в БД
- Coverage-панель с PG counts + строка про SQLite Антона как legacy
- Collapsible «🛠 Bootstrap ETL» — legacy-инструмент
Beat schedule: вторник 06:00 МСК, ~10-15 мин на 4 группы (Свердл.обл +
Челябинск + Тюмень + Пермь = ~700K квартир УрФО). Расписание и параметры
меняются через админку без редеплоя (cron требует restart beat).
Эмпирические находки об API (probe 2026-05-10):
- 13 из 21 проверенных group_name доступны на тарифе (включая «Свердловская
область», «Челябинск», «Тюмень», «Пермь», но НЕ «Свердловская обл» —
формат имени строгий)
- ComplexName требует БЕЗ префикса «ЖК» и БЕЗ кавычек
- Поле «Банк» для всех 303k = NULL (тариф не отдаёт), но «Тип обременения»
работает (36% строк = ипотека) → ipoteka_share возможен, банковская
атрибуция — нет
Docker:
- bind-mount /opt/gendesign/site-finder:/data/anton-sqlite:ro в worker
GitHub backlog: добавлены #22-25 (recommend_mix v3 — 4 сабтаска по
улучшению алгоритма на основе фидбэка про POI / границы районов /
конкурентов / окно данных / success-driven mix).
Knowledge graph (memory/memory-gendesign.jsonl): обновлены entities
Objective_Integration_May07_2026, Schema_Objective_v2_May07,
Objective_API_Findings_May07, Module_Objective_Client + новый
Session_End_May07_2026.
TODO для прода: прописать OBJECTIVE_API_KEY=<key> в backend/.env +
docker compose restart worker beat.
