Round 1 (commit bcd7dc8) был broken: на 2-bucket входах surplus уходил в free
полностью без учёта capacity → free превышал cap → следующая итерация
clamp'ировала его и наоборот. Infinite oscillation в FastAPI handler.
Round 2 fix per review BLOCK (#282 comment):
- Surplus распределяется пропорционально available capacity (cap - v),
не текущему v. Free никогда не вылетит выше cap.
- free = строго < cap (не <=) — иначе деление на 0 capacity.
- Hard guard `for _ in range(N+1)` — гарантированно завершается.
- Pathological (surplus > total_capacity): возвращаем оригинальный pct_map
+ cap_skipped=True (sum=100 invariant сохранён).
- Hamilton round вынесен в _hamilton_round() helper.
Tests:
- 2-bucket cases (90/10, 70/30, 99/1) expected cap_skipped=True
- test_cap_iteration_count_bounded — все pathological завершаются < 100ms
- All 13 cases verified standalone (3 fast-path + 7 reproduced + 3 pathological)
Раньше _VELOCITY_DIVISORS делил агрегаты mv_layout_velocity (24 мес)
на 4/12 для quarter/year, не меняя реальное окно данных. Теперь
inline SQL из objective_corpus_room_month с CAST(:window_interval AS interval).
velocity_per_month = deals_window / months_in_window (1.0/3.0/12.0).
Разные time_window → разные строки из БД → разный mix/velocity/jk_count.
Closes (epic part) #271 item 1
Backend (quarter_dump_lookup.py):
- _acquire_harvest_lock: Redis SETNX TTL=120s на quarter, защищает от burst
N concurrent analyze, ставящих N одинаковых harvest task в очередь
- _trigger_harvest: использует lock перед apply_async, возвращает False если
lock уже взят (другой запрос триггернул раньше)
- make_empty_result/EMPTY_DUMP_RESULT: новое поле harvest_eta_seconds в
nspd_dump dict, типичный harvest_quarter = 60с
- /analyze: пробрасывает поле через nspd_dump dict (нет typed schema —
response_model=None для /analyze endpoint, dict уходит как есть)
Frontend (NspdFreshnessBadge, NspdZoningBlock):
- Countdown «НСПД: загрузка ~Nс» вместо бесконечного спиннера
- После остановки countdown (remaining=0) NspdZoningBlock показывает
«загрузка дольше обычного» + ссылку на ПКК вместо infinite skeleton
Tests: 5 новых unit + 2 для empty_result schema (всего +7, pass)
Closes#234 (UX-side; data-side resolves когда Sub-PR B + D merged).
cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD
occasionally returns Point geometry for parcels without detailed
boundary, causing INSERT failure:
psycopg.errors.InvalidParameterValue:
Geometry type (Point) does not match column type (Polygon)
This killed pilot v8 at quarter 66:41:0104002 — worker autoretry
exhausted, job hung at 25/50 (heartbeat stale).
Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit
(same pattern as upsert_quarter_geom_from_feature). Non-Polygon
geometry → geom=NULL, raw_props preserved.
Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions,
cad_enk, cad_oncs) unaffected — they accept any geometry type.
Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit).
Co-authored-by: lekss361 <claudestars@proton.me>
cadastre_jobs.phase_state is shared between ALL parallel Celery workers
of the same job. The early-exit triggered as soon as the FIRST worker
wrote phase=done via progress_cb — all subsequent workers (reading same
shared row) bailed without work.
Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter).
Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers
early-exited without doing snapshot phase.
Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped
because most workers never reached Phase 1/1.5.
Solution: delete the early-exit. Idempotency is guaranteed by
ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks
write same rows without duplicates.
Closes part of #168 (root-cause fix for race condition revealed by #182 metrics).
Co-authored-by: lekss361 <claudestars@proton.me>
scrape_cadastre.py previously incremented requests_count only by
grid_walk_requests, so snapshot-only quarters (and the entire ekb_full
job after PR #179 skips grid-walk on broken geom) reported 0 NSPD
requests — misleading for observability.
Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5
per-cat probes) and a total_requests property. Update Celery task to
write total_requests into cadastre_jobs.requests_count.
Tests added for the new property + Phase 1 snapshot_requests=1 assertion.
Co-authored-by: lekss361 <claudestars@proton.me>
72% of EKB quarters (1735/2408) have broken micro-precision geom in
cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS
(36381) feature but Phase 4 was only consuming stats.
Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms
3857 polygon to 4326 and writes only when existing geom is NULL or broken
(bbox width outside 100-10000m), verified via SQL-side sanity check.
Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON
(self-intersecting ring) rollbacks only geom UPDATE, preserves
upsert_quarter_stats.
Closes part of #168 (follow-up data quality fix for #179).
Co-authored-by: lekss361 <claudestars@proton.me>
Switch base search_by_quarter from thematicSearchId=1 to thematic=1
(includes ZOUIT in features). Add optional category_id param that routes
ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5,
ENK (39663) -> thematicSearchId=15.
Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS
category present in meta_counts (meta_total>0), run dedicated
per-category search call. Skips if category in GRID_WALK_CATS and
bbox is valid (grid-walk gives better coverage).
Curl-probe confirmed NSPD API caps at 20/category with no pagination;
categoryId param is server-ignored; thematic=1 strictly better than
thematicSearchId=1 (includes ZOUIT).
Closes part of #168 (ZOUIT/ENK coverage gap).
Co-authored-by: lekss361 <claudestars@proton.me>
* fix(cadastre): NSPD response parse — properties.category + top-level meta (#168)
* fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168)
Bot review of #177 flagged that fixtures still used legacy properties.categoryId
+ data.meta — meaning CI exercised only fallback branches, not primary paths
where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes
(#175 verify, #176 headers, #177 parse) confirmed need for test coverage.
Changes:
- test_nspd_bulk_client.py: sample_quarter_response → properties.category +
meta moved to top-level (real NSPD shape verified live)
- test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace)
- test_nspd_bulk_feature_parse_basic: primary path now exercised
Plus schema hardening per bot review:
- NSPDBulkFeature.category_id: `is not None` check (not truthy `or`)
to avoid edge case category=0; int() wrapped in try/except so
non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop.
35/35 tests pass locally.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
