94 commits

Author	SHA1	Message	Date
lekss361	bb7fbd5bc0	feat(sf-b4): GET /landing/stats — 5 KPI + paradox for landing page hero (#331)	2026-05-17 21:47:10 +00:00
lekss361	ea558b241c	feat(22d): admin endpoint POST /admin/scrape/kn-catalog-objects (#338)	2026-05-17 21:41:53 +00:00
lekss361	76112870e2	fix(pilot): drop pydantic EmailStr — email-validator dep missing → backend startup crash (#337)	2026-05-17 21:32:23 +00:00
lekss361	c61c8c86af	feat(sf-b1): GET /parcels/by-bbox — map entry endpoint + parcel_user_status table (#336)	2026-05-17 21:25:48 +00:00
lekss361	dbeed80b26	Merge pull request 'feat(sf-b3): POST /pilot/request — lead gen + pilot_requests table (Telegram TODO)' (#330) from feat/sf-b3-pilot-leads into main	2026-05-17 21:14:28 +00:00
lekss361	b013fd886c	feat(sf-b6): GET /parcels/{cad}/poi-score — weighted top-7 (#333)	2026-05-17 21:11:25 +00:00
lekss361	493238447a	feat(sf-b5): extend /parcels/{cad}/analyze with 15 fields (EGRN/encumbrance/red_lines/risks/district prices) (#332)	2026-05-17 21:11:18 +00:00
lekss361	4040cba4c0	feat(sf-b7): GET /parcels/{cad}/snapshot.pdf — 1-page WeasyPrint export (#334)	2026-05-17 21:11:03 +00:00
lekss361	bf55cc9e56	merge: resolve main.py conflict (keep both pilot + users routers)	2026-05-18 00:08:48 +03:00
lekss361	8673290009	feat(sf-b2): GET /users/me/recent-parcels — stub endpoint (auth TODO) (#329)	2026-05-17 21:04:16 +00:00
lekss361	99e55c0bc2	feat(sf-b3): POST /pilot/request + pilot_requests table ... Добавляет endpoint для приёма заявок на пилот (lead-gen). INSERT в pilot_requests, response {id, created_at, status}. Telegram-уведомление — TODO (creds не настроены, #307 SF-B3). Migration: data/sql/118_pilot_requests.sql	2026-05-18 00:01:35 +03:00
lekss361	e872b1ebe4	fix(sf): exclude NULL competitors from aggregate calculations (velocity, market_avg, top_sellers) (#328)	2026-05-17 20:46:07 +00:00
lekss361	7a8d59beb4	feat(obj-3): SF backend migrate to objective_lots ground truth (price 0.3% -> 81%) (#324)	2026-05-17 20:08:17 +00:00
lekss361	3978aa5905	feat(trade-in): TI-2 PDF export 4-page + frontend enable button (#319)	2026-05-17 17:16:48 +00:00
lekss361	29dbbee2ac	feat(trade-in): TI-1 mock /trade-in/estimate endpoint + Pydantic + SQL migration (#316)	2026-05-17 16:45:06 +00:00
lekss361	f8c382f885	feat(22k): object page — full exposure 9 blocks (sales/quartirography/metro/photos/docs/specs/checks) (#310)	2026-05-17 15:56:39 +00:00
lekss361	e1a8ad9395	fix(sf-17): rosreestr velocity fallback via cad_quarter (~100% EKB coverage) ... When Objective mapping coverage falls below 50% of competitors in the radius, fall back to rosreestr_deals JOIN on the parcel's cadastral quarter. Audit shows 237/237 EKB quarters (100%) have rosreestr data for the last 12 months, compared to <20% Objective coverage before bulk mapping. - velocity.py: add _compute_rosreestr_fallback(), _OBJECTIVE_COVERAGE_MIN_RATIO constant, velocity_source field on VelocityResult (objective/rosreestr_fallback/none) - parcels.py: extract cad_quarter from cad_num, pass to compute_velocity - site-finder.ts: add velocity_source field to Velocity interface - VelocityBlock.tsx: badge "Источник: квартальные сделки" when rosreestr_fallback Epic #271 item #17	2026-05-17 16:50:19 +03:00
lekss361	9eed55cd30	Merge pull request 'fix(sf-20): adaptive district threshold 30→15 with data_confidence flag' (#288) from fix/sf-20-district-threshold-adaptive into main	2026-05-17 13:39:52 +00:00
lekss361	989d4e5b7d	fix(sf-20): adaptive district threshold 30→15 with data_confidence flag ... - v_bucket_success_score HAVING >= 30 → >= 15 so sparse districts (Кировский etc.) are no longer silently excluded from the block - Backend: SUCCESS_REC_MIN_DEALS=15, SUCCESS_REC_STRONG_DEALS=30; data_confidence='weak' (15-29 deals) or 'strong' (≥30) in response - TS type ParcelSuccessRecommendation: add data_confidence literal - SuccessRecommendationBlock: amber badge for weak, all hex → CSS tokens, emoji star removed (ui-ux.md), empty-state text min 30→15	2026-05-17 16:34:52 +03:00
lekss361	2694e3180b	feat(etl): fuzzy matcher v2 — pruned threshold 0.85→0.80 for objective mapping coverage ... - Add AUTO_ACCEPT_THRESHOLD_V2 = 0.80 constant to objective_backfill.py - Add min_threshold param to find_match_candidates() (default REVIEW_THRESHOLD=0.6) - Add match_method param to auto_apply_matches() (default 'fuzzy_trgm') - Add ?v2=true query param to POST /api/v1/admin/etl/objective-backfill - v2 mode: threshold=0.80, method='fuzzy_v2', search from 0.80 (not 0.60) - Return type widened to dict[str,object] to include threshold_used + match_method_used DB audit: 1068 unmapped EKB objs; v1 adds ~133 rows, v2 adds ~47 more. Expected coverage: 8.5% → ~20% after sequential v1+v2 runs on prod. Run: POST /objective-backfill (v1) then POST /objective-backfill?v2=true (v2). Part of task #44 Part A, epic #271.	2026-05-17 16:28:20 +03:00
lekss361	4d39ae5b5d	fix(sf-07): use median_12m from mv_quarter_price_per_m2 for district price ... ekb_districts.median_price_per_m2 was populated from a 24-month window. LEFT JOIN mv_quarter_price_per_m2 on the cad quarter and COALESCE(median_12m, fallback) so the API returns the 12-month median where available. Frontend OverviewTab adds a "(12 мес)" hint next to the value. Closes #271 item 7	2026-05-17 14:28:32 +03:00
lekss361	1799e1fb30	fix(sf-02): добавить site_status/ready_dt в /analyze competitors + sort строящиеся first ... SQL в /analyze не выбирал site_status, ready_dt из domrf_kn_objects; ORDER BY flat_count DESC ставил сданные ЖК в топ. Closes (epic part) #271 item 2	2026-05-17 13:29:59 +03:00
lekss361	485f489bd2	feat(#254): user_custom_pois — backend schema + CRUD + scoring integration ... - SQL migration 101_user_custom_pois.sql: table with GEOGRAPHY(POINT) + GIST index - Pydantic schemas: CustomPoiCreate / CustomPoiUpdate / CustomPoiOut - Service custom_pois.py: CRUD + get_overlaps_for_scoring (1km radius, psycopg v3 CAST) - API /api/v1/custom-pois: POST(201)/GET/PATCH/DELETE, X-Session-Id header auth - Scoring in analyze_parcel: custom POI block after OSM POI loop, decay weight * max(0, 1 - dist/1000), custom_poi_score_items in response - Tests: CRUD + validation (weight/lon/lat 422) + scoring integration (mock-based) - Vault: Schema_User_Custom_Pois.md + Endpoint_Parcel_Analyze v3.9 section	2026-05-17 09:34:33 +03:00
lekss361	7d0bc4bb6b	Merge pull request 'feat(#29,#232): wire cad_parcels.permitted_use в analyze + cad_zouit fallback (G2+G3)' (#240) from feat/g2-g3-analyze-parcel-meta-zouit-fallback into main	2026-05-17 05:26:02 +00:00
lekss361	f70be684da	feat(#29,#232): wire cad_parcels.permitted_use in analyze + cad_zouit fallback (G2+G3) ... TASK A (#29 G2): add parcel_meta to analyze response - New ParcelMeta Pydantic schema in app/schemas/parcel.py - SELECT from cad_parcels WHERE cad_num=:c in analyze_parcel() (step 9f) - Returns permitted_use_established_by_document, land_record_category_type, land_record_subtype, cost_value; None when row absent - Tests: test_analyze_parcel_meta.py (found + not-found cases) TASK B (#232 G3): cad_zouit fallback in _get_zouit_overlaps - When nspd_quarter_dumps has zouit_count==0, fall back to ST_Intersects query on cad_zouit (3483 rows, GIST indexed) - Overlaps tagged with source='cad_zouit'; format compatible with NSPD path - gate_verdict.py: BLOCKER_TYPE_ZONE_KEYWORDS tuple for keyword-based classification (охранная зона / трубопровод / электр / газ -> blocker; СЗЗ -> warning); NSPD subcategory path preserved backward-compat - Tests: 6 new test cases in test_gate_verdict.py covering cad_zouit path and backward-compat for NSPD subcategory path Updated db.execute call sequence in test_analyze_*.py (index shift +1 at pos 10).	2026-05-17 08:17:22 +03:00
lekss361	0135f3c7c4	feat(#104): add POST /admin/scrape/ekburg-permits for manual trigger ... Wraps refresh_all / refresh_year Celery tasks behind the existing AdminTokenAuth gate so the table can be populated on demand after first deploy instead of waiting for the monthly beat (1st of month). - TriggerEkburgPermitsRequest: year int|None, ge=2022 le=2030 - year=None -> refresh_all.apply_async() (scope all_years_2022_2026) - year=N -> refresh_year.apply_async(args=[N]) (scope year_N) - 4 smoke tests: all/year/invalid_year/no_token	2026-05-17 08:13:32 +03:00
lekss361	a6e4ff0407	feat(#114): seed 3 default weight presets + include_system API param ... Add SQL migration 100_user_weight_profiles_default_seed.sql with system presets Эконом/Комфорт/Бизнес (user_id='__system__'). Migration is idempotent via ON CONFLICT DO UPDATE. Backend: - weight_profiles.py: add SYSTEM_USER_ID constant + list_profiles_with_system() - admin_weight_profiles.py: add include_system query param to GET list endpoint Tests: 3 new tests covering include_system flag and service sentinel behaviour.	2026-05-16 22:49:56 +03:00
lekss361	db1ff5b1fc	feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) ... - data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%') - nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump - SAVEPOINT per row via with db.begin_nested() - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326) - json.dumps(geom, ensure_ascii=False) for psycopg bind - All CAST(:x AS type) — no ❌:type - nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal) - nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit) - admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth) - 19 tests: coerce helpers, denorm parcel/building, aggregate counts Foundation for downstream features (МКД neighbors lookup, parcel attrs). Part of #94	2026-05-16 20:20:03 +03:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	1595157ebf	feat(nspd): enable TIER 3 risk zones harvest + extract in analyze (#94 PR1) ... - harvest_stale_quarters fanout passes include_risks=True to all queued tasks - _get_risk_zones() with ST_Intersects + ST_Area intersection calc - _extract_features_by_layer() generic helper for future PR2+ reuse - RiskZone Pydantic schema; /analyze returns nspd_risk_zones list - NspdRiskZonesBlock component with severity color-coding (red/yellow/orange) - 11 unit tests for extract/intersect/null-area/db-exception paths Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)	2026-05-16 16:11:59 +03:00
lekss361	dbf31cc911	feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)	2026-05-16 12:47:09 +00:00
lekss361	46203ab365	feat(permits): Celery beat monthly + analyze recent_permits (#105 Phase 4+5) (#213)	2026-05-16 12:23:52 +00:00
lekss361	27a0957bb5	feat(analyze): add market_price из mv_quarter_price_per_m2 (#33 PR B) (#210)	2026-05-16 11:28:16 +00:00
lekss361	092976f656	feat(site-finder): inline POI weights pass-through в /analyze (#201 Phase 1) (#206)	2026-05-16 11:00:41 +00:00
lekss361	256909d28b	feat(layouts): PDF ТЗ endpoint + BestLayoutsBlock UI (#113 PR D) (#199)	2026-05-16 09:53:05 +00:00
lekss361	ad16fc0e42	feat(parcels): best-layouts endpoint + service (#113 PR C) (#196)	2026-05-16 09:10:06 +00:00
lekss361	4deb1b0284	feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)	2026-05-16 05:58:27 +00:00
lekss361	e561df1b55	feat(parcels): competitors endpoint (Forgejo #112 Phase 1 backend) (#191)	2026-05-15 22:21:05 +00:00
lekss361	608490fbba	fix(parcels): _parse_floors handle int (post-migration #169) (#176) ... * fix(parcels): _parse_floors handle int (post-migration #169 schema change) After PR #169 cad_buildings schema migration, `floors` column is INT (was TEXT in legacy schema). Existing `_parse_floors(r.get("floors"))` call in analyze_parcel → _neighbors_summary crashes with: AttributeError: 'int' object has no attribute 'strip' Fix: type union str | int | None. If int → return directly (no strip). Preserve TEXT range parsing ("5-7" → 7) for backwards-compat with any legacy data still in cad_buildings_old_apr26. * test(smoke): production smoke tests for post-deploy regressions (#168) Add tests/smoke/test_prod_smoke.py covering known regression surfaces. Marks: prod_smoke + slow. Env: PROD_SMOKE_BASE_URL, PROD_SMOKE_ADMIN_TOKEN. Run manually: cd backend && uv run pytest tests/smoke/ -m prod_smoke -v * fix(cadastre): exact-match headers to nspd_lite to bypass NSPD WAF (#168) Pilot v2 (job_id=2) failed 50/50 with HTTP 403 WAF block. Comparing nspd_bulk_client.DEFAULT_HEADERS vs legacy nspd_lite.HEADERS (which works on VPS IP since April 2026): PascalCase keys → lowercase keys Chrome/148 UA → Chrome/144 UA No cache-control / pragma → "no-cache" both accept-language ru first → en first No origin → "https://nspd.gov.ru" referer "/map" → "/map?thematic=PKK" NSPD WAF (BotShield-class) likely fingerprints на header order + values combined with TLS fingerprint. Matching legacy exactly minimizes deltas. Test plan: retry pilot job after deploy, expect 0 WAF blocks for first 5 quarters. * fix(test): exclude prod_smoke tests by default (#168) CI ran tests/smoke/test_prod_smoke.py and they hit production https://gendsgn.ru/api/v1/parcels/.../analyze which currently returns 500 (parse_floors regression — exactly what this PR fixes). Catch-22: PR can't merge because smoke tests fail against pre-merge prod. Fix: add `addopts = ["-m", "not prod_smoke"]` so default pytest excludes them. Run manually post-deploy with: pytest -m prod_smoke -v --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 15:08:16 +03:00
lekss361	5da66bbc93	feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) ... * feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5) Add bulk cadastre harvest pipeline: - services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases) + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats) using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert - services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent) + generate_grid_click_points (15x15 = 225 sub-bbox grid) - workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True, dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies - api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume - Tests: 13 service unit + 8 API tests * fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3) * fixup(cadastre): tile_size unified param + register slow marker (#168 PR3) Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points. Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had tile_width+tile_height → TypeError at runtime. Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1: TypeError at import chain (app.main → admin_cadastre → bulk_harvest) broke FastAPI app load → dependency_overrides AttributeError. Now resolved. Also register `slow` pytest marker in pyproject.toml to suppress PytestUnknownMarkWarning. * fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3) Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module top-level REBOUND `app` from FastAPI instance (line 14) to the Python package. All subsequent `app.dependency_overrides[get_db] = ...` failed with AttributeError because `app` was now the namespace module. Fix: probe module availability with importlib.util.find_spec — does not import or rebind names. FastAPI `app` instance stays intact. All 28 cadastre tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:31:32 +03:00
lekss361	30d78c5030	fix(pipeline): enrich obj_class via objective_lots COALESCE fallback (#165) ... domrf_kn_objects.obj_class всегда NULL (Bug_Kn_API_Obj_Class_Always_Null_OPEN). Pipeline 24mo UI показывал 'Не указан: 4075 квартир'. Fix: SQL COALESCE с correlated subquery через objective_complex_mapping → objective_lots.class. Mapped complexes (114/4548) теперь резолвят корректный класс — комфорт/стандарт/бизнес/премиум. ## Coverage impact (verified via MCP) Pre-fix: 0 of 1316 planned ЖК имели obj_class Post-fix: 339 of 1316 (25.8%) — 249 комфорт, 45 стандарт, 42 бизнес Не-mapped objects остаются 'Не указан' (upstream kn-scraper fix всё ещё open). Closes #162 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 10:07:52 +03:00
lekss361	c39b53cf53	fix(site-finder): cascade InFailedSqlTransaction — velocity internal rollback (#155) ... * fix(site-finder): db.rollback() в velocity catches + SAVEPOINT zoning/success_rec velocity.py 3 internal try/except возвращали None БЕЗ db.rollback(). PR #154 SAVEPOINT не помог — exception не propagates из velocity. Postgres tx aborted внутри, cascade InFailedSqlTransaction на следующих queries. Fix: - velocity.py: db.rollback() в 3 catch блоках - parcels.py: SAVEPOINT обёртки для zoning + success_recommendation Refs: user reports 2026-05-15 cascade 500 * fix(velocity): SAVEPOINT per query (not db.rollback) — correct SQLAlchemy 2.0 pattern Per PR #155 bot review — мой db.rollback() в catches был НЕПРАВИЛЬНЫЙ. Session.rollback() = ENTIRE outer tx (не savepoint). Outer SAVEPOINT context становится orphaned → InvalidRequestError на __exit__. Correct fix: wrap каждый db.execute в свой with db.begin_nested(): - Failure → __exit__ propagates exception → SAVEPOINT rolls back - Outer tx остаётся clean - velocity returns None gracefully → caller продолжает Все 3 SQL queries в velocity.py теперь SAVEPOINT-wrapped: - compute_velocity: competitor_query - compute_velocity: sales_rows query - _get_ekb_median: median query db.rollback() removed из всех 3 catches. Refs: PR #155 bot review --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:52:38 +03:00
lekss361	b13c4be0ed	fix(parcels): SAVEPOINT around velocity compute → не abort outer tx (#154) ... Production POST /parcels/{cad}/analyze падает с 500: sqlalchemy.exc.InternalError: (psycopg.errors.InFailedSqlTransaction) current transaction is aborted, commands ignored until end of transaction block ## Root cause `compute_velocity()` SQL fails (probably на cad's без conkurrentов / sparse sale_graph data) → exception caught в try/except → НО db.rollback() отсутствует → transaction остаётся в aborted state. Следующая query (_geotech_risk на line 828) пытается выполниться → крашится с InFailedSqlTransaction. ## Fix Wrap velocity в `with db.begin_nested()` — SAVEPOINT pattern (consistent with PR #124 pzz_loader fix). Failure внутри savepoint: - Rollbacks ТОЛЬКО savepoint - Outer transaction остаётся clean - Subsequent queries (_geotech_risk и пр.) работают Pattern matches feedback_subagent_delegation audit recommendation для in-loop / per-section exception handling. ## Impact POST /parcels/{cad}/analyze больше не 500 при velocity failure. Возвращает `velocity: null` + остальные fields normal. Refs: user report 2026-05-15 InFailedSqlTransaction Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:29:24 +03:00
lekss361	e93bb6121e	feat(site-finder): D2 velocity-score from domrf_kn_sale_graph (#34 sub-PR 1/2) (#146) ... compute_velocity service queries competitor sales в радиусе 3км: - ST_MakePoint(longitude, latitude) — domrf_kn_objects не имеет geom column - JOIN domrf_kn_sale_graph за 6 мес (area_sq primary, realised*45 fallback) - Normalize vs ЕКБ-wide median → velocity_score 0..1 - confidence: high/medium/low (competitors_count + months_observed) - Top 5 sample competitors для UI Integration: analyze_parcel.response['velocity'] top-level field. Schema corrections vs spec: - obj_name → comm_name - region_code → region_cd - contracted (INT) → area_sq (м²) Tests: 102/102 pass. Vault: Module_Velocity_Service.md NEW. Closes #144 (sub-PR 2 frontend закроет #34) Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 01:24:53 +03:00
lekss361	76a88b2827	feat(site-finder): gate verdict aggregator — can-build-MKD signal (#32 G5 sub-PR 1/2) (#142) ... Pure-function aggregator collapses nspd_zoning / nspd_zouit_overlaps / nspd_engineering_nearby / nspd_dump в один GateVerdict TypedDict. Logic: - ПЗЗ не Ж-* → BLOCKER - ЗОУИТ sub=17 (инжен. охранная) → BLOCKER - Другие ЗОУИТ → WARNING - Нет инжен. сетей в 200м → WARNING - nspd_dump stale → source: nspd_dump_partial - Нет dump → can_build_mkd: 'unknown' Integration: новое поле gate_verdict в analyze_parcel response. Tests: 15/15 pass (mock-based). Vault: code/modules/Module_Gate_Verdict.md NEW. Closes #140 (sub-PR 2 frontend закроет #32) Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:59:53 +03:00
lekss361	7250aa6187	feat(site-finder): weight profiles endpoints + analyze integration (#114 sub-PR 3/4) (#138) ... Admin CRUD router + non-breaking analyze_parcel integration. 46 tests pass. Refs: #114 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:40:06 +03:00
lekss361	389fd31fad	refactor(admin): extract _check_token into shared AdminTokenAuth dep (#133) ... Per audit batch #127 P2 hygiene (issue #128). ## Problem Identical `_check_token` function в 3 admin files: - backend/app/api/v1/admin_scrape.py:37-44 - backend/app/api/v1/admin_leads.py:26-33 - backend/app/api/v1/admin_jobs.py:24-31 ## Fix New `backend/app/core/deps.py` с `verify_admin_token` + Annotated alias: ```python AdminTokenAuth = Annotated[None, Depends(verify_admin_token)] ``` Заменено в 27 endpoints (18 admin_scrape + 6 admin_leads + 3 admin_jobs): - Removed local `_check_token` definitions - `_: None = Depends(_check_token)` → `_: AdminTokenAuth` - Removed unused imports (`Header`, `HTTPException`, `settings` где не используется) ## Semantics preserved Все 3 original implementations были identical (503 if env missing + 401 if mismatch, same header `X-Admin-Token`, same env `SCRAPE_ADMIN_TOKEN`). Минор: 503 detail text unified to `"admin disabled — set SCRAPE_ADMIN_TOKEN"` (в оригинале три разные строки — `"admin scrape disabled..."` etc). Cosmetic — detail в HTTP 503 не парсится клиентским кодом. ## Tests `uv run pytest tests/ -x -k admin` → 64/64 passed. ## Vault `code/patterns/Pattern_Admin_Token_Dependency_May14.md` — created. Closes #128 Refs: #127 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-14 23:45:44 +03:00
lekss361	1c2f80a4b8	feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116) ... * feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL) Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит свежие НСПД данные в UI (frontend integration — отдельный PR). Backend (new app/services/site_finder/quarter_dump_lookup.py): - `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment) - `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint: - Reads nspd_quarter_dumps row для derived quarter - Freshness threshold: 180 days - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire- and-forget (lazy import против circular), return EMPTY_DUMP_RESULT - Fresh + parcel_wkt=None → metadata only (no spatial queries) - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform (3857→4326) + ST_Intersects / ST_DWithin - 3 private helpers: - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1 - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance - `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used in get_quarter_dump_data internal + analyze_parcel try/except wrap) Backend (parcels.py): - Import EMPTY_DUMP_RESULT + get_quarter_dump_data - Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с resilience pattern других optional fetches) - Response gets 4 new fields: - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source) - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group) - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m) - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale, harvest_triggered, total_features) Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB): - derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace) - get_quarter_dump no_row → harvest triggered - stale (>180d) → harvest triggered, stale=True - harvest_error row → retry harvest triggered - parcel_wkt=None → metadata only (1 DB call) - fresh + zoning extraction - fresh + zouit_overlaps list - fresh integration: все 4 keys present 47 pre-existing tests still pass. Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4: - ✅ #1: try/except wrap around get_quarter_dump_data в analyze_parcel (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const - ✅ #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в nspd_dump.fetched_at_utc) - ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для harvest_error rows (только при high traffic + persistent NSPD errors); #4 raw_props в response — tech debt, убрать вместе с frontend PR Performance note: 3 spatial queries per analyze adds ~10-50ms on typical ~100-feature quarter. Mitigation if quarters grow dense: materialized per-layer sub-table (отдельная DB issue). Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный PR (next: #112 / #115 / #114). * fix(site-finder): address PR #116 auto-review M1-M5 M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with _make_empty_result() factory. dict(...) shallow copy left nested nspd_dump shared by reference across concurrent requests — single mutation pollutes module sentinel for all subsequent calls. Now каждый caller gets independent dict. M2 (O(N) spatial scan): SELECT extended denormalized counts (territorial_zones_count, zouit_count, engineering_count). Each spatial helper accepts layer_counts and early-returns when count=0 — skips heavy jsonb_array_elements + ST_Transform/ST_Intersects scan entirely. Critical для quarters с 2000+ features. M4 (documentation): _trigger_harvest docstring describes known burst/no-dedup limitation + TODO Redis SETNX (отдельный PR). M5 (test fragility): _make_db_mock_with_spatial docstring describes positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit, 3=engineering) и зависимость от count-values. +4 new tests (17 total, all pass): - test_make_empty_result_returns_independent_copies (mutation safety) - test_make_empty_result_overrides - test_early_exit_all_counts_zero_no_spatial_queries - test_early_exit_partial_counts Per auto-review on 3068a9c. * fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT singleton ref вместо factory. Сейчас readonly access, но нарушает documented invariant модуля. Rename `_make_empty_result` → `make_empty_result` (public API), import в parcels.py, использовать в try/except fallback. Каждый request получает независимый dict — никаких shared references. M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review, documented в code/issue. Acceptable trade-offs: - M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят WAF traffic впустую но не повреждают данные. - M5: docstring contractually describes positional-call order. 17/17 tests pass. ruff/format clean. Per auto-review on aef8308. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-13 09:14:19 +03:00
lekss361	f3b0ce5687	feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111) ... * feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A) Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR). Backend: - New `backend/app/workers/tasks/nspd_sync.py`: - `harvest_quarter(quarter_cad, region_code=66, include_zouit=True, include_risks=False)` — single-quarter harvest. autoretry_for= (NspdLiteWafError,) с retry_backoff=True, max_retries=3, soft_time_limit=120s. Generic exception → upsert error row с harvest_error=str(exc), counts=0 (НЕ re-raise). - `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` — beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad. - UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326)) для quarter_geom (schema MultiPolygon strict per database-expert contract). - `celery_app.py`: - Add `nspd_sync` в `include` - Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50, max_age_days=90 - `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0` (non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern) - `admin_scrape.py`: - `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only) - `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger, requires X-Admin-Token, returns task_id. Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB): - happy path: search_by_quarter mock → verify SQL params/structure - empty quarter → row with quarter_geom=NULL, total=0, no harvest_error - WAF error → autoretry kicks in (raise re-propagated) - Generic error → upsert error row - Fanout: 50 stale → 50 apply_async calls - Layer names preservation (all 5 zouit + 11 risks if enabled) - Geometry SRID handling Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred: - inline datetime import in error branch (style) - layers_fetched array literal via string concat (current layer names safe) - harvest_stale_quarters error rows always re-tried (intended, undocumented) - test call_args.kwargs fragile (works for current callsite) - beat collision with refresh-ekb-districts-medians every ~2 years (negligible) Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache. * fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT in harvest_stale_quarters — column does not exist in prod schema (cad_number, geom, raw_props, fetched_at, source). Would crash on first beat tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged. M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед db.close() — prevents PendingRollbackError на следующем session use, если PostGIS отбракует geometry или JSON cast failure. M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback — consistency с остальными session usages в celery_app.py. M4 (style): layers_fetched — заменил manual string concat '{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует list → text[]. Защита от future layer names с запятыми/фигурными скобками. M5 (style): import datetime as _dt поднят из else branch в top-level imports. M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не использует session как context manager — moc'и no-op. 14/14 tests pass. ruff/format clean. Per auto-review on 9b2a289. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 18:45:57 +03:00
lekss361	bc092b58f1	feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98) ... * feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels, I3 #44 engineering, и поддержки on-demand #93. Backend (`app/services/scrapers/nspd_client.py`): - `NSPDClient` с 4 core методами: - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo на точке: какие feature'ы layer'а покрывают (lon,lat) - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка) - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665) - Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses) - LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838, zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94 - Coordinate helpers: lonlat_to_3857(), bbox_around_point_m() - Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible urllib trick). Rate limit через rate_ms. - WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff. Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network. - LAYERS catalog sanity (territorial_zones=875838 закрывает G1) - Coordinate transforms (zero, ЕКБ center, bbox) - NSPDFeature.from_raw parsing (full / missing fields) - NSPDSearchResult helpers (empty/first) - _walk_layer_tree (flat / nested / empty) - search_by_cad через monkeypatch fetch_geoportal Также включён micro-fix из PR #95 review (cosmetic): - parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S) Scope NOT в этом PR (отдельные follow-up issues, sequential rule): - Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones, nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR - Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) → отдельный PR использует client - Admin endpoints (trigger / status) → отдельный PR - Замена on-demand fetch на nspd_client → PR after schema ready Closes part of #94 (foundation only — sub-issues for schema/tasks). * fix(scrapers): address PR #98 auto-review minor feedback Backend (nspd_client.py): 1. (line 100+) bbox_around_point_m signature split на multi-line. 2. (lazy imports) import json, time → module-level. 3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import. 4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty. 5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format. Tests: +5 mock тестов (19/19 passed): - get_feature_info URL builder + parsing - get_features_in_bbox bbox propagation - list_layers_walks_tree_response - list_layers_handles_data_wrapper - list_layers_handles_garbage_response Backend (parcels.py): docstring '~25с' → '~15с'. Per auto-review on be14388. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 09:47:45 +03:00