235 commits

Author	SHA1	Message	Date
lekss361	698ef4f003	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) ... Replaces tradein.cad_buildings manual snapshot (36k rows, frozen 2026-05-22) with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings directly. Also reverts PR #492 HTTP-based cadastral.py — superseded by FDW. Architecture: - gendesign DB: new role tradein_fdw_reader + flat view v_tradein_cad_buildings (EKB-only slice of cad_buildings with lat/lon flattened from geom) - networks: gendesign-postgres added to gendesign_shared (alias); tradein-postgres added to gendesign_shared for FDW connect - tradein DB: postgres_fdw extension + FOREIGN TABLE gendesign_cad_buildings - tradein backend: startup hook creates/refreshes USER MAPPING with password from env GENDESIGN_FDW_PASSWORD (password rotation handled via restart) - geocoder.py: cadastral primary for forward + reverse + suggest; Yandex/Nominatim fallback. reverse_geocode wraps Nominatim in try/except — no more HTTPStatusError → 500. - house_metadata.py and trade_in.py admin stats switched to foreign table - DROP TABLE cad_buildings in tradein (legacy snapshot removed entirely) - import-cadastre.sh deleted (no manual sync needed) Fixes: - /trade-in/api/v1/geocode/reverse 500 (Nominatim ban → no fallback) - estimate confidence_explanation=address_not_geocoded for addresses in our cadastre (e.g. Хохрякова 81) that Nominatim doesn't return Deploy ordering: main migration 100_tradein_fdw_role.sql adds role+view first (strict deploy.yml). Tradein next deploy applies 060_postgres_fdw_extension.sql and 061_drop_legacy_cad_buildings.sql (idempotent, errors ignored). Tradein backend startup creates USER MAPPING when env var present. Verify post-deploy with: docker exec tradein-postgres psql -U tradein -d tradein -c \ "SELECT count(*) FROM gendesign_cad_buildings" # expect ~36k EKB buildings.	2026-05-24 11:13:44 +03:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	b233bf91cc	refactor(security): убрать X-Admin-Token (Caddy basic_auth достаточен) (#437)	2026-05-23 10:41:22 +00:00
lekss361	f242d9bc44	feat(sf-fe-a12): Landing redesign + PilotCTA с real B3/B4 data (v2 rebased) (#367) ... Hero + 5 KPI cards from GET /api/v1/landing/stats (B4, real DB), Pricing card 30k руб/мес, FAQ accordion, PilotRequestModal -> POST /api/v1/pilot/request (B3). Round 2 fixes: - B4 fail-loud: LandingStatsOut.stale flag, zeroed fallback, frontend skeleton - B3 UUID contract: id:string, display tracking GD-{first8hex} - PilotRequestModal: replaced manual overlay with <Drawer side="bottom"> (A3 #341) — focus trap, ESC, scroll-lock, focus restore - Design tokens: all hex literals replaced with var(--*) - HeadlineBar component used instead of hand-rolled dark div Closes #82 Supersedes #366	2026-05-18 05:16:57 +00:00
lekss361	9ffeee7ed2	feat(22d): admin UI buttons (skip/force) + reduce catalog jitter 800→300ms (#352)	2026-05-17 23:38:22 +00:00
lekss361	f9adb5cc2f	feat(22d): catalog scraper — skip-today filter + force flag (#348)	2026-05-17 23:15:22 +00:00
lekss361	c363ea133f	fix(api): /parcels/by-bbox 500 — area via ST_Area, drop missing land_category (#347)	2026-05-17 23:09:18 +00:00
lekss361	e5498682ba	fix(22d): commit outer transaction in scrape_catalog_objects orchestrator ... PR #335 использовал `with db.begin_nested():` SAVEPOINT для per-row UPDATE, но outer transaction никогда не commit'ился. После цикла Celery task закрывал session (`db.close()` в finally) → autobegin'd outer tx rollback → все UPDATE'ы теряются. Симптом: task завершается с processed=3 succeeded=3, в Glitchtip нет ошибок, но в БД ни одного catalog_scraped_at != NULL. Fix: добавлен db.commit() после цикла + rollback на ошибку commit'а.	2026-05-18 01:10:06 +03:00
lekss361	96b03872ac	fix(22d): register scrape_kn_catalog_objects in celery include list (#339)	2026-05-17 21:54:13 +00:00
lekss361	bb7fbd5bc0	feat(sf-b4): GET /landing/stats — 5 KPI + paradox for landing page hero (#331)	2026-05-17 21:47:10 +00:00
lekss361	ea558b241c	feat(22d): admin endpoint POST /admin/scrape/kn-catalog-objects (#338)	2026-05-17 21:41:53 +00:00
lekss361	76112870e2	fix(pilot): drop pydantic EmailStr — email-validator dep missing → backend startup crash (#337)	2026-05-17 21:32:23 +00:00
lekss361	0567ad2130	feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)	2026-05-17 21:26:22 +00:00
lekss361	c61c8c86af	feat(sf-b1): GET /parcels/by-bbox — map entry endpoint + parcel_user_status table (#336)	2026-05-17 21:25:48 +00:00
lekss361	dbeed80b26	Merge pull request 'feat(sf-b3): POST /pilot/request — lead gen + pilot_requests table (Telegram TODO)' (#330) from feat/sf-b3-pilot-leads into main	2026-05-17 21:14:28 +00:00
lekss361	b013fd886c	feat(sf-b6): GET /parcels/{cad}/poi-score — weighted top-7 (#333)	2026-05-17 21:11:25 +00:00
lekss361	493238447a	feat(sf-b5): extend /parcels/{cad}/analyze with 15 fields (EGRN/encumbrance/red_lines/risks/district prices) (#332)	2026-05-17 21:11:18 +00:00
lekss361	4040cba4c0	feat(sf-b7): GET /parcels/{cad}/snapshot.pdf — 1-page WeasyPrint export (#334)	2026-05-17 21:11:03 +00:00
lekss361	bf55cc9e56	merge: resolve main.py conflict (keep both pilot + users routers)	2026-05-18 00:08:48 +03:00
lekss361	8673290009	feat(sf-b2): GET /users/me/recent-parcels — stub endpoint (auth TODO) (#329)	2026-05-17 21:04:16 +00:00
lekss361	99e55c0bc2	feat(sf-b3): POST /pilot/request + pilot_requests table ... Добавляет endpoint для приёма заявок на пилот (lead-gen). INSERT в pilot_requests, response {id, created_at, status}. Telegram-уведомление — TODO (creds не настроены, #307 SF-B3). Migration: data/sql/118_pilot_requests.sql	2026-05-18 00:01:35 +03:00
lekss361	e872b1ebe4	fix(sf): exclude NULL competitors from aggregate calculations (velocity, market_avg, top_sellers) (#328)	2026-05-17 20:46:07 +00:00
lekss361	028c6a1356	Merge pull request 'fix(scrape-kn): real documents endpoints (5x) + skip obj_checks + sem 3→8' (#325) from fix/domrf-real-doc-endpoints-skip-checks-sem6 into main	2026-05-17 20:08:50 +00:00
lekss361	7a8d59beb4	feat(obj-3): SF backend migrate to objective_lots ground truth (price 0.3% -> 81%) (#324)	2026-05-17 20:08:17 +00:00
lekss361	2aa33e0eb8	fix(stealth): bump Semaphore 6->8 for higher concurrency in gather ... Per-object 7 endpoints через asyncio.gather упираются в semaphore limit. 6 -> 8 — ещё +33% concurrency для batch fetch'а.	2026-05-17 23:04:51 +03:00
lekss361	0f289c05b1	fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 ... PR #322 used placeholder URLs: - /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints: document/rpd / developer/report / project/documentation / documentation/other / document/permits). Replaced with 5 parallel fetches + merge into single upsert_documents call. - /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk kn/object payload; endpoint /checks does not exist. Commented out _fetch_obj_checks_safe with TODO for separate investigation PR. - BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency for intra-object gather. GlitchTip: kn_scrape_failures run #19 showed 40 failures each for both placeholder endpoints across first 30 objects of 1532.	2026-05-17 22:59:44 +03:00
lekss361	f2983c86d7	feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322)	2026-05-17 19:40:16 +00:00
lekss361	3978aa5905	feat(trade-in): TI-2 PDF export 4-page + frontend enable button (#319)	2026-05-17 17:16:48 +00:00
lekss361	29dbbee2ac	feat(trade-in): TI-1 mock /trade-in/estimate endpoint + Pydantic + SQL migration (#316)	2026-05-17 16:45:06 +00:00
lekss361	389e141ec9	fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)	2026-05-17 16:37:30 +00:00
lekss361	fce48a8924	fix(scrape-objective): drop redundant self arg in sync_all_groups loop (#313)	2026-05-17 16:13:41 +00:00
lekss361	fd2b09b806	fix(22k): quartirography — use f.flat_type not f.type (column doesn't exist) (#312)	2026-05-17 16:09:00 +00:00
lekss361	f8c382f885	feat(22k): object page — full exposure 9 blocks (sales/quartirography/metro/photos/docs/specs/checks) (#310)	2026-05-17 15:56:39 +00:00
lekss361	4bed58b43c	feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309)	2026-05-17 15:40:01 +00:00
lekss361	b37e011aaa	feat(22c): domrf_kn_flat_plans new table + parser stub (#308)	2026-05-17 15:39:34 +00:00
lekss361	6612c1cd2b	feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 (#306)	2026-05-17 15:26:59 +00:00
lekss361	171a029997	feat(22i): domrf_kn_documents new table + parser (#305)	2026-05-17 15:21:31 +00:00
lekss361	3183abe41b	fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297) (#304)	2026-05-17 15:15:03 +00:00
lekss361	0b12a55eb6	fix(bug-22a): map missing kn-API fields (flat_number/living_area/wall_type/energy_eff) + backfill (#302)	2026-05-17 14:59:02 +00:00
lekss361	bfe16ae71b	fix(sf-16): derive price_per_m2 from price_rub in _norm_flat + backfill SQL (#298)	2026-05-17 14:32:23 +00:00
lekss361	e054395464	Merge pull request 'fix(sf-15): extract obj_class from aiDescription (DOM.РФ list API never returns objClass)' (#293) from fix/sf-15-obj-class-scraping into main	2026-05-17 14:18:49 +00:00
lekss361	3daf2a1fb1	fix(sf-15): extract obj_class from aiDescription (DOM.RF list API never returns objClass field) ... Add _extract_obj_class_from_ai() with regex patterns for Комфорт/Бизнес/Премиум/Элит/Стандарт. _norm_object() uses it as fallback; objClass from API still takes priority if ever returned. Add backfill SQL 105 to retroactively populate existing rows from raw JSON payloads. Coverage: ~1806/4548 rows will be filled; remainder NULL is expected (Нежилое/no AI desc).	2026-05-17 17:02:07 +03:00
lekss361	2104ed879b	fix(sf-21b): weighted AVG in _INLINE_VELOCITY_SQL (best_layouts.py) ... Replace unweighted AVG(deals_total_avg_area_m2) and AVG(deals_total_avg_price_thousand_rub_per_m2) with SUM(x * count) / NULLIF(SUM(count), 0) pattern in _INLINE_VELOCITY_SQL. Months with zero deals no longer dilute the weighted mean 2-4x. P0 follow-up to PR #290 (mv fix, issue #21).	2026-05-17 16:56:22 +03:00
lekss361	e1a8ad9395	fix(sf-17): rosreestr velocity fallback via cad_quarter (~100% EKB coverage) ... When Objective mapping coverage falls below 50% of competitors in the radius, fall back to rosreestr_deals JOIN on the parcel's cadastral quarter. Audit shows 237/237 EKB quarters (100%) have rosreestr data for the last 12 months, compared to <20% Objective coverage before bulk mapping. - velocity.py: add _compute_rosreestr_fallback(), _OBJECTIVE_COVERAGE_MIN_RATIO constant, velocity_source field on VelocityResult (objective/rosreestr_fallback/none) - parcels.py: extract cad_quarter from cad_num, pass to compute_velocity - site-finder.ts: add velocity_source field to Velocity interface - VelocityBlock.tsx: badge "Источник: квартальные сделки" when rosreestr_fallback Epic #271 item #17	2026-05-17 16:50:19 +03:00
lekss361	9eed55cd30	Merge pull request 'fix(sf-20): adaptive district threshold 30→15 with data_confidence flag' (#288) from fix/sf-20-district-threshold-adaptive into main	2026-05-17 13:39:52 +00:00
lekss361	989d4e5b7d	fix(sf-20): adaptive district threshold 30→15 with data_confidence flag ... - v_bucket_success_score HAVING >= 30 → >= 15 so sparse districts (Кировский etc.) are no longer silently excluded from the block - Backend: SUCCESS_REC_MIN_DEALS=15, SUCCESS_REC_STRONG_DEALS=30; data_confidence='weak' (15-29 deals) or 'strong' (≥30) in response - TS type ParcelSuccessRecommendation: add data_confidence literal - SuccessRecommendationBlock: amber badge for weak, all hex → CSS tokens, emoji star removed (ui-ux.md), empty-state text min 30→15	2026-05-17 16:34:52 +03:00
lekss361	2694e3180b	feat(etl): fuzzy matcher v2 — pruned threshold 0.85→0.80 for objective mapping coverage ... - Add AUTO_ACCEPT_THRESHOLD_V2 = 0.80 constant to objective_backfill.py - Add min_threshold param to find_match_candidates() (default REVIEW_THRESHOLD=0.6) - Add match_method param to auto_apply_matches() (default 'fuzzy_trgm') - Add ?v2=true query param to POST /api/v1/admin/etl/objective-backfill - v2 mode: threshold=0.80, method='fuzzy_v2', search from 0.80 (not 0.60) - Return type widened to dict[str,object] to include threshold_used + match_method_used DB audit: 1068 unmapped EKB objs; v1 adds ~133 rows, v2 adds ~47 more. Expected coverage: 8.5% → ~20% after sequential v1+v2 runs on prod. Run: POST /objective-backfill (v1) then POST /objective-backfill?v2=true (v2). Part of task #44 Part A, epic #271.	2026-05-17 16:28:20 +03:00
lekss361	b7ce4a589c	fix(sf-10): velocity LEFT JOIN + velocity_data_available flag ... Replace INNER JOIN on objective_complex_mapping with LEFT JOIN approach: competitors without mapping now return velocity=0 with velocity_data_available=False instead of being silently dropped. UI: VelocityBlock shows 'нет данных velocity' badge when flag is False and hides the gauge (meaningless zero). TS type updated (additive optional field, backward compat). Audit (EKB, region_cd=66): mapped=129, unmapped=1387, total=1516. Closes #271 item #10	2026-05-17 15:51:43 +03:00
lekss361	689327909c	fix(sf-09): capacity-aware redistribute (round 2) — bounded + correct invariant ... Round 1 (commit bcd7dc8) был broken: на 2-bucket входах surplus уходил в free полностью без учёта capacity → free превышал cap → следующая итерация clamp'ировала его и наоборот. Infinite oscillation в FastAPI handler. Round 2 fix per review BLOCK (#282 comment): - Surplus распределяется пропорционально available capacity (cap - v), не текущему v. Free никогда не вылетит выше cap. - free = строго < cap (не <=) — иначе деление на 0 capacity. - Hard guard `for _ in range(N+1)` — гарантированно завершается. - Pathological (surplus > total_capacity): возвращаем оригинальный pct_map + cap_skipped=True (sum=100 invariant сохранён). - Hamilton round вынесен в _hamilton_round() helper. Tests: - 2-bucket cases (90/10, 70/30, 99/1) expected cap_skipped=True - test_cap_iteration_count_bounded — все pathological завершаются < 100ms - All 13 cases verified standalone (3 fast-path + 7 reproduced + 3 pathological)	2026-05-17 15:30:50 +03:00
lekss361	bcd7dc8f75	fix(sf-09): iterative cap algorithm + cap_skipped flag + unit tests ... Review verdict MINOR (PR #282) — single-pass cap_and_redistribute не гарантировал invariant max ≤ MAX_BUCKET_SHARE_PCT. Surplus от clamped поднимал free выше cap (например {1k:75, studio:15, 2k:10} → studio=39). - backend/app/services/site_finder/best_layouts.py - _cap_and_redistribute → iterative while-loop до сходимости - Returns (result_map, cap_skipped) — флаг для pathological all-clamped - Float work map, Hamilton финальный pass для invariant sum=100 - backend/app/schemas/parcel.py - LayoutTzRecommendation.cap_skipped: bool = False - backend/tests/services/site_finder/test_best_layouts.py - 6 parametrized tests: invariants, 7 failing cases from review, no-dominant unchanged, empty, cap_skipped propagation, normal-case - frontend/src/types/best-layouts.ts — cap_skipped: boolean - frontend/src/components/site-finder/BestLayoutsBlock.tsx - banner условие сменилось с maxPct > 60 на rec.cap_skipped (honest signal)	2026-05-17 15:21:51 +03:00