384 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
lekss361	0c4d7f1ffb	feat(obj-2): mapping backfill via pre-materialized temp tables (migration 117) (#327)	2026-05-17 20:34:42 +00:00
lekss361	2bd80c5719	fix(obj-2): trim migration 116 to GIST index only — INSERT CTE timed out in deploy (#326)	2026-05-17 20:21:58 +00:00
lekss361	028c6a1356	Merge pull request 'fix(scrape-kn): real documents endpoints (5x) + skip obj_checks + sem 3→8' (#325) from fix/domrf-real-doc-endpoints-skip-checks-sem6 into main	2026-05-17 20:08:50 +00:00
lekss361	7a8d59beb4	feat(obj-3): SF backend migrate to objective_lots ground truth (price 0.3% -> 81%) (#324)	2026-05-17 20:08:17 +00:00
lekss361	2aa33e0eb8	fix(stealth): bump Semaphore 6->8 for higher concurrency in gather ... Per-object 7 endpoints через asyncio.gather упираются в semaphore limit. 6 -> 8 — ещё +33% concurrency для batch fetch'а.	2026-05-17 23:04:51 +03:00
lekss361	0f289c05b1	fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 ... PR #322 used placeholder URLs: - /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints: document/rpd / developer/report / project/documentation / documentation/other / document/permits). Replaced with 5 parallel fetches + merge into single upsert_documents call. - /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk kn/object payload; endpoint /checks does not exist. Commented out _fetch_obj_checks_safe with TODO for separate investigation PR. - BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency for intra-object gather. GlitchTip: kn_scrape_failures run #19 showed 40 failures each for both placeholder endpoints across first 30 objects of 1532.	2026-05-17 22:59:44 +03:00
lekss361	79c9a8a034	feat(obj-2): multi-feature mapping backfill (name+dev+district) — 142 → 200+ (#323)	2026-05-17 19:53:50 +00:00
lekss361	f2983c86d7	feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322)	2026-05-17 19:40:16 +00:00
lekss361	c80d79e2d4	fix(objective parser): use_float=True in ijson + default=str in json.dumps (#321)	2026-05-17 19:11:25 +00:00
lekss361	d1b721a3be	fix(deploy): trigger workflow on all data/sql/** changes, not only .sql (#320)	2026-05-17 18:51:28 +00:00
lekss361	bb9a003080	fix(objective parser): wrap bytes-iter in file-like reader for ijson (#318)	2026-05-17 17:52:55 +00:00
lekss361	3978aa5905	feat(trade-in): TI-2 PDF export 4-page + frontend enable button (#319)	2026-05-17 17:16:48 +00:00
lekss361	90eb5af883	feat(trade-in): TI-3 frontend /trade-in page + 5 components + hooks (#317)	2026-05-17 16:58:53 +00:00
lekss361	29dbbee2ac	feat(trade-in): TI-1 mock /trade-in/estimate endpoint + Pydantic + SQL migration (#316)	2026-05-17 16:45:06 +00:00
lekss361	389e141ec9	fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)	2026-05-17 16:37:30 +00:00
lekss361	fce48a8924	fix(scrape-objective): drop redundant self arg in sync_all_groups loop (#313)	2026-05-17 16:13:41 +00:00
lekss361	fd2b09b806	fix(22k): quartirography — use f.flat_type not f.type (column doesn't exist) (#312)	2026-05-17 16:09:00 +00:00
lekss361	a93bef4147	feat(deploy): inject OBJECTIVE_API_KEY into backend/.env.runtime (#311)	2026-05-17 16:05:13 +00:00
lekss361	f8c382f885	feat(22k): object page — full exposure 9 blocks (sales/quartirography/metro/photos/docs/specs/checks) (#310)	2026-05-17 15:56:39 +00:00
lekss361	4bed58b43c	feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309)	2026-05-17 15:40:01 +00:00
lekss361	b37e011aaa	feat(22c): domrf_kn_flat_plans new table + parser stub (#308)	2026-05-17 15:39:34 +00:00
lekss361	6612c1cd2b	feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 (#306)	2026-05-17 15:26:59 +00:00
lekss361	171a029997	feat(22i): domrf_kn_documents new table + parser (#305)	2026-05-17 15:21:31 +00:00
lekss361	3183abe41b	fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297) (#304)	2026-05-17 15:15:03 +00:00
lekss361	16e3d5c86d	feat(22f): domrf_obj_checks table — 6 "Проверено на наш.дом.рф" checks (#303)	2026-05-17 15:07:24 +00:00
lekss361	0b12a55eb6	fix(bug-22a): map missing kn-API fields (flat_number/living_area/wall_type/energy_eff) + backfill (#302)	2026-05-17 14:59:02 +00:00
lekss361	bfe16ae71b	fix(sf-16): derive price_per_m2 from price_rub in _norm_flat + backfill SQL (#298)	2026-05-17 14:32:23 +00:00
lekss361	e054395464	Merge pull request 'fix(sf-15): extract obj_class from aiDescription (DOM.РФ list API never returns objClass)' (#293) from fix/sf-15-obj-class-scraping into main	2026-05-17 14:18:49 +00:00
lekss361	a86cce79af	refactor(sf-19): BestLayouts inline hex → Tailwind tokens (#296)	2026-05-17 14:17:05 +00:00
lekss361	4b967343f8	fix(sf-13): MarketTab above-the-fold headline + 3 KPI cards (#294)	2026-05-17 14:10:06 +00:00
lekss361	33f44b59cb	feat(sf-18): is_sales_started flag для domrf_kn_objects (#292)	2026-05-17 14:09:21 +00:00
lekss361	440bba99cc	fix(sf-15): rename migration 105 → 106 (collision with PR #292 sf-18)	2026-05-17 17:03:16 +03:00
lekss361	9f5f887393	Merge pull request 'fix(sf-21): mv_layout_velocity weighted AVG (P0 data corruption)' (#290) from fix/sf-21-mv-weighted-avg into main	2026-05-17 14:02:46 +00:00
lekss361	9f7224de33	Merge pull request 'fix(sf-21b): _INLINE_VELOCITY_SQL weighted AVG (P0 follow-up)' (#291) from fix/sf-21b-inline-velocity-weighted into main	2026-05-17 14:02:43 +00:00
lekss361	3daf2a1fb1	fix(sf-15): extract obj_class from aiDescription (DOM.RF list API never returns objClass field) ... Add _extract_obj_class_from_ai() with regex patterns for Комфорт/Бизнес/Премиум/Элит/Стандарт. _norm_object() uses it as fallback; objClass from API still takes priority if ever returned. Add backfill SQL 105 to retroactively populate existing rows from raw JSON payloads. Coverage: ~1806/4548 rows will be filled; remainder NULL is expected (Нежилое/no AI desc).	2026-05-17 17:02:07 +03:00
lekss361	2104ed879b	fix(sf-21b): weighted AVG in _INLINE_VELOCITY_SQL (best_layouts.py) ... Replace unweighted AVG(deals_total_avg_area_m2) and AVG(deals_total_avg_price_thousand_rub_per_m2) with SUM(x * count) / NULLIF(SUM(count), 0) pattern in _INLINE_VELOCITY_SQL. Months with zero deals no longer dilute the weighted mean 2-4x. P0 follow-up to PR #290 (mv fix, issue #21).	2026-05-17 16:56:22 +03:00
lekss361	f0589bcdff	Merge pull request 'fix(sf-17): rosreestr velocity fallback via cad_quarter (~100% EKB coverage)' (#289) from fix/sf-17-rosreestr-velocity-fallback into main	2026-05-17 13:55:20 +00:00
lekss361	913cdcfcef	fix(sf-21): mv_layout_velocity weighted AVG — P0 data corruption ... AVG(deals_total_avg_area_m2) averaged pre-aggregated monthly values including months with 0 deals (area=0, price=0), pulling results down 2-14x. Replace with SUM(x * count) / NULLIF(SUM(count), 0). Verified on prod: obj 65307 room 4+ area 7.4->105.8 m2, price 7.9->112 k/m2. Migration 100_ includes REFRESH — no manual step needed post-deploy.	2026-05-17 16:53:36 +03:00
lekss361	e1a8ad9395	fix(sf-17): rosreestr velocity fallback via cad_quarter (~100% EKB coverage) ... When Objective mapping coverage falls below 50% of competitors in the radius, fall back to rosreestr_deals JOIN on the parcel's cadastral quarter. Audit shows 237/237 EKB quarters (100%) have rosreestr data for the last 12 months, compared to <20% Objective coverage before bulk mapping. - velocity.py: add _compute_rosreestr_fallback(), _OBJECTIVE_COVERAGE_MIN_RATIO constant, velocity_source field on VelocityResult (objective/rosreestr_fallback/none) - parcels.py: extract cad_quarter from cad_num, pass to compute_velocity - site-finder.ts: add velocity_source field to Velocity interface - VelocityBlock.tsx: badge "Источник: квартальные сделки" when rosreestr_fallback Epic #271 item #17	2026-05-17 16:50:19 +03:00
lekss361	9eed55cd30	Merge pull request 'fix(sf-20): adaptive district threshold 30→15 with data_confidence flag' (#288) from fix/sf-20-district-threshold-adaptive into main	2026-05-17 13:39:52 +00:00
lekss361	fcfc492ab5	Merge pull request 'feat: fuzzy matcher v2 — Objective mapping coverage 8.5%→20%' (#286) from feat/fuzzy-matcher-v2-coverage into main	2026-05-17 13:39:28 +00:00
lekss361	edb0a3e728	Merge pull request 'fix(sf-12): MarketTrend — adaptive n&lt;30/n&lt;100 confidence warnings' (#287) from fix/sf-12-trend-confidence-warning into main	2026-05-17 13:39:03 +00:00
lekss361	989d4e5b7d	fix(sf-20): adaptive district threshold 30→15 with data_confidence flag ... - v_bucket_success_score HAVING >= 30 → >= 15 so sparse districts (Кировский etc.) are no longer silently excluded from the block - Backend: SUCCESS_REC_MIN_DEALS=15, SUCCESS_REC_STRONG_DEALS=30; data_confidence='weak' (15-29 deals) or 'strong' (≥30) in response - TS type ParcelSuccessRecommendation: add data_confidence literal - SuccessRecommendationBlock: amber badge for weak, all hex → CSS tokens, emoji star removed (ui-ux.md), empty-state text min 30→15	2026-05-17 16:34:52 +03:00
lekss361	1e1737e829	fix(sf-12): MarketTrend — adaptive n<30/n<100 confidence warnings ... n<30: hide trend block, show 'insufficient data — expand radius'. 30≤n<100: show trend + amber banner 'weak data (n=X) — expand radius'. n≥100: unchanged, no banner.	2026-05-17 16:33:22 +03:00
lekss361	2694e3180b	feat(etl): fuzzy matcher v2 — pruned threshold 0.85→0.80 for objective mapping coverage ... - Add AUTO_ACCEPT_THRESHOLD_V2 = 0.80 constant to objective_backfill.py - Add min_threshold param to find_match_candidates() (default REVIEW_THRESHOLD=0.6) - Add match_method param to auto_apply_matches() (default 'fuzzy_trgm') - Add ?v2=true query param to POST /api/v1/admin/etl/objective-backfill - v2 mode: threshold=0.80, method='fuzzy_v2', search from 0.80 (not 0.60) - Return type widened to dict[str,object] to include threshold_used + match_method_used DB audit: 1068 unmapped EKB objs; v1 adds ~133 rows, v2 adds ~47 more. Expected coverage: 8.5% → ~20% after sequential v1+v2 runs on prod. Run: POST /objective-backfill (v1) then POST /objective-backfill?v2=true (v2). Part of task #44 Part A, epic #271.	2026-05-17 16:28:20 +03:00
lekss361	eb865a08bc	Merge pull request 'fix(sf-11): Top layouts — sales period column + drill-in expand' (#285) from fix/sf-11-sales-period-drill-in into main	2026-05-17 13:19:24 +00:00
lekss361	1959283a41	fix(sf-11): review-pass — DOM.РФ latin URL + Fragment key + a11y keyboard ... Per review #285 (HIGH BLOCK): - DOM.РФ URL: кириллический slug → latin path (/services/catalog-newbuildings/object/{id}) для stable behavior - Replace <> Fragment без key → <Fragment key={signature}> (убирает React warning при .map). Closing </> → </Fragment> - a11y: <tr onClick> теперь имеет tabIndex, role="button", aria-expanded, onKeyDown (Enter/Space) — keyboard drill-in Inline hex tokens (medium) — отложено в follow-up (epic #271 #19).	2026-05-17 16:16:24 +03:00
lekss361	d3a51c3657	fix(sf-11): Top layouts — sales period column + drill-in expand ... Add «Срок продажи (мес)» column (supply / velocity, colored by urgency) and click-to-expand drill-in row showing competitor_obj_ids as DOM.РФ linked badges. Both fields were already present in TopLayoutRow type. Closes #271 item 11	2026-05-17 16:07:30 +03:00
lekss361	e1300b1955	Merge pull request 'fix(sf-10): velocity LEFT JOIN + velocity_data_available flag' (#284) from fix/sf-10-velocity-left-join-v2 into main	2026-05-17 13:00:19 +00:00
lekss361	b7ce4a589c	fix(sf-10): velocity LEFT JOIN + velocity_data_available flag ... Replace INNER JOIN on objective_complex_mapping with LEFT JOIN approach: competitors without mapping now return velocity=0 with velocity_data_available=False instead of being silently dropped. UI: VelocityBlock shows 'нет данных velocity' badge when flag is False and hides the gauge (meaningless zero). TS type updated (additive optional field, backward compat). Audit (EKB, region_cd=66): mapped=129, unmapped=1387, total=1516. Closes #271 item #10	2026-05-17 15:51:43 +03:00