1728 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
bot-backend	264701a4e4	Merge pull request 'fix(estimator): deterministic comp tie-breaker + n_analogs counts priced analogs' (#1939) from fix/estimator-tiebreaker-nanalogs into main	2026-06-27 00:58:15 +00:00
Light1YT	bcdc7ccb04	fix(estimator): deterministic comp tie-breaker + n_analogs counts priced analogs ... Found by adversarial valuation audit (2 confirmed, bot-safe). FIX A (#5): both radius comp queries (Tier H ~3990, Tier W ~4135) ended with a bare ORDER BY relevance_score; on score ties Postgres returned rows in undefined order, so the same /analyze could pick different comps across runs. Append deterministic tiebreaker: relevance_score ASC, distance_m ASC, scraped_at DESC NULLS LAST, id ASC (id = listings PK → total order). Added id to each base CTE; outer projection unchanged (no leak downstream). FIX B (#2): _filter_outliers keeps rows with price_per_m2 IS NULL, but the median is built from prices_ppm2 (drops them) while n_analogs counted all of listings_clean — overstating contributing comps ("Найдено N аналогов" misleading; all-price-less -> median=0 but n_analogs>0). Count n_analogs from prices_ppm2 in the radius path. #698 anchor overwrite + #691 zero-analog->low guard unaffected; listings_clean itself not filtered. Adds tests/test_estimator_n_analogs_priced.py (verified to fail on old code). Audit also flagged velocity fan-out (false-positive: 0 duplicate domrf_obj_id on prod) and >/>= disclosure tweaks (cosmetic) — deliberately not changed. Refs #1871	2026-06-27 05:57:04 +05:00
bot-backend	5968d78049	Merge pull request 'fix(tradein-browser): in-page fetch retry + settle tuning for /fetch-json (#1917)' (#1938) from fix/1917-fetch-json-network-retry into main	2026-06-27 00:31:20 +00:00
Light1YT	2687f75b92	fix(tradein-browser): in-page fetch retry + configurable settle for /fetch-json (#1917) ... First XHR after goto(origin) intermittently hit "NetworkError when attempting to fetch resource" (page net stack/anti-bot not ready), healed only by the outer re-navigation retry (~30-45s/house in the house_imv_backfill, #562). _fetch_json_once now: - settles FETCH_JSON_SETTLE_MS (default 1200, was hardcoded 500) — fewer first-fails; - wraps the in-page fetch() in a JS retry loop (FETCH_JSON_INPAGE_RETRIES default 1, FETCH_JSON_RETRY_DELAY_MS default 800) that retries ONLY on network throw, never on HTTP status (4xx/5xx short-circuit, caller decides). An in-page retry costs ~retryDelayMs vs the ~30-45s outer re-navigation. Last error re-thrown — outer crash-retry contract intact. /fetch (SERP) path untouched. +2 tests (settle ms, retry params). Refs #1917, #562	2026-06-27 05:30:16 +05:00
bot-backend	dfdd978528	Merge pull request 'test(tradein): cover avito/cian + mark_failed total_seen persistence (#1926)' (#1937) from fix/1926-scrape-runs-total-seen into main	2026-06-27 00:17:23 +00:00
Light1YT	8bb302aadf	test(tradein): cover avito/cian + mark_failed total_seen/new_count persistence (#1926) ... #1928 fixed scrape_runs.total_seen/new_count for sweeps via _column_counts but only regression-tested the yandex mark_done path. Add equivalents for the CitySweepCounters (avito/cian) sweep and the mark_failed path — both explicitly named in the #1926 audit — so the all-sweeps + failure-path guarantee is locked in. Refs #1926	2026-06-27 05:16:14 +05:00
bot-backend	a246044583	Merge pull request 'fix(tradein): backfill house_id_fk via source-identity path (#1781)' (#1936) from fix/1781-house-fk-source-identity into main	2026-06-27 00:05:14 +00:00
Light1YT	6ce60a0d8c	fix(tradein): backfill house_id_fk via source-identity path (#1781) ... Migration 130 only filled house_id_fk for listings carrying house_source/house_ext_id. Prod (2026-06-27): 1884 active NULL-FK rows remain, 0 with house_source/ext_id (130 re-run = no-op). They were matched via the per-listing-identity path the realtime mirror uses when no house catalog id exists: match_or_create_house(ext_source=source, ext_id=source_id) -> Tier-1 house_sources lookup (confidence 1.0). 136 replays that exact lookup offline: UPDATE listings SET house_id_fk = house_sources.house_id WHERE (ext_source=source, ext_id=source_id) and house_id_fk IS NULL. Faithful to base.py:704-750; idempotent (NULL guard); deterministic (no fuzzy/geo); collision-safe (catalog ids live under distinct ext_source like 'cian_newbuilding'). Dry-run BEGIN/ROLLBACK fills 4608 NULL-FK rows (744 active), restoring estimator Tier-S "same-building" grouping. The remaining ~1140 active need re-scrape/ re-match (scraper coverage — #1781 Лёха). Refs #1781	2026-06-27 05:01:02 +05:00
bot-backend	94d3a981c8	Merge pull request 'feat(analyze): per-category OSRM routing — foot vs driving (#39 A3)' (#1935) from feat/39-osrm-per-category into main	2026-06-26 23:43:28 +00:00
Light1YT	8a3bae0cda	feat(analyze): per-category OSRM routing (foot vs driving) (#39 A3) ... Walk-relevant POIs (school/shop/park/kindergarten/pharmacy/stops) now route via a FOOT OSRM graph (osrm-walk service), car-relevant POIs (mall/hospital + unknown) keep the DRIVING graph. Validation showed driving overstated pedestrian-proximity distance — median 1.6-2.9x straight-line (#39). - config: osrm_walk_local_url + osrm_walk_categories (frozenset, 9 walk cats) - osrm_client_local: base_url override on get_road_distances_m (default unchanged) - _apply_osrm_road_distances: split POIs by category, per-group OSRM call with independent graceful fallback (one server down -> its group keeps straight-line), in-place write-back by original index; never raises; flag-OFF byte-identical - docker-compose: osrm-walk service (foot graph, internal, mem_limit 1.5g) - build_osrm.sh: CAR=0 gate for foot-only refresh (doesn't touch live car graph) - tests: per-category split, per-group fallback, asymmetric intra-group write-back Still flag-gated (use_osrm_distances OFF) — enabling is a product decision. Refs #39	2026-06-27 04:30:25 +05:00
bot-backend	6c8bf0e496	Merge pull request 'feat(site-finder): OSM инж.сети на карту analyze (#1746)' (#1934) from feat/1746-utility-infra-map into main	2026-06-26 22:43:59 +00:00
Light1YT	854fa69a8b	feat(site-finder): render OSM utility-infrastructure layer on analyze map (#1746) ... Section2 "Сети" map now shows the OSM engineering-networks layer (osm_utility_infrastructure_ekb, endpoint /parcels/{cad}/utility-infrastructure) alongside the NSPD connection-points + ЗОУИТ layers from #1751, on the same map. - useUtilityInfrastructure hook (react-query, mirrors useConnectionPoints) - UtilityInfrastructureLayer: Point→CircleMarker, LineString/Polygon(+Multi)→GeoJSON, per-kind colour (power/water/gas/heat/communication/sewage), popups - UtilityLayerControlPanel: collapsible legend, per-kind toggle + count - SiteMap: optional utilityInfrastructure prop + visibleKinds state (CP/ЗОУИТ unchanged) - 12 unit tests (geometry parsing + [lon,lat]→[lat,lon] swap + grouping) Refs #1746	2026-06-27 03:40:51 +05:00
bot-backend	63ac12dc23	feat(tradein): recurring house-dedup merge (schedule dormant) (#1772) (#1933)	2026-06-26 21:39:30 +00:00
bot-backend	f48db87d39	feat(site-finder): OSRM road-distance in /analyze behind flag (#39 A2) (#1932)	2026-06-26 21:25:50 +00:00
bot-backend	7a1d2a4ed6	fix(site-finder): Overpass retry-backoff + mirror fallback for utility loader (#1746) (#1931)	2026-06-26 21:13:06 +00:00
bot-backend	3753079dee	feat(site-finder): OSM engineering-networks loader + endpoint (#1746) (#1930)	2026-06-26 20:01:00 +00:00
bot-backend	4fe20a9cbd	feat(devops): self-hosted OSRM routing engine for site-finder (#39) (#1929)	2026-06-26 19:18:21 +00:00
bot-backend	4846197d54	fix(site-finder): align competitor flats_sold to objective_lots.contract_date (#1926) (#1927)	2026-06-26 18:33:51 +00:00
bot-backend	89d05bbfec	fix(tradein): persist scrape_runs.total_seen/new_count from sweep counters (#1926) (#1928)	2026-06-26 18:09:17 +00:00
bot-backend	87364eebca	fix(test): test_849 expects 3 begin_nested/lot after #1921 upsert-reconcile SAVEPOINT — unblock red main gate (#1925)	2026-06-26 17:23:24 +00:00
bot-backend	d89210200b	perf(tradein): GIST index on (geom::geography) for estimator radius filter (#1189) (#1924)	2026-06-26 17:12:49 +00:00
lekss361	d111573a4b	Merge pull request 'fix(trade-in/ui): расшифровка ДКП/IMV/CV + графики на --viz токены + убрать emoji (#1871)' (#1922) from fix/tradein-redesign-pack into main ... Reviewed-on: #1922	2026-06-26 16:19:23 +00:00
lekss361	74ea6659ea	Merge pull request 'fix(scrapers): avito_full_load краш (source_id-reconcile) + cian/yandex ban-ротация' (#1921) from fix/scraper-resilience-upsert-rotation into main ... Reviewed-on: #1921	2026-06-26 16:19:06 +00:00
lekss361	5d5b2981fc	Merge pull request 'fix(estimator): флаг + тесты для ghost-anchor guard и radius-dedup (#1871)' (#1919) from fix/estimator-trust-ghost-dedup into main ... Reviewed-on: #1919	2026-06-26 16:18:58 +00:00
bot-backend	27766acf38	fix(trade-in/ui): расшифровка ДКП/IMV/CV + графики на --viz токены + убрать emoji из вердиктов (#1871) ... - HeroSummary: все вхождения ДКП/IMV/CV обёрнуты в <abbr title="..."> с расшифровкой - IMVBenchmark: «Avito IMV» → abbr с title - HeroSummary/HeroTransparency: emoji ⚠️ заменены на span-точки var(--warn) + текст - PriceHistoryChart: stroke="#2563eb"/"#ea580c" → var(--viz-1)/var(--viz-4) - PriceRangeBar: "#bfdbfe"/"#1d4ed8" → color-mix(--viz-1)/var(--viz-1) - SellTimeSensitivity: hardcoded bg-цвета → --success-soft/--accent-soft/--warn-soft/--danger-soft; border → var(--viz-1) - MapCard: пины обновлены на hex-значения --viz-1/--viz-3/--viz-4 (Leaflet JS, не CSS-контекст) - MapPicker: комментарий -- viz-1 hex (Leaflet constraint)	2026-06-26 19:16:55 +03:00
bot-backend	c1cdf3969c	fix(scrapers): ban/stuck-ротация для cian+yandex (provider-aware _rotate_proxy_ip)	2026-06-26 19:10:26 +03:00
bot-backend	197068c515	fix(scrapers): save_listings — UPDATE by (source,source_id) при dedup_hash-дрейфе (avito_full_load краш)	2026-06-26 19:10:12 +03:00
bot-backend	2fe34ac785	fix(estimator): ghost-anchor — confidence='low' when radius analogs empty (#1871) (#1920)	2026-06-26 16:07:12 +00:00
bot-backend	9678aface5	fix(estimator): dedup radius-аналогов — не раздувать n_analogs (#1871) ... Добавляет тест test_n_analogs_reflects_deduped_count: 3 дубля одного объекта (один source_id) → 1 кандидат (freshest scraped_at); 3 разных объекта → все 3. Верифицирует ключевое требование #1871 P2: n_analogs отражает дедупнутое число, а не raw кол-во строк в listings с дублями (prod: 797 yandex + 93 cian дублей раздували n_analogs и искажали медиану/перцентили).	2026-06-26 18:52:11 +03:00
bot-backend	68f7c0e015	fix(estimator): запрет confidence=high при n_analogs=0 (#1871 ghost-anchor) ... Добавляет settings-флаг estimate_confidence_floor_no_analogs (дефолт True), гейтящий вызов _enforce_zero_analog_low перед сборкой AggregatedEstimate. При n_analogs==0 и confidence!='low' форсит 'low' + добавляет caveat в explanation («без сопоставимых аналогов рядом»), предотвращая показ выдуманного «высокого» доверия когда оценка построена только на внешних оценщиках (yandex_valuation/cian_valuation) без реальных рыночных аналогов.	2026-06-26 18:50:19 +03:00
bot-backend	e03c810306	feat(tradein-imv): route IMV backfill через BrowserFetcher.fetch_json за dormant-флагом (#915 Stage 3) (#1916)	2026-06-26 09:54:28 +00:00
bot-backend	d479edeb3f	feat(tradein-browser): additive POST /fetch-json + BrowserFetcher.fetch_json (#915 Stage 2) (#1915)	2026-06-26 09:30:35 +00:00
bot-backend	e0cd378d19	fix(migration-133): dedup via cascade-delete, not child re-point (prod hotfix) ... Migration 133 failed on prod (offer_price_history UNIQUE (listing_id,change_time) + multi-loser collisions). Drop data-child re-points, rely on FK ON DELETE CASCADE; keep only merged_into. Dry-run on prod: 9840 groups, 9889 losers deleted, clean. Refs #1773	2026-06-26 08:16:27 +00:00
bot-backend	1d5b081cb4	fix(cian): recover new-build house via cadastral anchor + listings (source,source_id) dedup (#1773) (#1912) ... cian new-build cadastral anchor + listings (source,source_id) dedup (#1773): migration 133 (deep-reviewed data-loss-safe). Refs #1773	2026-06-26 07:45:22 +00:00
bot-backend	3f1925052b	feat(scheduler): wire house_imv_backfill as scheduled source (#854) (#1910) ... wire house_imv_backfill as scheduled source (#854): trigger fn + seed migration 132. Refs #854	2026-06-26 07:45:08 +00:00
bot-backend	f091554942	fix(scrapers): named BrowserSession concurrency const, fix Semaphore(3) doc drift (#1347) (#1913) ... named BrowserSession concurrency const + fix Semaphore(3) doc drift (#1347): keeps 8 (issue-sanctioned doc option). Refs #1347	2026-06-26 07:44:29 +00:00
bot-backend	9c52e0b29f	feat(etl): housing-class normalization fallback via yandex_realty trigram match (#38) (#1911) ... housing-class normalization fallback via yandex_realty trigram (#38): migration 169 + ETL + COALESCE consumers. Refs #38	2026-06-26 07:44:25 +00:00
bot-backend	5ed88c41a4	fix(avito): post-merge polish — unify fetch_mode guard + empty-render guard (#1811) (#1909) ... avito SERP browser post-merge polish (#1811): unify fetch_mode guard, empty-render guard, debug logging. Refs #1811	2026-06-26 07:44:22 +00:00
bot-backend	67d41df880	fix(freshness): repoint nspd source to live nspd_quarter_dumps (data-table mode) ... nspd freshness watched a defunct ledger (nspd_scrape_runs, WAF-banned manual April runs) → false "no successful runs". Add data-table mode (timestamp_col) + repoint nspd to live nspd_quarter_dumps.fetched_at_utc (fresh<14/stale<30). Run-ledger sources untouched. Refs #73	2026-06-26 06:51:00 +00:00
bot-backend	2304e09e37	fix(freshness): calibrate kn staleness thresholds to weekly cadence ... kn (DOM.РФ ЖК) runs weekly (Mon cron) but thresholds were 2/5 → false "stale" every Wed-Sun. Recalibrate to 8/14 (weekly, mirrors objective) + regression test. Surfaced by the freshness monitor's first prod run. Refs #73	2026-06-26 06:33:29 +00:00
bot-backend	66da75681f	feat(financial/dcf): dormant separate-index inflation capability (default OFF) ... smr/sale_inflation_annual in _build_monthly_cashflow, gated by module constants =0.0 → byte-identical zero-diff (deep-review verified IEEE-754 identity + all 57 prior tests unchanged). Land not escalated; marketing/VAT/tax retimed on escalated revenue (total preserved). Capability wired+tested, OFF until calibrated indices + invariant reframe. No schema/codegen. Refs #1881	2026-06-25 12:12:02 +00:00
bot-backend	b82963d12e	feat(financial): ground-floor нежилое (office/commercial) tranche ... Carve a documented office share (comfort/business 5%, econom 0) out of GFA before residential — fixes prior overstatement (all non-parking GFA = residential). total_floor_area unchanged → no double-count; office is additive revenue (+15% premium), нежилое → VAT-able (vat base = parking_va + office_va). Σ invariant holds. Additive schema + api-types regen + PDF/concept display. Deep-review ✅. Refs #1881	2026-06-25 11:40:01 +00:00
bot-backend	55811ff7cc	feat(workers): data-freshness monitor (Celery beat → GlitchTip) ... Extract compute_freshness(db) from the /admin/scrape/freshness handler (endpoint unchanged) + daily beat task scrape_freshness_check that alerts via sentry_sdk.capture_message when a source is stale/failed (error on critical-failed, else warning). Registered in celery include + beat (09:00 MSK). Refs #73	2026-06-25 11:29:34 +00:00
bot-backend	f249212695	fix(site-finder): isolate zone-regulation cache write off the request session (#1850 item 4) ... Cache-miss upsert now runs in a fresh SessionLocal() instead of committing the shared /analyze transaction mid-request. Read-back via shared session unchanged. Refs #1850	2026-06-25 11:16:26 +00:00
bot-backend	39f099bc17	chore(mocks): regenerate parcel-analyze financial_estimate (consistent with current backend) ... Regenerate the drifted dev fixture financial_estimate via compute_financial() from its own teap_synth inputs — parking comfort-priced (#1893), VAT parking-only (#1898), levered_irr refreshed (#1900). Dev-only. Refs #1881	2026-06-25 11:02:36 +00:00
bot-backend	d2773e0c91	fix(site-finder): optimize zone-regulation resolve in /analyze (#1850) ... Memoize coord→zone_index WFS lookup (thread-safe bounded LRU, negative caching), resolve once and thread into IRD block (no double resolve when both flags on), unify timeout into settings.geoportal_timeout_s. Lock guards the threadpool-shared memo (WFS call outside lock). Documented the cache-miss db.commit() smell (separate task). Deep-review ✅. Closes #1850	2026-06-25 09:12:41 +00:00
bot-backend	d589ddf6c4	feat(financial): levered (equity) IRR — capstone финмодели #1881 ... Levered (equity) IRR from the LTC debt-schedule equity cashflow, mirroring the unlevered IRR + roi-proxy fallback. Honest negative-leverage (can be below project IRR). Σ equity_cf == net_profit_after_financing invariant holds. Unlevered untouched. Additive schema fields (api-types regenerated). Cockpit drawer + concept display. Deep-review ✅. Refs #1881	2026-06-25 09:06:46 +00:00
bot-backend	dc659da655	feat(site-finder): geo-radius market-price calibration in /analyze (#1881 follow-up) ... New price tier objective_geo_radius: ST_DWithin median of Objective new-build prices (objective_lots ⋈ complexes) within 3km of parcel centroid, between quarter-MV and district_reference. Closes the name-match gap (5 of 9 EKB districts had no Objective name-match). data/sql/168 functional GIST index (prod EXPLAIN 114ms→28ms). Degenerate-centroid guard + honest RU price_source captions. Deep-review ✅. Refs #1881	2026-06-25 07:59:56 +00:00
bot-backend	3073651812	fix(generative/financial): VAT only on parking, residential ДДУ exempt (ст. 149 НК) ... VAT charged on full gross margin → only on parking value-added; residential (реализация жилья + услуги застройщика по ДДУ) exempt per пп.22/23.1 п.3 ст. 149 НК; input VAT embedded in gross СМР. Σ cashflow == net_profit invariant preserved. PDF + concept-UI caveats synced. +3 tests, deep-review ✅. Refs #1881	2026-06-25 07:22:58 +00:00
bot-backend	89509824c2	chore(agents): bump all worker + analyst agents to opus 4.8 ... All worker + analyst project agents → model: opus (Opus 4.8); reviewers already opus. Guard files (_autonomous_pickup, auto-code-reviewer) untouched. Per Anton request.	2026-06-25 06:54:14 +00:00