1622 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Light1YT	d93553c489	fix(site-finder): перенумеровать под-подписи Section3 4.1/4.2/4.3 под новый порядок + doc-хедеры (#1741 хвост)	2026-06-18 12:38:54 +05:00
bot-backend	f9769571a3	fix(cadastral): single-statement match UPDATE — OFFSET chunk skipped rows ... The chunked matcher paged the candidate set with OFFSET over the very predicate it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET (reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched listings (beyond threshold) accumulate at the low-id front, the chunk matches 0 and the loop breaks early, leaving higher-id matchable listings unfilled. Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks, missing 7 and 10. Unit tests used single-chunk fixtures so never caught it. The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the signature for schedule-param compat (now unused).	2026-06-18 10:28:19 +03:00
bot-backend	edd187b47f	feat(cadastral): geo-nearest building matcher via local cad mirror ... Backfill listings.building_cadastral_number (was 0%) from the nearest cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k listings). Instead materialize the FDW once into a LOCAL cad_buildings_local table (Point geom + GIST), then run a fast local KNN nearest-neighbour join. Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE defeated the index (58s+, full update never finished); the geometry-gate design does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end), matching 16027 listings at 50m across 2229 distinct buildings. The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches the nearest building within threshold_m (default 50m), not necessarily its exact cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels FDW not exposed). Threshold is logged. - 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW (deploy-independent); populated by the refresh job. - 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at tomorrow 09:00 UTC (after geocode_missing). - tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan), match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match run-lifecycle wrapper. - scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).	2026-06-18 10:18:54 +03:00
Light1YT	8cd0620543	feat(site-finder): продуктовые решения по фидбеку analyze (#1741 #1742 #1743 #1744 #1745) ... После ролевой дискуссии (аналитик×комдир×собственник): - #1741 секции в 2 группы «Участок»→«Стройка и рынок» (Section1→4→2→3→5→6), gate-баннер вынесен НАД группой (убран дубль из Section4Estimate), nav-группы - #1742 объяснение выбора класса: фраза-причина + мини-таблица 3 классов по deficit_index («+1 острый дефицит, −1 затоварка»). POI «обеспеченность»/market-overstock relabel — N/A (в UI не выводятся) - #1743 «Остатки и скорость»→«Как продаётся рынок рядом» + headline-вердикт + «Темп продаж»/«Будущее предложение конкурентов (24 мес)» (убран англицизм Velocity-score/Pipeline) + интерпретированные единицы - #1744 горизонт: caption+tooltip «пересчёт ~10-30с» + статус «Пересчитываем…»; колонки прогноза «…, квартир» + caveat advisory - #1745 фаза1: таблица «Прогноз по всем форматам» (студия/1-5к × индекс дефицита/прогноз спроса/сигнал); backend пробросил per-format projected_demand_units + signal. Слайдеры/рубли — фаза2 после бэктеста #951 Verify: tsc 0 ошибок; py_compile; what_to_build/recommendation/report/exporter тесты pass. Closes #1741 Closes #1742 Closes #1743 Closes #1744 Closes #1745	2026-06-18 12:18:04 +05:00
lekss361	6214baf0c9	Merge pull request 'fix(site-finder): 5 дефектов из фидбека analyze (#1736-1740)' (#1748) from fix/sf-feedback-batch into main	2026-06-18 07:06:45 +00:00
bot-backend	3280859170	Merge remote fix/sf-feedback-batch (#1736 minimap test) into local	2026-06-18 10:05:56 +03:00
bot-backend	c859872217	fix(parcels): wrap forecast-export PDF in try/except+logger (#1739 completion)	2026-06-18 10:05:25 +03:00
bot-backend	0f9babb7ff	test(site-finder): wrap MiniMap test render in QueryClientProvider (#1736) ... #1736 added useConnectionPoints (useQuery) to MiniMap, but MiniMap.test.tsx rendered without a QueryClient -> 'No QueryClient set' (2 failing tests in CI frontend-tests). Wrap renders in a QueryClientProvider (retry:false).	2026-06-18 10:03:48 +03:00
bot-backend	bcd24923b8	Merge remote-tracking branch 'forgejo-backend/main' into fix/sf-feedback-batch	2026-06-18 10:01:43 +03:00
bot-backend	e7342d9f98	chore(frontend): regenerate api-types.ts for forecast pdf export (#1739) ... #1748 added format=pdf to the forecast export endpoint but didn't regenerate the frontend OpenAPI types, failing CI openapi-codegen-check. Regenerated via the CI recipe (app.openapi() dump -> openapi-typescript -> prettier).	2026-06-18 10:00:26 +03:00
lekss361	7448db0be0	Merge pull request 'feat(avito): dedicated newbuilding (novostroyka) citywide sweep' (#1747) from feat/avito-newbuilding-sweep into main	2026-06-18 07:00:22 +00:00
Light1YT	b1c0ea1268	fix(site-finder): 5 дефектов из боевого фидбека analyze (#1736 #1737 #1738 #1739 #1740) ... - #1736 MiniMap: проброс useConnectionPoints → точки подключения на карте analyze (были только в /legacy) - #1737 confidence: пронесено имя сервиса → RU-ярлык (Рынок/Будущее предложение/…) вместо «Компонент вкладывающий сервис» - #1738 pipeline: self-exclusion субъекта (ST_DWithin 80м) — проект не считает сам себя будущим конкурентом - #1739 PDF: snapshot_pdf обёрнут в try/except+logger.exception (причина 500 видна) + format=pdf в forecast export + font_url fallback - #1740 gate↔recommendation: при can_build_mkd=False — gate_caveat на обоих рекомендаторах (противоречие явное, не молчит) Verify: py_compile 5/5, tsc 0, ruff clean, pytest confidence/forecast 95 passed. Closes #1736 Closes #1737 Closes #1738 Closes #1739 Closes #1740	2026-06-18 11:47:21 +05:00
bot-backend	1a658a65a6	feat(avito): dedicated newbuilding (novostroyka) citywide sweep	2026-06-18 09:45:54 +03:00
lekss361	bf91eb0be0	Merge pull request 'feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1)' (#1735) from feat/browser-per-source-proxy-pool into main ... Reviewed-on: #1735	2026-06-18 06:06:57 +00:00
bot-backend	b9eb478c8f	feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) ... Route each scraper source (avito/cian/yandex/domclick) to its own camoufox browser+proxy so they no longer wedge each other through a single global egress. Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the /fetch and /login paths are byte-for-byte the existing single-browser behavior. When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP (BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its own lazy-launched lock. /login stays single-browser in Phase 1. BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body; all scraper callsites pass their source. No docker-compose/.env.runtime changes (Phase 2, owner-gated).	2026-06-18 09:01:50 +03:00
lekss361	635851a71c	Merge pull request 'fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725)' (#1734) from fix/yandex-predicted-price-path-1725 into main ... Reviewed-on: #1734	2026-06-18 05:12:25 +00:00
bot-backend	41f2402b25	fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725) ... Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice, not predictedPrice.predictedPrice (that key does not exist). _predicted_prices read the wrong outer key, so predicted_price_rub/_min/_max were None for every offer (prod: pp=0 across all 6189 yandex rows). Verified live against the gate-API 2026-06-18. The rich-field test fixture used the same wrong wrapper, so the test passed vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice shape — with the old code the assertions now fail, confirming non-vacuity.	2026-06-18 08:03:05 +03:00
lekss361	8f9c11a45f	Merge pull request 'feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g' (#1733) from feat/browser-concurrent-pool into main	2026-06-17 21:07:31 +00:00
bot-backend	a6717705b9	fix(browser): reset page_counter on recycle + close login page in finally (#796 review)	2026-06-18 00:06:50 +03:00
bot-backend	820fd819bb	feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g	2026-06-17 23:55:11 +03:00
lekss361	6d3fad6b96	Merge pull request 'fix(backfill): drop nonexistent kadastr_num column from listings SELECT' (#1732) from fix/backfill-kadastr-column into main	2026-06-17 20:53:26 +00:00
bot-backend	73fa739e0f	fix(backfill): drop nonexistent kadastr_num column from listings SELECT	2026-06-17 23:47:30 +03:00
lekss361	367b2719a9	Merge pull request 'fix(avito): proactive IP rotate at sweep start + raise max_rotations 2→4 (Datadome ban recovery)' (#1731) from fix/avito-proactive-ip-rotate into main	2026-06-17 20:39:13 +00:00
bot-backend	687c9e5e9d	fix(avito): proactive IP rotate at sweep start + raise max_rotations 2->4 (Datadome ban recovery)	2026-06-17 23:35:24 +03:00
lekss361	84902984e3	Merge pull request 'feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)' (#1730) from feat/activate-domclick-sweep into main	2026-06-17 20:29:43 +00:00
lekss361	07eed4b55e	Merge pull request 'feat(avito): capture newbuilding ЖК id/url from SERP card' (#1729) from feat/avito-newbuilding-capture into main	2026-06-17 20:29:39 +00:00
bot-backend	2ed3ae81bc	feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)	2026-06-17 23:21:49 +03:00
lekss361	1e3642f14c	Merge pull request 'docs(readme): refresh to Jun 2026 — Trade-In/Мера + Site Finder v2 + structure + remove debug.log' (#1728) from docs/readme-refresh-jun2026 into main ... Reviewed-on: #1728	2026-06-17 20:18:05 +00:00
lekss361	dd7b2268b7	Merge pull request 'fix(scrapers): write resolved house_id to listings.house_id_fk in realtime link hook' (#1727) from fix/house-id-fk-realtime-link into main	2026-06-17 20:16:27 +00:00
bot-backend	dd59ef0516	feat(avito): capture newbuilding ЖК id/url from SERP card; persist newbuilding_id/url	2026-06-17 23:15:35 +03:00
bot-backend	d46367da63	docs(readme): refresh to Jun 2026 — Trade-In/Мера + Site Finder v2 + structure + remove debug.log	2026-06-17 23:14:40 +03:00
bot-backend	e63b21eccb	fix(scrapers): write resolved house_id to listings.house_id_fk in realtime link hook	2026-06-17 23:07:22 +03:00
lekss361	97fce43484	Merge pull request 'feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)' (#1725) from feat/yandex-rich-fields into main ... Reviewed-on: #1725	2026-06-17 20:03:25 +00:00
lekss361	f87f3a15d3	Merge pull request 'fix(sql): ALTER TABLE IF EXISTS in migration 162 (deploy red — district_economics absent)' (#1726) from fix/migration-162-table-if-exists into main	2026-06-17 20:02:34 +00:00
lekss361	d46695ba50	fix(sql): ALTER TABLE IF EXISTS in migration 162 — district_economics absent on main DB (deploy red)	2026-06-17 20:02:06 +00:00
lekss361	df83e0256b	Merge pull request 'fix(tradein): correct SberIndex hedonic/asking dataset filter dims + 404 warning (#902)' (#1724) from fix/sber-dataset-paths-902 into main	2026-06-17 19:58:48 +00:00
bot-backend	1fe670ea6a	feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)	2026-06-17 22:58:16 +03:00
lekss361	4ebc271a6c	Merge pull request 'feat(site-finder): РИАСУРТ Свердл 14-layer harvest client+schema+task for ЕКБ agglomeration (#108)' (#1712) from feat/riasurt-sverdl-layers-108 into main	2026-06-17 19:52:58 +00:00
bot-backend	0a72ef9491	merge main + renumber 163→167 + guard riasurt harvest (#108 review)	2026-06-17 22:52:15 +03:00
bot-backend	5ca987fcd3	fix(tradein): correct SberIndex hedonic/asking dataset-paths + 404 warning (#902)	2026-06-17 22:50:52 +03:00
lekss361	f25e89767b	Merge pull request 'fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)' (#1722) from fix/conflict-resolution-freshness-1539 into main	2026-06-17 19:49:51 +00:00
bot-backend	ddf4873874	fix(riasurt): renumber migration 163→167 + guard quarterly harvest behind flag (#108 review)	2026-06-17 22:45:23 +03:00
lekss361	13db14c8c9	Merge pull request 'feat(objective): diff-gated objective_lots_history writer trigger (#307 OBJ-6)' (#1719) from feat/objective-lots-history-writer-307-obj6 into main	2026-06-17 19:44:25 +00:00
bot-backend	f30f6acbf7	fix(matching): conflict_resolution picks freshest (last_seen_at) value, deterministic (#1539)	2026-06-17 22:43:59 +03:00
bot-backend	e4cc2d9625	chore(sql): renumber objective history writer migration 164→166 (avoid collision)	2026-06-17 22:43:55 +03:00
lekss361	81e708b953	Merge pull request 'feat(sql): sales-tracker velocity + absorption MVs for Site Finder (#61)' (#1711) from feat/velocity-mv-61 into main	2026-06-17 19:43:14 +00:00
lekss361	8a24b057ac	Merge pull request 'fix(site-finder): gate velocity on confidence of objective_complex_mapping (#307 OBJ-2)' (#1720) from fix/objective-mapping-is-reviewed-gate-307-obj2 into main	2026-06-17 19:43:04 +00:00
lekss361	45d8e9e4a5	Merge pull request 'fix(cian): persist detail rich fields (description) + stamp detail_enriched_at' (#1718) from fix/cian-detail-rich-fields into main	2026-06-17 19:42:58 +00:00
lekss361	f6b3aad03d	Merge pull request 'test(etl): fix stale assert in objective_backfill dry_run test (#1709 systemic CI red)' (#1721) from fix/objective-backfill-test-1709 into main ... Reviewed-on: #1721	2026-06-17 19:40:00 +00:00
bot-backend	65b0c02b85	test(etl): fix stale assert in objective_backfill dry_run test (#1709) ... backend-tests падал СИСТЕМНО на каждом PR на test_auto_apply_matches_dry_run_no_inserts (assert 1 == 0). Root cause: аудит-коммит 14f3ef20 (#1660) намеренно изменил auto_apply_matches — dry_run теперь возвращает projected-счётчики (auto_accepted=len(auto), сколько БЫ приняли — смысл preview в admin_etl endpoint'е), но тест в том же коммите не обновили, он остался с assert auto_accepted == 0. Функция корректна (dry_run делает ранний return до db.execute/commit — execute.assert_not_called() валиден). Правлю устаревшую ассерцию: == 0 → == 1 (кандидат 0.95 >= AUTO_ACCEPT_THRESHOLD 0.85), + assert skipped == 0. Полный сьют: 3277 passed, 0 failed. Refs #1709	2026-06-17 22:38:44 +03:00