1453 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
bot-backend	9dc3be56e3	fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy ... Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or firewall interstitial aborted the entire avito_full_load sweep via AvitoBlockedError -> mark_banned. Each new connection through backconnect gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to _AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.	2026-06-18 20:41:32 +03:00
lekss361	8d516d0781	Merge pull request 'fix(tradein): avito _extract_total_count returns 0 on no-results page — stop empty-bracket split/rotate waste' (#1763) from fix/avito-empty-page-total-zero into main ... Reviewed-on: #1763	2026-06-18 17:05:34 +00:00
bot-backend	6454aa312d	fix(tradein): avito _extract_total_count returns 0 on no-results page (stop empty-bracket split waste + fallback-listing scrape)	2026-06-18 20:01:41 +03:00
lekss361	f82ee32f50	Merge pull request 'feat(tradein): avito curl_cffi 429 short-retry — enables backconnect curl SERP (×50 faster than browser)' (#1762) from feat/avito-curl-429-retry into main ... Reviewed-on: #1762	2026-06-18 16:26:28 +00:00
bot-backend	2c6d0e4c71	feat(tradein): avito curl_cffi 429 short-retry (backconnect conn-limit transient, no rotate/ban)	2026-06-18 19:24:05 +03:00
lekss361	8a7538ab13	Merge pull request 'perf(browser): block image/media/font resources in camoufox fetch (~10x smaller pages, faster via proxy)' (#1761) from perf/camoufox-block-resources into main ... Reviewed-on: #1761	2026-06-18 15:18:28 +00:00
bot-backend	1533baad97	perf(browser): block image/media/font resources in camoufox fetch (~10x smaller pages, faster via proxy)	2026-06-18 18:16:53 +03:00
lekss361	e05e81f654	Merge pull request 'perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)' (#1760) from perf/shared-price-brackets-cian-yandex into main ... Reviewed-on: #1760	2026-06-18 14:57:41 +00:00
bot-backend	ffecc6962d	perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)	2026-06-18 17:55:55 +03:00
lekss361	0538834c6b	Merge pull request 'perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)' (#1759) from perf/avito-seed-price-brackets into main ... Reviewed-on: #1759	2026-06-18 14:33:07 +00:00
bot-backend	3f85fa59ff	perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)	2026-06-18 17:29:31 +03:00
lekss361	c8f6fd1535	Merge pull request 'fix(tradein): praktika brand only under praktika account — drop kopylov mapping (brand leak)' (#1758) from fix/praktika-brand-only-praktika-user into main ... Reviewed-on: #1758	2026-06-18 13:36:20 +00:00
bot-backend	5c37fcbfad	fix(tradein): praktika brand only under praktika account — drop kopylov mapping (brand leak)	2026-06-18 16:29:18 +03:00
lekss361	add002e8d7	Merge pull request 'fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run' (#1757) from fix/avito-fullload-probe-resilience into main ... Reviewed-on: #1757	2026-06-18 13:11:46 +00:00
lekss361	5b14d88290	Merge pull request 'feat(site-finder): snapshot.pdf exclude_dev param — hide competitor by developer name' (#1756) from feat/snapshot-exclude-dev into main ... Reviewed-on: #1756	2026-06-18 13:11:29 +00:00
bot-backend	cbeeb9c414	fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run	2026-06-18 16:03:39 +03:00
bot-backend	5408466905	chore(frontend): regen api-types for snapshot exclude_dev param	2026-06-18 15:51:56 +03:00
bot-backend	1302fe63d7	feat(site-finder): snapshot.pdf exclude_dev param — hide competitor by developer name ... Optional query param exclude_dev (CSV, case-insensitive) drops competitor rows by dev_name from the «Конкуренты» block of the parcel snapshot PDF. Client case: generate a report without a specific developer's offer. Without the param behavior is unchanged (zero behavior change).	2026-06-18 15:23:28 +03:00
lekss361	493142b990	Merge pull request 'feat(tradein): populate offer_price_history for yandex from gate price.previous/trend' (#1754) from feat/yandex-price-history into main	2026-06-18 11:37:05 +00:00
lekss361	0f975e8a53	Merge pull request 'fix(tradein): avito sweep page-level retry — transient page error no longer aborts run' (#1755) from fix/avito-sweep-page-retry into main	2026-06-18 11:36:53 +00:00
bot-backend	6c132d7f0a	fix(tradein): avito sweep page-level retry — transient page error no longer aborts run	2026-06-18 14:29:01 +03:00
bot-backend	6c82528db5	feat(tradein): populate offer_price_history for yandex from gate price.previous/trend	2026-06-18 14:23:23 +03:00
lekss361	00f6464f54	Merge pull request 'feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian' (#1753) from feat/avito-exhaustive-fullload into main	2026-06-18 11:05:26 +00:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
lekss361	10b93011a3	Merge pull request 'feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only' (#1752) from feat/cian-schedule-fullload-nb-only into main	2026-06-18 10:38:31 +00:00
bot-backend	158a24c25b	feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only ... Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared kf-прокси). run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.	2026-06-18 13:25:23 +03:00
lekss361	ca4a61124e	Merge pull request 'feat(cadastral): geo-nearest building matcher via local cad mirror' (#1750) from feat/cadastral-geo-match into main ... Reviewed-on: #1750	2026-06-18 08:44:24 +00:00
lekss361	2b4f80e39f	Merge pull request 'feat(site-finder): карта точек подключения (НСПД) в секции «Сети» (#1746 part)' (#1751) from fix/1746-networks-section-map into main ... Reviewed-on: #1751	2026-06-18 08:44:16 +00:00
lekss361	48bedb5f65	Merge branch 'main' into feat/cadastral-geo-match	2026-06-18 08:44:06 +00:00
lekss361	6fe72d2374	Merge pull request 'feat(site-finder): продуктовые решения по фидбеку analyze (#1741-1745)' (#1749) from fix/sf-product-decisions into main ... Reviewed-on: #1749	2026-06-18 08:43:43 +00:00
Light1YT	e89fd66639	feat(site-finder): карта точек подключения (НСПД) в секции «Сети» (#1746, доступная часть) ... Section2 «Сети и точки подключения» был только таблицей расстояний без карты — пользователь не видел ГДЕ точка. Добавлена карта (dynamic SiteMap ssr:false + useConnectionPoints + ConnectionPointsLayer/CpLayerControlPanel), label «Точки подключения и охранные зоны сетей (НСПД)», empty-state при отсутствии дампа квартала. Таблица расстояний не тронута. Frontend-only. Полное обогащение 191-ФЗ (владелец/диаметр/мощность, единый источник истины) — остаётся за датасетом Минстроя (#1746 не закрыт целиком).	2026-06-18 12:48:23 +05:00
Light1YT	d93553c489	fix(site-finder): перенумеровать под-подписи Section3 4.1/4.2/4.3 под новый порядок + doc-хедеры (#1741 хвост)	2026-06-18 12:38:54 +05:00
bot-backend	f9769571a3	fix(cadastral): single-statement match UPDATE — OFFSET chunk skipped rows ... The chunked matcher paged the candidate set with OFFSET over the very predicate it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET (reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched listings (beyond threshold) accumulate at the low-id front, the chunk matches 0 and the loop breaks early, leaving higher-id matchable listings unfilled. Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks, missing 7 and 10. Unit tests used single-chunk fixtures so never caught it. The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the signature for schedule-param compat (now unused).	2026-06-18 10:28:19 +03:00
bot-backend	edd187b47f	feat(cadastral): geo-nearest building matcher via local cad mirror ... Backfill listings.building_cadastral_number (was 0%) from the nearest cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k listings). Instead materialize the FDW once into a LOCAL cad_buildings_local table (Point geom + GIST), then run a fast local KNN nearest-neighbour join. Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE defeated the index (58s+, full update never finished); the geometry-gate design does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end), matching 16027 listings at 50m across 2229 distinct buildings. The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches the nearest building within threshold_m (default 50m), not necessarily its exact cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels FDW not exposed). Threshold is logged. - 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW (deploy-independent); populated by the refresh job. - 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at tomorrow 09:00 UTC (after geocode_missing). - tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan), match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match run-lifecycle wrapper. - scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).	2026-06-18 10:18:54 +03:00
Light1YT	8cd0620543	feat(site-finder): продуктовые решения по фидбеку analyze (#1741 #1742 #1743 #1744 #1745) ... После ролевой дискуссии (аналитик×комдир×собственник): - #1741 секции в 2 группы «Участок»→«Стройка и рынок» (Section1→4→2→3→5→6), gate-баннер вынесен НАД группой (убран дубль из Section4Estimate), nav-группы - #1742 объяснение выбора класса: фраза-причина + мини-таблица 3 классов по deficit_index («+1 острый дефицит, −1 затоварка»). POI «обеспеченность»/market-overstock relabel — N/A (в UI не выводятся) - #1743 «Остатки и скорость»→«Как продаётся рынок рядом» + headline-вердикт + «Темп продаж»/«Будущее предложение конкурентов (24 мес)» (убран англицизм Velocity-score/Pipeline) + интерпретированные единицы - #1744 горизонт: caption+tooltip «пересчёт ~10-30с» + статус «Пересчитываем…»; колонки прогноза «…, квартир» + caveat advisory - #1745 фаза1: таблица «Прогноз по всем форматам» (студия/1-5к × индекс дефицита/прогноз спроса/сигнал); backend пробросил per-format projected_demand_units + signal. Слайдеры/рубли — фаза2 после бэктеста #951 Verify: tsc 0 ошибок; py_compile; what_to_build/recommendation/report/exporter тесты pass. Closes #1741 Closes #1742 Closes #1743 Closes #1744 Closes #1745	2026-06-18 12:18:04 +05:00
lekss361	6214baf0c9	Merge pull request 'fix(site-finder): 5 дефектов из фидбека analyze (#1736-1740)' (#1748) from fix/sf-feedback-batch into main	2026-06-18 07:06:45 +00:00
bot-backend	3280859170	Merge remote fix/sf-feedback-batch (#1736 minimap test) into local	2026-06-18 10:05:56 +03:00
bot-backend	c859872217	fix(parcels): wrap forecast-export PDF in try/except+logger (#1739 completion)	2026-06-18 10:05:25 +03:00
bot-backend	0f9babb7ff	test(site-finder): wrap MiniMap test render in QueryClientProvider (#1736) ... #1736 added useConnectionPoints (useQuery) to MiniMap, but MiniMap.test.tsx rendered without a QueryClient -> 'No QueryClient set' (2 failing tests in CI frontend-tests). Wrap renders in a QueryClientProvider (retry:false).	2026-06-18 10:03:48 +03:00
bot-backend	bcd24923b8	Merge remote-tracking branch 'forgejo-backend/main' into fix/sf-feedback-batch	2026-06-18 10:01:43 +03:00
bot-backend	e7342d9f98	chore(frontend): regenerate api-types.ts for forecast pdf export (#1739) ... #1748 added format=pdf to the forecast export endpoint but didn't regenerate the frontend OpenAPI types, failing CI openapi-codegen-check. Regenerated via the CI recipe (app.openapi() dump -> openapi-typescript -> prettier).	2026-06-18 10:00:26 +03:00
lekss361	7448db0be0	Merge pull request 'feat(avito): dedicated newbuilding (novostroyka) citywide sweep' (#1747) from feat/avito-newbuilding-sweep into main	2026-06-18 07:00:22 +00:00
Light1YT	b1c0ea1268	fix(site-finder): 5 дефектов из боевого фидбека analyze (#1736 #1737 #1738 #1739 #1740) ... - #1736 MiniMap: проброс useConnectionPoints → точки подключения на карте analyze (были только в /legacy) - #1737 confidence: пронесено имя сервиса → RU-ярлык (Рынок/Будущее предложение/…) вместо «Компонент вкладывающий сервис» - #1738 pipeline: self-exclusion субъекта (ST_DWithin 80м) — проект не считает сам себя будущим конкурентом - #1739 PDF: snapshot_pdf обёрнут в try/except+logger.exception (причина 500 видна) + format=pdf в forecast export + font_url fallback - #1740 gate↔recommendation: при can_build_mkd=False — gate_caveat на обоих рекомендаторах (противоречие явное, не молчит) Verify: py_compile 5/5, tsc 0, ruff clean, pytest confidence/forecast 95 passed. Closes #1736 Closes #1737 Closes #1738 Closes #1739 Closes #1740	2026-06-18 11:47:21 +05:00
bot-backend	1a658a65a6	feat(avito): dedicated newbuilding (novostroyka) citywide sweep	2026-06-18 09:45:54 +03:00
lekss361	bf91eb0be0	Merge pull request 'feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1)' (#1735) from feat/browser-per-source-proxy-pool into main ... Reviewed-on: #1735	2026-06-18 06:06:57 +00:00
bot-backend	b9eb478c8f	feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) ... Route each scraper source (avito/cian/yandex/domclick) to its own camoufox browser+proxy so they no longer wedge each other through a single global egress. Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the /fetch and /login paths are byte-for-byte the existing single-browser behavior. When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP (BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its own lazy-launched lock. /login stays single-browser in Phase 1. BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body; all scraper callsites pass their source. No docker-compose/.env.runtime changes (Phase 2, owner-gated).	2026-06-18 09:01:50 +03:00
lekss361	635851a71c	Merge pull request 'fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725)' (#1734) from fix/yandex-predicted-price-path-1725 into main ... Reviewed-on: #1734	2026-06-18 05:12:25 +00:00
bot-backend	41f2402b25	fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725) ... Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice, not predictedPrice.predictedPrice (that key does not exist). _predicted_prices read the wrong outer key, so predicted_price_rub/_min/_max were None for every offer (prod: pp=0 across all 6189 yandex rows). Verified live against the gate-API 2026-06-18. The rich-field test fixture used the same wrong wrapper, so the test passed vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice shape — with the old code the assertions now fail, confirming non-vacuity.	2026-06-18 08:03:05 +03:00
lekss361	8f9c11a45f	Merge pull request 'feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g' (#1733) from feat/browser-concurrent-pool into main	2026-06-17 21:07:31 +00:00
bot-backend	a6717705b9	fix(browser): reset page_counter on recycle + close login page in finally (#796 review)	2026-06-18 00:06:50 +03:00