1470 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Light1YT	8adbbef88c	fix(db): #1781 backfill listings.house_id_fk from house_sources ... The realtime FK mirror (base.py:607-616) only landed 2026-06-17 (e63b21e). Listings matched into house_sources before that — notably 9205 cian novostroyki frozen at 2026-05-31 — never had house_id_fk mirrored back, breaking the canonical "same-building via house_id_fk" estimator path (93% of active cian novostroyki had NULL fk citywide). Add data/sql/130_backfill_listings_house_id_fk.sql: deterministic source-id backfill via house_sources (ext_source, ext_id → house_id), idempotent (WHERE house_id_fk IS NULL), no fuzzy/address fallback. Prod (read-only): 9235/9235 target rows resolve deterministically, zero unresolvable. One-time backlog drain; future rows linked in realtime by the e63b21e mirror.	2026-06-19 16:20:32 +05:00
lekss361	cf6dc59cb6	Merge pull request 'fix(avito): reconnect-retry on detail 429 under backconnect proxy' (#1771) from fix/avito-detail-429-reconnect into main ... Reviewed-on: #1771	2026-06-19 04:42:11 +00:00
bot-backend	d65a88f704	fix(avito): reconnect-retry on detail 429 under backconnect proxy ... Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched because the detail fetcher only had a same-session 429 short-retry (no reconnect). 429s are persistent: keep-alive reuses one backconnect exit IP across sequential detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769): widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP) on 429 exhaustion before raising. Caller session never touched (as with 403 path).	2026-06-18 23:29:47 +03:00
lekss361	59728a197d	Merge pull request 'feat(avito): incremental daily + exhaustive weekly schedule split (+ scheduler interval_days)' (#1770) from feat/avito-incremental-schedule-split into main ... Reviewed-on: #1770	2026-06-18 20:25:39 +00:00
lekss361	0fd1d63ab3	Merge pull request 'fix(avito): tolerate 429 per-bucket instead of banning whole sweep' (#1769) from fix/avito-429-bucket-skip into main ... Reviewed-on: #1769	2026-06-18 20:25:26 +00:00
bot-backend	1f7018d415	feat(scheduler): per-source interval_days cadence (weekly exhaustive avito) ... scrape_schedules had no interval column -> every source ran daily. Add interval_days from default_params (default 1, backward-compatible) so compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive to interval_days=7 (weekly full pass for last_seen/delisting + silent price edits); avito_full_load stays daily incremental.	2026-06-18 23:10:18 +03:00
bot-backend	bf63683833	feat(avito): incremental daily + exhaustive weekly schedule split ... run_avito_full_load gains incremental_days param -> passes since= to the (merged) incremental SERP engine. avito_full_load schedule flipped to incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429 bans). New avito_full_load_exhaustive source runs the full walk weekly to refresh last_seen (10-day delisting TTL) and catch silent price edits.	2026-06-18 23:04:35 +03:00
bot-backend	bee44ffd3d	fix(avito): tolerate 429 per-bucket instead of banning whole sweep ... Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36 (deep pagination, backconnect conn-limit). Widen the 429 retry budget + one reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip a blocked bucket and continue rather than aborting the run; only N consecutive blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.	2026-06-18 23:03:28 +03:00
lekss361	b7d43db6da	Merge pull request 'fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy' (#1768) from fix/avito-detail-403-reconnect-retry into main ... Reviewed-on: #1768	2026-06-18 19:14:39 +00:00
bot-backend	c993243bc7	fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy ... fetch_detail raised on the first 403/429, aborting the detail backfill (run 191 enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active. Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy, rebuild an ephemeral session (fresh exit IP) and retry up to N before raising; short-retry 429. Never close/rebuild the caller-provided shared session. Also add the missing HTTP-200 firewall-interstitial check on the curl detail path.	2026-06-18 22:12:03 +03:00
lekss361	bf2b1fca79	Merge pull request 'feat(avito): incremental SERP pagination with date early-stop (param-gated, default off)' (#1766) from feat/avito-incremental-pagination into main ... Reviewed-on: #1766	2026-06-18 19:03:37 +00:00
lekss361	73c4a90946	Merge pull request 'feat(listings): card_hash column + snapshot-dedup on unchanged cards' (#1767) from feat/listings-card-hash into main ... Reviewed-on: #1767	2026-06-18 19:03:28 +00:00
bot-backend	8436532bf4	feat(listings): card_hash column + snapshot-dedup on unchanged cards ... Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and stored on every scrape. Skips the redundant per-day listings_snapshots write when a listing's card is unchanged. listings.last_seen_at/is_active still bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh gating (C2).	2026-06-18 22:01:54 +03:00
bot-backend	bdedac3012	feat(avito): incremental SERP pagination with date early-stop (param-gated, default off) ... Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate sequentially newest-first and stop once cards fall below the watermark, skipping deep pagination (and the 403 clusters it triggers). since=None keeps the exact exhaustive bisection walk. No scheduler/schedule change here — wiring follows in a separate PR.	2026-06-18 22:01:46 +03:00
lekss361	c58c5a5aab	Merge pull request 'fix(avito): gate 403 reconnect-retry on proxy presence only (was dead code in prod)' (#1765) from fix/avito-403-reconnect-gate into main ... Reviewed-on: #1765	2026-06-18 18:32:54 +00:00
bot-backend	44e316c514	fix(avito): gate 403 reconnect-retry on proxy presence only, not rotate_url absence ... PR #1764's reconnect-retry was dead code in prod: the gate excluded configs with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL (what curl_cffi egresses through) AND a leftover changeip URL for the separate auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403 still aborted the sweep (run 212). The curl session always uses scraper_proxy_url, so session-rebuild rotates the exit IP regardless of changeip config.	2026-06-18 21:31:20 +03:00
lekss361	69ef61beb8	Merge pull request 'fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy' (#1764) from fix/avito-403-backconnect-reconnect-retry into main ... Reviewed-on: #1764	2026-06-18 18:13:20 +00:00
bot-backend	9dc3be56e3	fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy ... Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or firewall interstitial aborted the entire avito_full_load sweep via AvitoBlockedError -> mark_banned. Each new connection through backconnect gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to _AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.	2026-06-18 20:41:32 +03:00
lekss361	8d516d0781	Merge pull request 'fix(tradein): avito _extract_total_count returns 0 on no-results page — stop empty-bracket split/rotate waste' (#1763) from fix/avito-empty-page-total-zero into main ... Reviewed-on: #1763	2026-06-18 17:05:34 +00:00
bot-backend	6454aa312d	fix(tradein): avito _extract_total_count returns 0 on no-results page (stop empty-bracket split waste + fallback-listing scrape)	2026-06-18 20:01:41 +03:00
lekss361	f82ee32f50	Merge pull request 'feat(tradein): avito curl_cffi 429 short-retry — enables backconnect curl SERP (×50 faster than browser)' (#1762) from feat/avito-curl-429-retry into main ... Reviewed-on: #1762	2026-06-18 16:26:28 +00:00
bot-backend	2c6d0e4c71	feat(tradein): avito curl_cffi 429 short-retry (backconnect conn-limit transient, no rotate/ban)	2026-06-18 19:24:05 +03:00
lekss361	8a7538ab13	Merge pull request 'perf(browser): block image/media/font resources in camoufox fetch (~10x smaller pages, faster via proxy)' (#1761) from perf/camoufox-block-resources into main ... Reviewed-on: #1761	2026-06-18 15:18:28 +00:00
bot-backend	1533baad97	perf(browser): block image/media/font resources in camoufox fetch (~10x smaller pages, faster via proxy)	2026-06-18 18:16:53 +03:00
lekss361	e05e81f654	Merge pull request 'perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)' (#1760) from perf/shared-price-brackets-cian-yandex into main ... Reviewed-on: #1760	2026-06-18 14:57:41 +00:00
bot-backend	ffecc6962d	perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)	2026-06-18 17:55:55 +03:00
lekss361	0538834c6b	Merge pull request 'perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)' (#1759) from perf/avito-seed-price-brackets into main ... Reviewed-on: #1759	2026-06-18 14:33:07 +00:00
bot-backend	3f85fa59ff	perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)	2026-06-18 17:29:31 +03:00
lekss361	c8f6fd1535	Merge pull request 'fix(tradein): praktika brand only under praktika account — drop kopylov mapping (brand leak)' (#1758) from fix/praktika-brand-only-praktika-user into main ... Reviewed-on: #1758	2026-06-18 13:36:20 +00:00
bot-backend	5c37fcbfad	fix(tradein): praktika brand only under praktika account — drop kopylov mapping (brand leak)	2026-06-18 16:29:18 +03:00
lekss361	add002e8d7	Merge pull request 'fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run' (#1757) from fix/avito-fullload-probe-resilience into main ... Reviewed-on: #1757	2026-06-18 13:11:46 +00:00
lekss361	5b14d88290	Merge pull request 'feat(site-finder): snapshot.pdf exclude_dev param — hide competitor by developer name' (#1756) from feat/snapshot-exclude-dev into main ... Reviewed-on: #1756	2026-06-18 13:11:29 +00:00
bot-backend	cbeeb9c414	fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run	2026-06-18 16:03:39 +03:00
bot-backend	5408466905	chore(frontend): regen api-types for snapshot exclude_dev param	2026-06-18 15:51:56 +03:00
bot-backend	1302fe63d7	feat(site-finder): snapshot.pdf exclude_dev param — hide competitor by developer name ... Optional query param exclude_dev (CSV, case-insensitive) drops competitor rows by dev_name from the «Конкуренты» block of the parcel snapshot PDF. Client case: generate a report without a specific developer's offer. Without the param behavior is unchanged (zero behavior change).	2026-06-18 15:23:28 +03:00
lekss361	493142b990	Merge pull request 'feat(tradein): populate offer_price_history for yandex from gate price.previous/trend' (#1754) from feat/yandex-price-history into main	2026-06-18 11:37:05 +00:00
lekss361	0f975e8a53	Merge pull request 'fix(tradein): avito sweep page-level retry — transient page error no longer aborts run' (#1755) from fix/avito-sweep-page-retry into main	2026-06-18 11:36:53 +00:00
bot-backend	6c132d7f0a	fix(tradein): avito sweep page-level retry — transient page error no longer aborts run	2026-06-18 14:29:01 +03:00
bot-backend	6c82528db5	feat(tradein): populate offer_price_history for yandex from gate price.previous/trend	2026-06-18 14:23:23 +03:00
lekss361	00f6464f54	Merge pull request 'feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian' (#1753) from feat/avito-exhaustive-fullload into main	2026-06-18 11:05:26 +00:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
lekss361	10b93011a3	Merge pull request 'feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only' (#1752) from feat/cian-schedule-fullload-nb-only into main	2026-06-18 10:38:31 +00:00
bot-backend	158a24c25b	feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only ... Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared kf-прокси). run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.	2026-06-18 13:25:23 +03:00
lekss361	ca4a61124e	Merge pull request 'feat(cadastral): geo-nearest building matcher via local cad mirror' (#1750) from feat/cadastral-geo-match into main ... Reviewed-on: #1750	2026-06-18 08:44:24 +00:00
lekss361	2b4f80e39f	Merge pull request 'feat(site-finder): карта точек подключения (НСПД) в секции «Сети» (#1746 part)' (#1751) from fix/1746-networks-section-map into main ... Reviewed-on: #1751	2026-06-18 08:44:16 +00:00
lekss361	48bedb5f65	Merge branch 'main' into feat/cadastral-geo-match	2026-06-18 08:44:06 +00:00
lekss361	6fe72d2374	Merge pull request 'feat(site-finder): продуктовые решения по фидбеку analyze (#1741-1745)' (#1749) from fix/sf-product-decisions into main ... Reviewed-on: #1749	2026-06-18 08:43:43 +00:00
Light1YT	e89fd66639	feat(site-finder): карта точек подключения (НСПД) в секции «Сети» (#1746, доступная часть) ... Section2 «Сети и точки подключения» был только таблицей расстояний без карты — пользователь не видел ГДЕ точка. Добавлена карта (dynamic SiteMap ssr:false + useConnectionPoints + ConnectionPointsLayer/CpLayerControlPanel), label «Точки подключения и охранные зоны сетей (НСПД)», empty-state при отсутствии дампа квартала. Таблица расстояний не тронута. Frontend-only. Полное обогащение 191-ФЗ (владелец/диаметр/мощность, единый источник истины) — остаётся за датасетом Минстроя (#1746 не закрыт целиком).	2026-06-18 12:48:23 +05:00
Light1YT	d93553c489	fix(site-finder): перенумеровать под-подписи Section3 4.1/4.2/4.3 под новый порядок + doc-хедеры (#1741 хвост)	2026-06-18 12:38:54 +05:00
bot-backend	f9769571a3	fix(cadastral): single-statement match UPDATE — OFFSET chunk skipped rows ... The chunked matcher paged the candidate set with OFFSET over the very predicate it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET (reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched listings (beyond threshold) accumulate at the low-id front, the chunk matches 0 and the loop breaks early, leaving higher-id matchable listings unfilled. Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks, missing 7 and 10. Unit tests used single-chunk fixtures so never caught it. The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the signature for schedule-param compat (now unused).	2026-06-18 10:28:19 +03:00