1505 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
bot-backend	8467cc647f	test(browser): remove obsolete pool-routing tests (FEATURE_BROWSER_POOL_ENABLED/_build_proxy_map removed in #1793 per-provider rewrite; routing covered by browser/test_server.py)	2026-06-19 20:12:51 +03:00
lekss361	2e3291abc9	Merge pull request 'fix(scrapers): cian_detail — ceiling из offer.building.ceilingHeight, repair из decoration (#1791)' (#1800) from feat/cian-detail-field-mapping into main ... Reviewed-on: #1800	2026-06-19 17:07:14 +00:00
lekss361	95069a3daf	Merge pull request 'fix(scrapers): yandex_detail — area/ceiling/kitchen из INITIAL_STATE, покрытие area 63%→~95% (#1792)' (#1799) from feat/yandex-detail-field-coverage into main ... Reviewed-on: #1799	2026-06-19 17:07:02 +00:00
bot-backend	2d3ca78b0b	fix(scrapers): cian_detail — ceiling/repair актуальные пути state (#1791) ... ceiling_height на detail-странице лежит в offer.building.ceilingHeight, а не offer.ceilingHeight (всегда null) → было 0% покрытия. repairType часто пустой, структурное значение приходит как offer.decoration ('preFine'/...) — читаем оба + добавлен preFine->needs_repair и case-insensitive lookup в нормализаторе. kitchen/description пути уже верны (offer.kitchenArea/offer.description) — низкое покрытие = мёртвый CIAN_PROXY_URL, не маппинг. Регресс-тест на реальной карточке 330982715.	2026-06-19 19:57:18 +03:00
lekss361	7c9a3cf09d	Merge pull request 'fix(tradein-browser): camoufox hardening — block_images/webrtc + форс МСК tz (#1796)' (#1798) from fix/tradein-browser-camoufox-hardening into main ... Reviewed-on: #1798	2026-06-19 16:50:24 +00:00
bot-backend	b5f439a095	fix(tradein-browser): camoufox hardening — block_images/webrtc + forced MSK tz ... Со всеми знаниями из ресёрча camoufox (#1796 Avito anti-bot): - block_images=True — быстрее грузит (HTML/JSON, без рендера картинок). - block_webrtc=True — закрывает WebRTC-leak реального IP мимо прокси. - config timezone=Europe/Moscow — форсим +3 МСК: geoip+БэкКоннект (ротация IP на каждый запрос) давал геолик tz↔IP (Ульяновск UTC+4 при выходе МСК/СПб).	2026-06-19 19:49:14 +03:00
bot-backend	0084efc12a	fix(scrapers): yandex_detail — area/ceiling из структурного блока, покрытие area 63%->~95% (#1792)	2026-06-19 19:37:51 +03:00
lekss361	059fa83589	Merge pull request 'fix(scrapers): avito_houses — отзывы/история/тип ЖК долетают (виджет-payload вложен глубже) (#1789)' (#1797) from feat/avito-houses-mfe-state into main ... Reviewed-on: #1797	2026-06-19 16:27:03 +00:00
bot-backend	464e9faf7b	fix(scrapers): avito_houses читает MFE props-вложенность виджетов — отзывы/история/тип ЖК теперь долетают (#1789)	2026-06-19 19:21:49 +03:00
lekss361	4755f8bd13	Merge pull request 'feat(browser): per-provider concurrency — параллель между поставщиками, ≤1 коннект на поставщика (#1793)' (#1794) from feat/browser-per-provider-concurrency into main ... Reviewed-on: #1794	2026-06-19 16:02:13 +00:00
lekss361	fb7b3323e4	Merge pull request 'fix(tradein): verify=False для WFS геопортала ЕКБ (российский CA-серт)' (#1788) from fix/tradein-ekb-geoportal-ssl-verify into main ... Reviewed-on: #1788	2026-06-19 16:02:04 +00:00
bot-backend	1578dd5a5d	fixup(browser): source-fallback routing, lazy launch, test stability (#1793 review)	2026-06-19 19:00:10 +03:00
bot-backend	77ae4ca779	feat(browser): per-provider concurrency — параллель между поставщиками, ≤1 коннект на поставщика (#1793)	2026-06-19 18:52:55 +03:00
bot-backend	112e5cc2f1	fix(tradein): verify=False for EKB geoportal WFS (Russian-CA cert) ... The geoportal serves a Russian national-CA (Минцифры) TLS cert absent from httpx's default trust store → CERTIFICATE_VERIFY_FAILED, loader fetched 0. Public read-only data → verify=False (same as the curl -k used in recon).	2026-06-19 18:19:03 +03:00
lekss361	7494255180	Merge pull request 'fix(tradein): чинит main-red — патч _geoportal_house_match в cadastral-тесте (хвост #1786)' (#1787) from fix/tradein-geoportal-test-cadastral into main ... Reviewed-on: #1787	2026-06-19 15:07:31 +00:00
bot-backend	91b04a1851	test(tradein): patch _geoportal_house_match in cadastral-before-yandex test ... #1786 merged before the test fix landed → main red on test_geocode_uses_cadastral_before_yandex. The new geoportal house-match tier runs first in geocode(); this legacy cadastral test must mock it to None so it exercises the cadastral forward path (else the MagicMock db row leaks through the new tier as lat=1.0).	2026-06-19 18:06:28 +03:00
lekss361	7bf39e24df	Merge pull request 'fix(db): #1781 backfill listings.house_id_fk from house_sources (cian novostroyki)' (#1784) from fix/1781-cian-novostroyki-fk-backfill into main ... Reviewed-on: #1784	2026-06-19 14:56:09 +00:00
lekss361	84ceb9d300	Merge pull request 'feat(tradein): ингест зданий из геопортала ЕКБ + tier в геокодере (полный кадастр города)' (#1786) from feat/tradein-ekb-geoportal-buildings into main ... Reviewed-on: #1786	2026-06-19 14:55:43 +00:00
bot-backend	64d113d649	feat(tradein): EKB geoportal building ingest + geocoder tier (complete city buildings)	2026-06-19 17:51:07 +03:00
Light1YT	8adbbef88c	fix(db): #1781 backfill listings.house_id_fk from house_sources ... The realtime FK mirror (base.py:607-616) only landed 2026-06-17 (e63b21e). Listings matched into house_sources before that — notably 9205 cian novostroyki frozen at 2026-05-31 — never had house_id_fk mirrored back, breaking the canonical "same-building via house_id_fk" estimator path (93% of active cian novostroyki had NULL fk citywide). Add data/sql/130_backfill_listings_house_id_fk.sql: deterministic source-id backfill via house_sources (ext_source, ext_id → house_id), idempotent (WHERE house_id_fk IS NULL), no fuzzy/address fallback. Prod (read-only): 9235/9235 target rows resolve deterministically, zero unresolvable. One-time backlog drain; future rows linked in realtime by the e63b21e mirror.	2026-06-19 16:20:32 +05:00
lekss361	190accc4a7	Merge pull request 'feat(tradein): фронт передаёт координаты выбранной подсказки в /estimate (Variant A, frontend)' (#1783) from feat/tradein-estimate-send-coords into main ... Reviewed-on: #1783	2026-06-19 11:17:31 +00:00
lekss361	9701d3289d	Merge pull request 'feat(tradein): /estimate принимает координаты от автокомплита, минует geocode (Variant A, backend)' (#1782) from feat/tradein-estimate-accept-coords into main ... Reviewed-on: #1782	2026-06-19 11:16:51 +00:00
bot-backend	ce0aaa4fb1	feat(tradein): send picked autocomplete coords to /estimate (Variant A)	2026-06-19 14:16:08 +03:00
bot-backend	99e2a5d75c	feat(tradein): estimate accepts client-provided coords, skips geocode (Variant A)	2026-06-19 14:15:02 +03:00
lekss361	41b3f08303	Merge pull request 'fix(tradein): чинит кадастр-матчер геокодера — резолв адресов из своей базы (~60% упавших)' (#1780) from fix/tradein-cadastral-geocode-matcher into main ... Reviewed-on: #1780	2026-06-19 11:11:13 +00:00
bot-backend	c2a1dc8aeb	fix(tradein): parse street+house for cadastral geocode match (resolve ~60% of failing addresses from local DB)	2026-06-19 14:04:12 +03:00
lekss361	be3a5df8eb	Merge pull request 'fix(tradein): удалить white-label бренд «Практика» целиком (БД + маппинг + ассеты)' (#1779) from fix/tradein-remove-praktika-brand into main ... Reviewed-on: #1779	2026-06-19 10:09:38 +00:00
bot-backend	ea453c3ff2	fix(tradein): remove «Практика» white-label brand (DB row + account mapping + assets)	2026-06-19 12:53:37 +03:00
lekss361	374238daac	Merge pull request 'fix(tradein): budget Cian valuation call to stop 25s estimate stalls («не считает»)' (#1777) from fix/tradein-estimate-latency-budgets into main ... Reviewed-on: #1777	2026-06-19 09:48:41 +00:00
lekss361	54444cc4a5	Merge pull request 'fix(tradein): убрать хардкод «Практика» из offer-баннера' (#1778) from fix/tradein-offercard-praktika-leak into main ... Reviewed-on: #1778	2026-06-19 09:48:35 +00:00
bot-backend	80380dc32a	fix(tradein): budget Cian valuation call to stop 25s estimate stalls ... The Cian Valuation enrichment was the only ungated external call in estimate_quality() — its internal curl timeout (~25s) blocks the whole /estimate request on every cache-miss, intermittently pushing latency past the gateway/patience window (perceived as 'сервис не считает'). Wrap it in _with_budget() (mirrors the existing Yandex-valuation guard): on timeout it degrades to None — the same graceful path as a network error — instead of stalling. New setting estimate_cian_valuation_timeout_s (default 8.0s, env ESTIMATE_CIAN_VALUATION_TIMEOUT_S).	2026-06-19 12:43:28 +03:00
bot-backend	c5d469da36	fix(tradein): remove hardcoded «Практика» from offer banner + comments ... Visible warn-banner trailing sentence no longer names the brand. Code comments in the file docstring updated to be brand-neutral. No logic or behaviour changed.	2026-06-19 12:41:27 +03:00
lekss361	ee2703419b	Merge pull request 'fix(estimator): #1774 Tier A admits same-building novostroyki resales (secondary-dominant gate)' (#1776) from fix/1774-tier-a-same-building-primary into main ... Reviewed-on: #1776	2026-06-19 09:28:16 +00:00
lekss361	19f740f2e9	Merge pull request 'fix(avito): de-glue house number from district in _clean_address (#1773 part)' (#1775) from fix/1773-avito-address-deglue into main ... Reviewed-on: #1775	2026-06-19 09:26:26 +00:00
Light1YT	07d9cc8e8b	fix(estimator): #1774 Tier A admits same-building novostroyki resales when secondary-dominant ... Tier A (same building) dropped cian listings tagged listing_segment='novostroyki' even in delivered buildings where that tag marks owner resales/переуступки (sale_type=free). Gated relaxation of the #1186 guard, Tier A ONLY: include novostroyki rows iff the same house has secondary (vtorichka/NULL) listings AND secondary_count >= primary_count — secondary-dominant signal of a delivered house. Prevents a lone vtorichka + clustered developer-priced novostroyki from anchoring on застройщик prices after MAD-clip drops the rare secondary (~27% overvaluation). Pure-primary / primary-dominated houses keep the #1186 guard. Behind estimate_sb_tier_a_allow_primary_if_secondary_present (default True). Dedup duplicate active rows by (source, source_id) primary key (codebase canon, namespaced id/url tags, source_url fallback) — collapses rows whose source_url differs only by trailing slash / query-param (e.g. cian source_id=330047129 ×2). Tier C / radius / asking_to_sold_ratio paths untouched. 84 unit tests pass.	2026-06-19 14:00:13 +05:00
Light1YT	78e7705a5f	fix(avito): de-glue house number from district marker in _clean_address (#1773) ... Avito leaks addresses with house number glued to district marker without separator («…Савкова, 29р-н Академический»), breaking house-number extraction and same-house matching (41 unmatched avito listings on one street). _clean_address now inserts ", " between a house-number token (digits + optional Cyrillic letter + optional /N|сN|кN|корпус|стр) and a following marker (р-н, мкр, район, г., пос, снт, …). - avito.py: idempotent _deglue_house_marker step (CSS/tail-strip kept) - tests: de-glue positive/negative/idempotency (24 passed) - data/sql/124_deglue_avito_addresses.sql: idempotent backfill for stored rows normalize.py НЕ тронут (общий fingerprint матчинга, #1534). Закрывает avito-часть #1773 (cian-без-номера + дедуп — отдельно, зона cian).	2026-06-19 13:33:43 +05:00
lekss361	cf6dc59cb6	Merge pull request 'fix(avito): reconnect-retry on detail 429 under backconnect proxy' (#1771) from fix/avito-detail-429-reconnect into main ... Reviewed-on: #1771	2026-06-19 04:42:11 +00:00
bot-backend	d65a88f704	fix(avito): reconnect-retry on detail 429 under backconnect proxy ... Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched because the detail fetcher only had a same-session 429 short-retry (no reconnect). 429s are persistent: keep-alive reuses one backconnect exit IP across sequential detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769): widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP) on 429 exhaustion before raising. Caller session never touched (as with 403 path).	2026-06-18 23:29:47 +03:00
lekss361	59728a197d	Merge pull request 'feat(avito): incremental daily + exhaustive weekly schedule split (+ scheduler interval_days)' (#1770) from feat/avito-incremental-schedule-split into main ... Reviewed-on: #1770	2026-06-18 20:25:39 +00:00
lekss361	0fd1d63ab3	Merge pull request 'fix(avito): tolerate 429 per-bucket instead of banning whole sweep' (#1769) from fix/avito-429-bucket-skip into main ... Reviewed-on: #1769	2026-06-18 20:25:26 +00:00
bot-backend	1f7018d415	feat(scheduler): per-source interval_days cadence (weekly exhaustive avito) ... scrape_schedules had no interval column -> every source ran daily. Add interval_days from default_params (default 1, backward-compatible) so compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive to interval_days=7 (weekly full pass for last_seen/delisting + silent price edits); avito_full_load stays daily incremental.	2026-06-18 23:10:18 +03:00
bot-backend	bf63683833	feat(avito): incremental daily + exhaustive weekly schedule split ... run_avito_full_load gains incremental_days param -> passes since= to the (merged) incremental SERP engine. avito_full_load schedule flipped to incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429 bans). New avito_full_load_exhaustive source runs the full walk weekly to refresh last_seen (10-day delisting TTL) and catch silent price edits.	2026-06-18 23:04:35 +03:00
bot-backend	bee44ffd3d	fix(avito): tolerate 429 per-bucket instead of banning whole sweep ... Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36 (deep pagination, backconnect conn-limit). Widen the 429 retry budget + one reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip a blocked bucket and continue rather than aborting the run; only N consecutive blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.	2026-06-18 23:03:28 +03:00
lekss361	b7d43db6da	Merge pull request 'fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy' (#1768) from fix/avito-detail-403-reconnect-retry into main ... Reviewed-on: #1768	2026-06-18 19:14:39 +00:00
bot-backend	c993243bc7	fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy ... fetch_detail raised on the first 403/429, aborting the detail backfill (run 191 enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active. Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy, rebuild an ephemeral session (fresh exit IP) and retry up to N before raising; short-retry 429. Never close/rebuild the caller-provided shared session. Also add the missing HTTP-200 firewall-interstitial check on the curl detail path.	2026-06-18 22:12:03 +03:00
lekss361	bf2b1fca79	Merge pull request 'feat(avito): incremental SERP pagination with date early-stop (param-gated, default off)' (#1766) from feat/avito-incremental-pagination into main ... Reviewed-on: #1766	2026-06-18 19:03:37 +00:00
lekss361	73c4a90946	Merge pull request 'feat(listings): card_hash column + snapshot-dedup on unchanged cards' (#1767) from feat/listings-card-hash into main ... Reviewed-on: #1767	2026-06-18 19:03:28 +00:00
bot-backend	8436532bf4	feat(listings): card_hash column + snapshot-dedup on unchanged cards ... Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and stored on every scrape. Skips the redundant per-day listings_snapshots write when a listing's card is unchanged. listings.last_seen_at/is_active still bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh gating (C2).	2026-06-18 22:01:54 +03:00
bot-backend	bdedac3012	feat(avito): incremental SERP pagination with date early-stop (param-gated, default off) ... Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate sequentially newest-first and stop once cards fall below the watermark, skipping deep pagination (and the 403 clusters it triggers). since=None keeps the exact exhaustive bisection walk. No scheduler/schedule change here — wiring follows in a separate PR.	2026-06-18 22:01:46 +03:00
lekss361	c58c5a5aab	Merge pull request 'fix(avito): gate 403 reconnect-retry on proxy presence only (was dead code in prod)' (#1765) from fix/avito-403-reconnect-gate into main ... Reviewed-on: #1765	2026-06-18 18:32:54 +00:00