3 commits

Author	SHA1	Message	Date
bot-backend	7800f6014a	refactor(tradein/scraper-kit): yandex providers on shared _base.py (#2363) ... F4c-yandex (epic #2277 Group F, parent #2352): migrate the SERP scraper's BrowserFetcher construction onto build_browser_fetcher() from the Foundation module (#2358, providers/_base.py) instead of local ad-hoc construction. - yandex/serp.py::__aenter__ now calls build_browser_fetcher(config, source=, proxy_provider=, fetch_timeout_s=) instead of constructing BrowserFetcher(...) directly. Preserves the existing fetch_timeout_s=30 (see caveat below), endpoint/proxy_provider/use_pool resolution byte-for-byte. The retry/tarpit logic in fetch_around() (rotate_ip + sleep on status==0/JSON-error) is untouched, per the Foundation module's documented exclusion. Left unchanged (deliberate, not an oversight): - yandex/serp.py::_rotate_ip's curl_cffi AsyncSession(timeout=30) — this hits the mobile-proxy provider's own changeip API, not Yandex; it intentionally carries no impersonate/headers/proxy (same as the mirrored avito/serp.py rotate-ip session). Not listed among the _base.py docstring's documented extraction sites -- consistent exclusion, not a new one. - yandex/detail.py -- no BrowserFetcher/curl_cffi construction exists in this file (uses BaseScraper's httpx-based _http_get). Nothing to migrate. - yandex/newbuilding.py -- both BrowserFetcher(...) call sites take an OPTIONAL `config: ScraperConfig | None`, a genuinely different contract from build_browser_fetcher's mandatory `config`. Migrating would require making config mandatory (breaking admin.py:1511, yandex_newbuilding_sweep.py:353, and several tests that call without config -- out of scope for a call-site swap) and breaks test_scraper_kit_newbuilding_endpoint.py's "without config, endpoint=None" regression case (its SimpleNamespace mock lacks use_proxy_pool_browser, and the mock config path can never satisfy a mandatory-config helper). Left untouched; the class-bug #2322/#2330 this file already fixed (config now threaded, just not via the shared helper) remains fixed. Also updates test_kit_serp_proxy_pool.py's 3 yandex-specific tests: they monkeypatched the module-local `yandex_serp.BrowserFetcher` name, which the migrated call path no longer references directly (it now goes through scraper_kit.providers._base.build_browser_fetcher -> _base's own BrowserFetcher import). Patch target moved to scraper_kit.providers._base.BrowserFetcher; every assertion is unchanged -- this adapts the mock to the new (intentional) call path, it does not weaken what the test verifies. Cian's tests in the same file are untouched (cian/serp.py is not migrated yet, separate F4b issue).	2026-07-04 09:38:43 +03:00
bot-backend	96dd27c4ae	refactor(tradein/scraper-kit): cian providers on shared _base.py (#2362) ... Мигрирует call-site'ы cian/{serp,detail,newbuilding,session}.py на shared фабрики Foundation-модуля (#2358): build_browser_fetcher() (serp.py __aenter__) и build_curl_cffi_session() (detail.py own-session path, newbuilding.py resolve_cian_zhk_url[_via_search], session.py verify_session). Headers/timeout/ proxy-resolution сохранены байт-в-байт — чистый call-site swap, без изменения поведения. Caveat (config Optional footgun): newbuilding.py::fetch_newbuilding() НЕ переведена на build_browser_fetcher() — build_browser_fetcher() требует ScraperConfig mandatory (весь смысл фабрики, закрывает класс багов #2322/#2330), но здесь config: ScraperConfig | None = None легитимно Optional (admin.py debug-роут и test_scraper_kit_newbuilding_endpoint.py вызывают функцию без config= и ожидают graceful endpoint=None). Подмена уронила бы AttributeError на config=None вместо текущего degrade — реальная регрессия, не call-site swap. Оставлено как есть с комментарием в коде. Companion test updates (не ослабление, retarget под рефактор): CianScraper теперь строит BrowserFetcher внутри _base.py, а не в serp.py — два теста в test_kit_serp_proxy_pool.py, патчившие cian_serp.BrowserFetcher напрямую, переключены на scraper_kit.providers._base.BrowserFetcher (тот же инвариант, другая точка патча). Аналогично test_scraper_kit_pricehistory_session_parity.py ::test_fetch_detail_own_session_proxy_wiring_parity — патч AsyncSession переключён с cian.detail на _base (тот же assert на итоговый proxies=dict). Tests: full backend suite 3278 passed, 1 pre-existing flake (test_search_api.py::test_search_cache_hit, #2208), 6 skipped. Все parity/regression тесты (test_scraper_kit_pipeline_parity2.py, test_scraper_kit_newbuilding_endpoint.py, test_scraper_kit_cian_golden_parity.py, test_kit_serp_proxy_pool.py, test_scraper_kit_pricehistory_session_parity.py, test_cian_serp_scraper.py, test_cian_session.py, test_cian_detail.py) зелёные. ruff check + format --check чисто на всех изменённых файлах.	2026-07-04 09:29:49 +03:00
lekss361	e1c5f2184c	fix(tradein/scraper-kit): wire browser proxy-pool into yandex/cian SERP + persist activation flags (#2160) (#2268)	2026-07-03 06:36:56 +00:00