fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url #860

Merged
bot-reviewer merged 1 commit from fix/tradein-route-scrapers-through-proxy into main 2026-05-31 07:04:31 +00:00

1 commit

Author SHA1 Message Date
a07fe55db9 fix(tradein): route yandex/valuation/imv/newbuilding scrapers through scraper_proxy_url
Yandex SERP was being banned (datacenter IP) and on-demand scrapers (cian_valuation,
cian_newbuilding, avito_imv, avito_detail) silently returned None due to same cause.
Mobile proxy was already wired for cian.py/cian_detail.py/avito.py but missing elsewhere.

- yandex_realty: proxies= added to _CurlCffiSession in __aenter__
- yandex_valuation: proxies= added to _CurlCffiSession in __aenter__
- cian_valuation: proxies= added to AsyncSession (async with ... fetch block)
- cian_newbuilding: proxies= added to both own-session paths (fetch_newbuilding +
  resolve_cian_zhk_url); shared-session callers unaffected
- avito_imv: proxies= added to own-session path only (shared CffiAsyncSession from
  AvitoScraper already carries proxy)
- avito_detail: proxies= added to own-session path only (scrape_pipeline passes shared
  session with proxy; admin endpoint creates own-session -> now proxied)

All use pattern: _proxy = settings.scraper_proxy_url; proxies={"http":_proxy,"https":_proxy}
if _proxy else None -- matching cian.py (#806 reference). proxy=None = direct connection (dev).

Tests: 9 new proxy-wiring unit tests added to test_scraper_proxy.py (20 total, all pass).
2026-05-31 09:57:17 +03:00