fix(scraper-kit/cian): фаза houses cian_city_sweep через пул прокси + стоп на пустом пуле (#3394) #3396

Merged
bot-backend merged 2 commits from fix/3394-cian-sweep-houses-proxy-pool into main 2026-09-06 05:53:38 +00:00

2 commits

Author SHA1 Message Date
fdae823762 fix(#3394): стоп по пустому пулу — по цепочке причин на уровне якоря; один errors_count; ban_kind=infra под тестом
All checks were successful
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 9s
CI Trade-In / browser-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 4m55s
2026-09-06 10:15:43 +05:00
e12ece8fe6 fix(tradein/cian): houses-фаза cian_city_sweep идёт через пул прокси + стоп на пустом пуле (#3394)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 9s
CI Trade-In / browser-tests (pull_request) Has been skipped
CI / changes (pull_request) Successful in 10s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 4m59s
`run_cian_city_sweep` звал `fetch_newbuilding(zhk_url, config=config)` без
`proxy_provider`, хотя провайдер лежит в аргументах самого свипа и соседние
фазы (SERP через CianScraper, detail через cian_fetch_detail) его передают.
Внутри это давало `build_browser_fetcher(config, "cian", proxy_provider=None)`
→ `use_pool` эффективно False → POST /fetch без "proxy" → сайдкар брал свой
env-узел SCRAPER_PROXY_URL, на проде выключенный (407 → camoufox InvalidIP →
/fetch 503, факт #3386). Фаза houses давала 0/30 с 02.09 (run 6179: 24 ×
`houses failed ... 503 Service Unavailable`), строк `override=True` в логах
сайдкара по ней не было ни одной. Остальные вызывающие `fetch_newbuilding` /
`resolve_cian_zhk_url_via_search` провайдер уже передают (#2767/#2830/#3382),
этот вызов был последним мимо пула.

Второе: пустой пул поднимается ДО запроса, следующий дом упрётся ровно в то
же самое — общий `except Exception` на дом превращал это в 30 одинаковых
houses_failed и прогон уходил в 'done'. Теперь NoProxyAvailableError рвёт
фазу и свип: `no_proxy_stop=1` в counters, mark_banned с ban_kind='infra' и
сохранённым done_buckets (образец — #3389 yandex-nb-sweep, #3382). Ветка
стоит ДО generic-except, иначе наш отказ инфраструктуры читался бы как
«IP likely blocked» — бан площадки.
2026-09-06 09:45:56 +05:00