lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions

feat(tradein/scrapers): detail_backfill через curl_cffi+backconnect вместо браузера/auv (развязка прокси) #1838

Merged
lekss361 merged 1 commit from feat/tradein-detail-backfill-curl into main 2026-06-20 12:59:56 +00:00
Conversation 0 Commits 1 Files changed 3 +119 -5

1 commit

Author SHA1 Message Date
bot-backend
cc3ff7c3d2 feat(tradein/scrapers): detail_backfill через curl_cffi+backconnect вместо браузера/auv (развязка прокси)
All checks were successful
CI / changes (pull_request) Successful in 7s
Details
CI / backend-tests (pull_request) Has been skipped
Details
CI / frontend-tests (pull_request) Has been skipped
Details
CI / openapi-codegen-check (pull_request) Has been skipped
Details 
SERP (full_load/city_sweep) и detail_backfill делили один прокси-аккаунт auv (~5 коннектов).
Browser-фетч в backfill открывал десятки CONNECT-туннелей → cap превышался → HTTP 500/краши.

- config.py: avito_detail_backfill_use_curl=True (ENV AVITO_DETAIL_BACKFILL_USE_CURL)
- avito_detail_backfill.py: browser_mode гейтится флагом; use_curl=True → session=None,
  browser_fetcher=None → fetch_detail строит _build_detail_session() через scraper_proxy_url
  (backconnect mproxy, авто-ротация, 1 коннект/запрос)
- tests: 2 новых теста (use_curl=True → BrowserFetcher не создаётся; False → создаётся)
 2026-06-20 15:55:24 +03:00