Commit graph

4 commits

Author SHA1 Message Date
e65b4d44d5 fix(avito): graceful sidecar HTTP 500/503 in browser SERP fetch (#1790)
All checks were successful
CI / changes (pull_request) Successful in 8s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
_fetch_serp_html_browser вызывал BrowserFetcher.fetch без обёртки, поэтому
httpx.HTTPStatusError от tradein-browser sidecar (500 NS_ERROR_PROXY_TOO_MANY_REQUESTS
/ Page.goto Timeout, 503 browser-unavailable) всплывал в generic except
run_avito_full_load → mark_failed + re-raise, роняя весь обход после тысяч
инкрементально сохранённых карточек (прод run #356: 3186 сохранено, один поздний
500 → весь run failed).

Классифицируем тело sidecar-ошибки и роутим в существующую rotation/retry-машинерию:
soft-ban (too_many_requests / 503 / browser unavailable) → ротация exit-IP;
transient (timeout / прочие 5xx) → bounded backoff-retry без ротации; оба бюджета
исчерпаны → AvitoRateLimitedError, чтобы run ушёл в graceful mark_banned
(partial preserved, scheduler retry next cycle), а не mark_failed.

Часть 1 issue #1790 (AVITO_PROXY_MAX_ROTATIONS=0→4) уже исправлена на проде;
это browser-resilience follow-up.

Refs #1790
2026-06-25 11:45:16 +05:00
93f7c9ae3f feat(scrapers): avito SERP дефолт на BrowserFetcher (camoufox), curl_cffi fallback (#1805)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
2026-06-19 20:42:04 +03:00
687c9e5e9d fix(avito): proactive IP rotate at sweep start + raise max_rotations 2->4 (Datadome ban recovery)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 23:35:24 +03:00
441e48141b feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Successful in 44s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 16:49:35 +00:00