fix(avito): post-merge polish — unify fetch_mode guard + empty-render guard (#1811)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped

Косметические минор'ы из ревью PR #1809 (avito SERP → BrowserFetcher),
функционально без изменений:

- __aenter__: settings.scraper_fetch_mode != "curl_cffi" → == "browser"
  (унификация с scrape_pipeline.py и avito_detail_backfill.py; consistency
  при гипотетическом 3-м значении режима).
- _fetch_serp_html_browser: явный guard `if not html: return None` — пустой
  рендер (таймаут sidecar'а) больше не утекает как «валидный» HTML; caller
  (fetch_around) трактует None как конец пагинации.
- scrape_pipeline.py: logger.debug на shared-browser sweep-путях — cffi-fallback
  пропускается (by design), логируем для observability при firewall.
- test_avito_browser_mode.py: переформулирован вводящий в заблуждение коммент
  про _cffi is None (direct _browser assignment bypasses __aenter__).

Refs #1811
This commit is contained in:
Light1YT 2026-06-26 12:20:13 +05:00
parent 67d41df880
commit 9b9cc99c11
3 changed files with 17 additions and 2 deletions

View file

@ -757,6 +757,11 @@ async def run_avito_city_sweep(
scraper = AvitoScraper()
if browser_mode:
scraper._browser = shared_bf
# Shared-browser режим: _cffi=None → curl_cffi-fallback на
# firewall пропускается (by design). Логируем для observability.
logger.debug(
"avito pipeline-mode: shared browser only, no cffi fallback"
)
else:
scraper._cffi = session
@ -1324,6 +1329,9 @@ async def run_avito_newbuilding_sweep(
scraper = AvitoScraper()
if browser_mode:
scraper._browser = shared_bf
# Shared-browser режим: _cffi=None → curl_cffi-fallback на firewall
# пропускается (by design). Логируем для observability.
logger.debug("avito pipeline-mode: shared browser only, no cffi fallback")
else:
scraper._cffi = session

View file

@ -352,7 +352,7 @@ class AvitoScraper(BaseScraper):
logger.warning(
"avito: proactive IP rotate at sweep start raised — proceeding", exc_info=True
)
if settings.scraper_fetch_mode != "curl_cffi":
if settings.scraper_fetch_mode == "browser":
# Default (#1805): SERP-фетч через camoufox (tradein-browser per-provider
# + ротирующий backconnect-прокси). curl_cffi поднимаем тут же как
# FALLBACK — если браузер отдаёт firewall/пусто, _fetch_serp_html делает
@ -546,6 +546,12 @@ class AvitoScraper(BaseScraper):
raise AvitoRateLimitedError(
f"avito SERP browser-sidecar error (page={page}): {error_text[:200]}"
) from exc
# Пустой рендер ("" / None) = таймаут рендера sidecar'а, НЕ firewall:
# _is_firewall_page("") вернул бы False → пустой HTML утёк бы как
# «валидный». Возвращаем None — caller (fetch_around) трактует это как
# конец пагинации (break), а не как бан.
if not html:
return None
if not _is_firewall_page(html):
return html
if rot_done < max_rot and await self._rotate_ip():

View file

@ -175,7 +175,8 @@ async def test_fetch_serp_html_browser_mode_returns_html() -> None:
assert result == _NORMAL_HTML
mock_browser.fetch.assert_awaited_once_with("https://www.avito.ru/test")
# _cffi was never set — assert remains None
# _cffi not armed (direct _browser assignment bypasses __aenter__, which would
# otherwise build the curl_cffi fallback session); the browser path never touches it.
assert scraper._cffi is None