lekss361/gendesign

Fork 0

[tradein][scrapers] Phase 1 — avito на BrowserFetcher (за флагом, default OFF) (#883) #901

New issue

Closed

opened 2026-05-31 13:49:02 +00:00 by bot-backend · 1 comment

bot-backend commented

2026-05-31 13:49:02 +00:00

Collaborator

Phase 1 эпика #883. Завести browser-mode путь для avito через BrowserFetcher (#884), оставив curl_cffi дефолтом — переключение per-source флагом scraper_fetch_mode, без смены поведения прода.

Контекст

Phase 0 (#884) дал BrowserFetcher + Docker. Cache-path баг пофикшен #899.
avito.py fetch-чокпоинт — _fetch_serp_html(url, page) (зовут все три entry: fetch_around/fetch_city_wide/fetch_by_rooms). curl_cffi AsyncSession в __aenter__, anti-block (_rotate_ip + _is_firewall_page) внутри _fetch_serp_html.

Задача

__aenter__: если settings.scraper_fetch_mode == "browser" → создать+войти в BrowserFetcher (self._browser); иначе как сейчас (curl). __aexit__: закрыть browser если поднят.
_fetch_serp_html: ветка — если self._browser поднят → html = await self._browser.fetch(url), прогнать через _is_firewall_page (soft-check; при firewall + остались ротации — _rotate_ip + retry, иначе AvitoBlockedError). Иначе — существующий curl-путь без изменений.
Конец пагинации в browser-mode (нет HTTP-кода как у curl return None): проверить, что callers (fetch_around/city_wide/by_rooms) уже брейкают при пустом _parse_listings. Если да — в browser-режиме _fetch_serp_html всегда возвращает html, терминирование идёт по empty-parse. Если callers НЕ брейкают на пустоту — добавить guard (parse-count==0 → stop), не ломая curl-путь.
Парсер (__preloadedState__/selectolax) НЕ трогать.

Не делать

НЕ менять дефолт scraper_fetch_mode (остаётся curl_cffi) — флип = отдельный ops-шаг post-demo.
НЕ трогать cian/yandex/n1 (Phase 2).

DoD

scraper_fetch_mode=browser → _fetch_serp_html дёргает BrowserFetcher.fetch, НЕ curl; firewall-detect работает; pagination терминируется корректно.
scraper_fetch_mode=curl_cffi (default) → путь идентичен текущему (no behavior change).
Unit-тесты (мок BrowserFetcher + мок curl): обе ветки, firewall-detect, pagination-end. uv run pytest зелёный, uv run ruff check clean.

Refs #883, #884, #899

Phase 1 эпика #883. Завести browser-mode путь для avito через `BrowserFetcher` (#884), оставив `curl_cffi` дефолтом — переключение per-source флагом `scraper_fetch_mode`, без смены поведения прода. ## Контекст - Phase 0 (#884) дал `BrowserFetcher` + Docker. Cache-path баг пофикшен #899. - avito.py fetch-чокпоинт — **`_fetch_serp_html(url, page)`** (зовут все три entry: `fetch_around`/`fetch_city_wide`/`fetch_by_rooms`). curl_cffi `AsyncSession` в `__aenter__`, anti-block (`_rotate_ip` + `_is_firewall_page`) внутри `_fetch_serp_html`. ## Задача 1. `__aenter__`: если `settings.scraper_fetch_mode == "browser"` → создать+войти в `BrowserFetcher` (`self._browser`); иначе как сейчас (curl). `__aexit__`: закрыть browser если поднят. 2. `_fetch_serp_html`: ветка — если `self._browser` поднят → `html = await self._browser.fetch(url)`, прогнать через `_is_firewall_page` (soft-check; при firewall + остались ротации — `_rotate_ip` + retry, иначе `AvitoBlockedError`). Иначе — существующий curl-путь без изменений. 3. **Конец пагинации в browser-mode** (нет HTTP-кода как у curl `return None`): проверить, что callers (`fetch_around`/`city_wide`/`by_rooms`) уже брейкают при пустом `_parse_listings`. Если да — в browser-режиме `_fetch_serp_html` всегда возвращает html, терминирование идёт по empty-parse. Если callers НЕ брейкают на пустоту — добавить guard (parse-count==0 → stop), не ломая curl-путь. 4. Парсер (`__preloadedState__`/selectolax) НЕ трогать. ## Не делать - НЕ менять дефолт `scraper_fetch_mode` (остаётся `curl_cffi`) — флип = отдельный ops-шаг post-demo. - НЕ трогать cian/yandex/n1 (Phase 2). ## DoD - [ ] `scraper_fetch_mode=browser` → `_fetch_serp_html` дёргает `BrowserFetcher.fetch`, НЕ curl; firewall-detect работает; pagination терминируется корректно. - [ ] `scraper_fetch_mode=curl_cffi` (default) → путь идентичен текущему (no behavior change). - [ ] Unit-тесты (мок BrowserFetcher + мок curl): обе ветки, firewall-detect, pagination-end. `uv run pytest` зелёный, `uv run ruff check` clean. Refs #883, #884, #899

bot-backend added the

status/wip

label 2026-05-31 14:05:01 +00:00

bot-backend self-assigned this 2026-05-31 14:05:01 +00:00

bot-backend referenced this issue

2026-05-31 14:12:41 +00:00

[tradein][scrapers][devops] Phase 0.5 — вынести camoufox в отдельный tradein-browser сервис (server/connect) (#883) #905

bot-backend referenced this issue

2026-05-31 14:30:28 +00:00

feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905) #907

bot-backend referenced this issue from a commit

2026-05-31 16:36:17 +00:00

feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901)

bot-backend referenced this issue

2026-05-31 16:36:57 +00:00

feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901) #913

bot-backend added

status/review

and removed

status/wip

labels 2026-05-31 16:37:13 +00:00

bot-backend referenced this issue from a commit

2026-05-31 16:46:54 +00:00

fixup(tradein): avito browser-mode — skip curl client alloc + rotation-false test (#901)

bot-reviewer referenced this issue from a commit

2026-05-31 16:49:37 +00:00

feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901)

bot-reviewer added

status/qa

and removed

status/review

labels 2026-05-31 16:49:46 +00:00

bot-backend commented

2026-05-31 16:55:07 +00:00

Author

Collaborator

QA handoff — smoke spec (merged #913 441e481, deployed, prod healthy).

⚠️ This is a flag-gated, dormant change: scraper_fetch_mode defaults curl_cffi, so there is no UI surface and no runtime behavior change on prod. Browser-mode only activates after a separate ops flip. So smoke scope = no regression + wiring present, not a click-flow.

Already verified by backend this session:

✅ Deploy green — tradein-backend {"status":"ok","environment":"production"}, new image live
✅ New code present — AvitoScraper.__aenter__ browser-branch confirmed in the running container
✅ pytest -k avito 172 passed, ruff clean (pre-merge)

Suggested QA checks (backend-level, no UI):

Default path unchanged — confirm settings.scraper_fetch_mode == "curl_cffi" in prod backend (no env override leaked).
Optional regression — trigger/observe a normal avito curl scrape (or last scheduled run) still parses listings.
No need to exercise browser-mode here — that's gated behind the ops flip (tracked separately).

If 1–2 hold, this is a pass.

**QA handoff — smoke spec** (merged #913 `441e481`, deployed, prod healthy). ⚠️ This is a **flag-gated, dormant** change: `scraper_fetch_mode` defaults `curl_cffi`, so there is **no UI surface and no runtime behavior change** on prod. Browser-mode only activates after a separate ops flip. So smoke scope = *no regression + wiring present*, not a click-flow. Already verified by backend this session: - ✅ Deploy green — `tradein-backend` `{"status":"ok","environment":"production"}`, new image live - ✅ New code present — `AvitoScraper.__aenter__` browser-branch confirmed in the running container - ✅ `pytest -k avito` 172 passed, ruff clean (pre-merge) Suggested QA checks (backend-level, no UI): 1. Default path unchanged — confirm `settings.scraper_fetch_mode == "curl_cffi"` in prod backend (no env override leaked). 2. Optional regression — trigger/observe a normal avito curl scrape (or last scheduled run) still parses listings. 3. No need to exercise browser-mode here — that's gated behind the ops flip (tracked separately). If 1–2 hold, this is a pass.

bot-backend referenced this issue

2026-05-31 17:23:39 +00:00

[tradein][scrapers] Завершить browser-миграцию avito: пайплайн SERP+detail+houses+IMV (#883) #915

bot-backend referenced this issue from a commit

2026-05-31 17:31:02 +00:00

feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher (#915)

bot-backend referenced this issue from a pull request that will close it,

2026-05-31 17:32:19 +00:00

feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher (#915) #916

bot-backend referenced this issue from a commit

2026-05-31 18:01:39 +00:00

fixup(tradein): real _cffi assert + module-level firewall import + SERP-routing test (#915)