lekss361/gendesign

Fork 0

feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901) #913

Merged

bot-reviewer merged 2 commits from feat/901-avito-browser-mode into main

2026-05-31 16:49:36 +00:00

bot-backend commented

2026-05-31 16:36:57 +00:00

Collaborator

Phase 1 of epic #883. Wires AvitoScraper to optionally fetch SERP HTML via BrowserFetcher (camoufox, #884) behind the existing scraper_fetch_mode flag. curl_cffi stays the default — zero prod behavior change (flip is a separate ops step post-demo).

Changes (`avito.py`)

import BrowserFetcher; self._browser: BrowserFetcher | None = None in __init__.
__aenter__: if scraper_fetch_mode == "browser" → create+enter BrowserFetcher, return early (skips curl setup). Else curl_cffi path unchanged.
__aexit__: tear down browser if up, then curl close.
_fetch_serp_html: browser branch before assert self._cffi is not None — fetch via BrowserFetcher.fetch(url), soft firewall-detect (_is_firewall_page) + _rotate_ip retry up to avito_proxy_max_rotations, else AvitoBlockedError. curl path below untouched.
Pagination-end needs no new guard: all three callers (fetch_around/fetch_city_wide/fetch_by_rooms) already break on empty-parse; browser-mode always returns html or raises.

Validation

uv run ruff check — clean.
New tests/test_avito_browser_mode.py — 6 passed (browser branch chosen, curl branch unchanged, firewall→AvitoBlockedError, pagination terminates via empty-parse).
Full avito regression — 171 passed.

Verified live (prod smoke, this session)

tradein-browser /health ok; /fetch avito SERP → 5.8 MB hydrated HTML, preloadedState present, firewall-container 0 — anti-bot bypassed.

Out of scope

Default scraper_fetch_mode stays curl_cffi (flip = ops step).
cian/yandex/n1 = Phase 2 (#883/#886).

Refs #883, #884, #899

Phase 1 of epic #883. Wires `AvitoScraper` to optionally fetch SERP HTML via `BrowserFetcher` (camoufox, #884) behind the existing `scraper_fetch_mode` flag. **curl_cffi stays the default — zero prod behavior change** (flip is a separate ops step post-demo). ## Changes (`avito.py`) - `import BrowserFetcher`; `self._browser: BrowserFetcher | None = None` in `__init__`. - `__aenter__`: if `scraper_fetch_mode == "browser"` → create+enter `BrowserFetcher`, return early (skips curl setup). Else curl_cffi path unchanged. - `__aexit__`: tear down browser if up, then curl close. - `_fetch_serp_html`: browser branch **before** `assert self._cffi is not None` — fetch via `BrowserFetcher.fetch(url)`, soft firewall-detect (`_is_firewall_page`) + `_rotate_ip` retry up to `avito_proxy_max_rotations`, else `AvitoBlockedError`. curl path below untouched. - Pagination-end needs no new guard: all three callers (`fetch_around`/`fetch_city_wide`/`fetch_by_rooms`) already `break` on empty-parse; browser-mode always returns html or raises. ## Validation - `uv run ruff check` — clean. - New `tests/test_avito_browser_mode.py` — **6 passed** (browser branch chosen, curl branch unchanged, firewall→`AvitoBlockedError`, pagination terminates via empty-parse). - Full avito regression — **171 passed**. ## Verified live (prod smoke, this session) tradein-browser `/health` ok; `/fetch` avito SERP → 5.8 MB hydrated HTML, `preloadedState` present, firewall-container 0 — anti-bot bypassed. ## Out of scope - Default `scraper_fetch_mode` stays `curl_cffi` (flip = ops step). - cian/yandex/n1 = Phase 2 (#883/#886). Refs #883, #884, #899

bot-backend added 1 commit 2026-05-31 16:36:57 +00:00

feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901 ) 1eb241da34

Phase 1 of #883. Browser-mode path in AvitoScraper: __aenter__/__aexit__
manage a BrowserFetcher when scraper_fetch_mode=="browser"; _fetch_serp_html
fetches via browser with soft firewall-detect + IP-rotate. curl_cffi remains
the default — no prod behavior change. Pagination terminates via empty-parse.

Refs #883, #884

bot-backend added the

status/review

label 2026-05-31 16:37:12 +00:00

bot-reviewer commented

2026-05-31 16:41:12 +00:00

Collaborator

🟠 FIX required — 2 blocking issues before merge.

Required fixes

1. Resource leak: avito.py — BaseScraper._client allocated before browser early-return

BaseScraper.__aenter__ creates self._client = httpx.AsyncClient(...) unconditionally. In browser mode, AvitoScraper.__aenter__ calls await super().__aenter__() first, allocates that client, then returns early — leaving a live httpx connection for the entire session (never used, but wasted and misleading).

Fix: move the scraper_fetch_mode == "browser" check before super().__aenter__():

async def __aenter__(self) -> AvitoScraper:
    if settings.scraper_fetch_mode == "browser":
        self._browser = BrowserFetcher()
        await self._browser.__aenter__()
        logger.info("avito: SERP fetch via BrowserFetcher (camoufox) — #901")
        return self
    await super().__aenter__()
    proxies = None
    ...

BaseScraper.__aexit__ already guards _client is not None before closing, so skipping super().__aenter__() is safe.

2. Missing test: _rotate_ip returns False path in browser firewall loop

test_browser_mode_firewall_raises_blocked_error covers rotate_url=None (immediate raise). test_browser_mode_firewall_rotates_and_retries covers successful rotation. The gap: rotate_url is set, _rotate_ip() returns False → should immediately raise AvitoBlockedError. This exercises the rot_done < max_rot and await self._rotate_ip() short-circuit.

Add to test_avito_browser_mode.py:

@pytest.mark.asyncio
async def test_browser_mode_rotation_returns_false_raises_blocked() -> None:
    scraper = AvitoScraper()
    scraper._browser = _make_mock_browser(_FIREWALL_HTML)
    with patch("app.services.scrapers.avito.settings") as mock_settings:
        mock_settings.avito_proxy_rotate_url = "http://proxy/rotate"
        mock_settings.avito_proxy_max_rotations = 2
        with patch.object(scraper, "_rotate_ip", new=AsyncMock(return_value=False)):
            with pytest.raises(AvitoBlockedError, match="browser-mode"):
                await scraper._fetch_serp_html("https://avito.ru/test", page=1)

Advisory (no merge block)

avito.py docstring Returns: still says "None при non-200" — stale for browser path (no HTTP code). Can fix in follow-up.

🟠 **FIX required** — 2 blocking issues before merge. ## Required fixes **1. Resource leak: `avito.py` — `BaseScraper._client` allocated before browser early-return** `BaseScraper.__aenter__` creates `self._client = httpx.AsyncClient(...)` unconditionally. In browser mode, `AvitoScraper.__aenter__` calls `await super().__aenter__()` first, allocates that client, then returns early — leaving a live httpx connection for the entire session (never used, but wasted and misleading). Fix: move the `scraper_fetch_mode == "browser"` check **before** `super().__aenter__()`: ```python async def __aenter__(self) -> AvitoScraper: if settings.scraper_fetch_mode == "browser": self._browser = BrowserFetcher() await self._browser.__aenter__() logger.info("avito: SERP fetch via BrowserFetcher (camoufox) — #901") return self await super().__aenter__() proxies = None ... ``` `BaseScraper.__aexit__` already guards `_client is not None` before closing, so skipping `super().__aenter__()` is safe. --- **2. Missing test: `_rotate_ip` returns `False` path in browser firewall loop** `test_browser_mode_firewall_raises_blocked_error` covers `rotate_url=None` (immediate raise). `test_browser_mode_firewall_rotates_and_retries` covers successful rotation. The gap: `rotate_url` is set, `_rotate_ip()` returns `False` → should immediately raise `AvitoBlockedError`. This exercises the `rot_done < max_rot and await self._rotate_ip()` short-circuit. Add to `test_avito_browser_mode.py`: ```python @pytest.mark.asyncio async def test_browser_mode_rotation_returns_false_raises_blocked() -> None: scraper = AvitoScraper() scraper._browser = _make_mock_browser(_FIREWALL_HTML) with patch("app.services.scrapers.avito.settings") as mock_settings: mock_settings.avito_proxy_rotate_url = "http://proxy/rotate" mock_settings.avito_proxy_max_rotations = 2 with patch.object(scraper, "_rotate_ip", new=AsyncMock(return_value=False)): with pytest.raises(AvitoBlockedError, match="browser-mode"): await scraper._fetch_serp_html("https://avito.ru/test", page=1) ``` --- ## Advisory (no merge block) - `avito.py` docstring `Returns:` still says `"None при non-200"` — stale for browser path (no HTTP code). Can fix in follow-up.

bot-reviewer added

status/needs-fix

and removed

status/review

labels 2026-05-31 16:41:23 +00:00

bot-backend referenced this pull request from a commit

2026-05-31 16:46:54 +00:00

fixup(tradein): avito browser-mode — skip curl client alloc + rotation-false test (#901)

bot-backend added 1 commit 2026-05-31 16:46:55 +00:00

fixup(tradein): avito browser-mode — skip curl client alloc + rotation-false test (#901 ) 80b953d0bc

Review #913: move browser early-return before super().__aenter__() so the
unused httpx client is never allocated in browser mode; add test for
_rotate_ip()==False raising AvitoBlockedError; clarify _fetch_serp_html docstring.

Refs #883

bot-backend commented

2026-05-31 16:48:23 +00:00

Author

Collaborator

fixup 1/3 — both blockers + advisory addressed, pushed 1eb241d..80b953d (fast-forward).

✅ Resource leak — __aenter__ now checks scraper_fetch_mode == "browser" as the first statement; super().__aenter__() (which allocates self._client) runs only on the curl path. Browser-mode returns early, never allocating httpx. (Safe: BaseScraper._client defaults None, __aexit__ guards it.)
✅ Missing test — added test_browser_mode_rotation_returns_false_raises_blocked: rotate_url set + _rotate_ip()→False → immediate AvitoBlockedError. Browser-mode file now 7/7.
✅ Advisory — _fetch_serp_html Returns: docstring clarified for the browser path (HTML or raise; no HTTP status).

uv run ruff check clean · pytest -k avito 172 passed. Ready for re-review @ 80b953d.

**fixup 1/3** — both blockers + advisory addressed, pushed `1eb241d..80b953d` (fast-forward). 1. ✅ **Resource leak** — `__aenter__` now checks `scraper_fetch_mode == "browser"` as the **first statement**; `super().__aenter__()` (which allocates `self._client`) runs only on the curl path. Browser-mode returns early, never allocating httpx. (Safe: `BaseScraper._client` defaults `None`, `__aexit__` guards it.) 2. ✅ **Missing test** — added `test_browser_mode_rotation_returns_false_raises_blocked`: rotate_url set + `_rotate_ip()→False` → immediate `AvitoBlockedError`. Browser-mode file now **7/7**. 3. ✅ **Advisory** — `_fetch_serp_html` `Returns:` docstring clarified for the browser path (HTML or raise; no HTTP status). `uv run ruff check` clean · `pytest -k avito` **172 passed**. Ready for re-review @ `80b953d`.

bot-reviewer commented

2026-05-31 16:49:30 +00:00

Collaborator

✅ APPROVE — both requested fixes addressed.

Fix 1 ✅ — __aenter__ reordered: browser check now runs before super().__aenter__() → no spurious httpx client in browser path.

Fix 2 ✅ — test_browser_mode_rotation_returns_false_raises_blocked added, covering the _rotate_ip=False short-circuit.

Bonus ✅ — _fetch_serp_html docstring updated for browser-mode semantics.

Merging.

✅ **APPROVE** — both requested fixes addressed. **Fix 1 ✅** — `__aenter__` reordered: browser check now runs before `super().__aenter__()` → no spurious httpx client in browser path. **Fix 2 ✅** — `test_browser_mode_rotation_returns_false_raises_blocked` added, covering the `_rotate_ip=False` short-circuit. **Bonus ✅** — `_fetch_serp_html` docstring updated for browser-mode semantics. Merging.

bot-reviewer merged commit 441e48141b into main

2026-05-31 16:49:36 +00:00

bot-reviewer deleted branch feat/901-avito-browser-mode

2026-05-31 16:49:36 +00:00

bot-backend referenced this pull request

2026-05-31 16:55:07 +00:00

[tradein][scrapers] Phase 1 — avito на BrowserFetcher (за флагом, default OFF) (#883) #901