feat(yandex): route gate-API fetch through camoufox (sustainable, no proxy burn) #1699

Merged

lekss361 merged 1 commit from feat/yandex-camoufox-transport into main

2026-06-17 18:28:51 +00:00

lekss361 commented

2026-06-17 18:18:42 +00:00

Owner

Summary

Minimal transport swap: yandex gate-API fetch now goes through camoufox BrowserFetcher instead of the system-curl SOCKS5 subprocess. Everything else (URL building, _parse_gate_json, pagination, combos, field mapping) unchanged.

Why

system-curl raw fingerprint → yandex tarpits + burns the mobile-proxy IP within ~dozens of requests (proven: abk/mg pools got flagged).
camoufox (real-browser fingerprint) → not tarpitted, doesn't burn the proxy — works even on already-flagged IPs (3/3 pages on burned abk). ~7-19s/page, sustainable.

Change (68 ins / 92 del)

__aenter__ opens one BrowserFetcher (reused across pages); __aexit__ closes it.
_http_get → await self._browser.fetch(url), extracts JSON from camoufox <pre>{…}</pre> wrapper (verified prod shape), returns _CurlResponse(200, json); failure → status 0 (existing retry path).
Removed dead _curl_subprocess_get, _CURL_UA, _CURL_STATUS_MARKER.

Test plan

Full gate: 1845 passed, 2 deselected; ruff clean
Test: _extract_json_from_content unwraps real camoufox <pre> fixture
Post-deploy smoke: yandex_city_sweep → lots>0, no tarpit, proxy not burned

Refs #759, #1658

## Summary Minimal transport swap: yandex gate-API fetch now goes through camoufox `BrowserFetcher` instead of the system-curl SOCKS5 subprocess. Everything else (URL building, `_parse_gate_json`, pagination, combos, field mapping) unchanged. ## Why - system-curl raw fingerprint → yandex tarpits + **burns the mobile-proxy IP** within ~dozens of requests (proven: abk/mg pools got flagged). - camoufox (real-browser fingerprint) → **not tarpitted, doesn't burn the proxy** — works even on already-flagged IPs (3/3 pages on burned abk). ~7-19s/page, sustainable. ## Change (68 ins / 92 del) - `__aenter__` opens one `BrowserFetcher` (reused across pages); `__aexit__` closes it. - `_http_get` → `await self._browser.fetch(url)`, extracts JSON from camoufox `<pre>{…}</pre>` wrapper (verified prod shape), returns `_CurlResponse(200, json)`; failure → status 0 (existing retry path). - Removed dead `_curl_subprocess_get`, `_CURL_UA`, `_CURL_STATUS_MARKER`. ## Test plan - [x] Full gate: **1845 passed, 2 deselected**; ruff clean - [x] Test: `_extract_json_from_content` unwraps real camoufox `<pre>` fixture - [ ] Post-deploy smoke: yandex_city_sweep → lots>0, no tarpit, proxy not burned Refs #759, #1658

lekss361 added 1 commit 2026-06-17 18:18:42 +00:00

feat(yandex): route gate-API fetch through camoufox BrowserFetcher

CI / changes (push) Has been cancelled

Details

CI / backend-tests (push) Has been cancelled

Details

CI / frontend-tests (push) Has been cancelled

Details

CI / openapi-codegen-check (push) Has been cancelled

Details

CI / changes (pull_request) Has been cancelled

Details

CI / backend-tests (pull_request) Has been cancelled

Details

CI / frontend-tests (pull_request) Has been cancelled

Details

CI / openapi-codegen-check (pull_request) Has been cancelled

Details

a8d7be68ad

Replace the system-curl SOCKS5 subprocess transport with the shared
BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not
tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl
got the proxy IP flagged.

One BrowserFetcher is opened per scraper session in __aenter__ and reused
across all page fetches. _http_get now fetches via the browser, extracts the
gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body
of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so
all existing callers and the tarpit retry/rotate logic keep working unchanged.

URL building, _parse_gate_json, pagination, combos and field mapping are
untouched. Removed the dead curl subprocess constants/method.