feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) #1735

Merged

lekss361 merged 1 commit from feat/browser-per-source-proxy-pool into main

2026-06-18 06:06:58 +00:00

lekss361 commented

2026-06-18 06:04:06 +00:00

Owner

What & why

The camoufox browser service runs ONE global AsyncCamoufox with the proxy fixed at launch (apw). All scraper sources egress through that single proxy → mutual wedge under load (apw degrades ~30min → page.goto timeout → 500 → 0 lots), and BROWSER_CONCURRENCY is effectively pinned because one mobile proxy can't take parallel load.

This adds a per-source proxy pool so each source gets its own camoufox instance + proxy, fully behind a feature flag.

Phase 1 = flag OFF by default (zero behavior change)

FEATURE_BROWSER_POOL_ENABLED defaults to false. With it off, fetch_handler delegates to _fetch_via_single — the current single-browser + semaphore path, unchanged. So merging + deploying this is a no-op for prod behavior. Env wiring + flipping the flag = Phase 2 (separate, owner-gated) — this PR does NOT touch .env.runtime or compose.

Design (ON path)

_build_proxy_map() → BROWSER_PROXY_MAP from env: BROWSER_PROXY_{AVITO,CIAN,YANDEX,DOMCLICK} with legacy fallbacks (AVITO_PROXY_URL/SCRAPER_PROXY_URL, CIAN_PROXY_URL, YANDEX_PROXY_URL).
Proxy-keyed pool: _browser_pool / _browser_cm_pool / _page_counter_pool + lazy per-proxy asyncio.Lock. Lazy-launch per proxy on first /fetch, per-proxy recycle at BROWSER_RECYCLE_PAGES, per-proxy crash-recovery (relaunch + retry once).
fetch_handler routes by body["source"] → BROWSER_PROXY_MAP[source] → fallback avito → "" (direct). BrowserFetcher(source=...) passes it; every scraper callsite sets its source.
Concurrency model: the per-proxy lock serializes to 1 fetch per proxy (the mobile-proxy-safe unit — parallel load on one mobile IP is exactly what wedges it), while different proxies run in parallel. That's the throughput unblock: N proxies → N concurrent fetches.
/login stays on the single-browser path for both flag states (cian-only, low volume; pooling login is out of scope).
No creds in logs (pool errors log type(exc).__name__ only, mirroring existing sanitization).

Callsites updated (every `BrowserFetcher(...)`)

avito/scrape_pipeline/avito_detail_backfill → avito; cian_newbuilding/cian_history_backfill/admin(login) → cian; yandex_realty/yandex_newbuilding → yandex; domclick → domclick. (cian_detail.py receives an injected fetcher — nothing to change.)

Tests

test_browser_fetcher.py: asserts source is sent in the /fetch body (default avito + explicit yandex).
test_browser_pool_routing.py (new): flag-ON routes by source + acquires that proxy's lock; unknown-source→avito fallback; 503 on unavailable proxy; flag-OFF takes the single path and ignores source; _build_proxy_map empty-drop/fallback. (Loads browser/server.py via an aiohttp stub since aiohttp isn't in the backend venv.)
Full gate: pytest -q (2 standard deselects) → 1913 passed, 2 deselected. ruff check+format clean on touched backend files. Pre-commit green.

Phase 2 (follow-up, not in this PR)

Add to backend/.env.runtime: BROWSER_PROXY_AVITO=apw, BROWSER_PROXY_CIAN=kf, BROWSER_PROXY_YANDEX=ti, FEATURE_BROWSER_POOL_ENABLED=true; recreate browser; smoke that avito/cian/yandex egress via distinct exit IPs. (domclick has no dedicated proxy yet → falls back to avito/apw, deferred.) Proxies already validated live: apw/kf/ti all reachable, authed-socks5 works, container mem limit 5GB (fits 4 instances).

Reviewer note

The intentional asymmetry: OFF path uses the global semaphore on one browser; ON path uses per-proxy locks (ignores the global semaphore) because per-proxy serialization is the real safety constraint for mobile proxies. mypy not run locally (not in dev group) — annotations are trivial (source: str | None); worth a CI mypy pass if applicable.

## What & why The camoufox browser service runs ONE global AsyncCamoufox with the proxy fixed at launch (apw). All scraper sources egress through that single proxy → mutual wedge under load (apw degrades ~30min → `page.goto` timeout → 500 → 0 lots), and `BROWSER_CONCURRENCY` is effectively pinned because one mobile proxy can't take parallel load. This adds a **per-source proxy pool** so each source gets its own camoufox instance + proxy, fully behind a feature flag. ## Phase 1 = flag OFF by default (zero behavior change) `FEATURE_BROWSER_POOL_ENABLED` defaults to **false**. With it off, `fetch_handler` delegates to `_fetch_via_single` — the **current** single-browser + semaphore path, unchanged. So merging + deploying this is a no-op for prod behavior. Env wiring + flipping the flag = **Phase 2** (separate, owner-gated) — this PR does NOT touch `.env.runtime` or compose. ## Design (ON path) - `_build_proxy_map()` → `BROWSER_PROXY_MAP` from env: `BROWSER_PROXY_{AVITO,CIAN,YANDEX,DOMCLICK}` with legacy fallbacks (`AVITO_PROXY_URL`/`SCRAPER_PROXY_URL`, `CIAN_PROXY_URL`, `YANDEX_PROXY_URL`). - Proxy-keyed pool: `_browser_pool` / `_browser_cm_pool` / `_page_counter_pool` + lazy **per-proxy** `asyncio.Lock`. Lazy-launch per proxy on first `/fetch`, per-proxy recycle at `BROWSER_RECYCLE_PAGES`, per-proxy crash-recovery (relaunch + retry once). - `fetch_handler` routes by `body["source"]` → `BROWSER_PROXY_MAP[source]` → fallback avito → `""` (direct). `BrowserFetcher(source=...)` passes it; every scraper callsite sets its source. - **Concurrency model**: the per-proxy lock serializes to **1 fetch per proxy** (the mobile-proxy-safe unit — parallel load on one mobile IP is exactly what wedges it), while *different* proxies run in parallel. That's the throughput unblock: N proxies → N concurrent fetches. - `/login` stays on the single-browser path for both flag states (cian-only, low volume; pooling login is out of scope). - No creds in logs (pool errors log `type(exc).__name__` only, mirroring existing sanitization). ## Callsites updated (every `BrowserFetcher(...)`) avito/scrape_pipeline/avito_detail_backfill → `avito`; cian_newbuilding/cian_history_backfill/admin(login) → `cian`; yandex_realty/yandex_newbuilding → `yandex`; domclick → `domclick`. (`cian_detail.py` receives an injected fetcher — nothing to change.) ## Tests - `test_browser_fetcher.py`: asserts `source` is sent in the `/fetch` body (default `avito` + explicit `yandex`). - `test_browser_pool_routing.py` (new): flag-ON routes by source + acquires that proxy's lock; unknown-source→avito fallback; 503 on unavailable proxy; flag-OFF takes the single path and ignores source; `_build_proxy_map` empty-drop/fallback. (Loads `browser/server.py` via an aiohttp stub since aiohttp isn't in the backend venv.) - Full gate: `pytest -q` (2 standard deselects) → **1913 passed, 2 deselected**. ruff check+format clean on touched backend files. Pre-commit green. ## Phase 2 (follow-up, not in this PR) Add to `backend/.env.runtime`: `BROWSER_PROXY_AVITO=apw`, `BROWSER_PROXY_CIAN=kf`, `BROWSER_PROXY_YANDEX=ti`, `FEATURE_BROWSER_POOL_ENABLED=true`; recreate browser; smoke that avito/cian/yandex egress via distinct exit IPs. (domclick has no dedicated proxy yet → falls back to avito/apw, deferred.) Proxies already validated live: apw/kf/ti all reachable, authed-socks5 works, container mem limit 5GB (fits 4 instances). ## Reviewer note The intentional asymmetry: OFF path uses the global semaphore on one browser; ON path uses per-proxy locks (ignores the global semaphore) because per-proxy serialization is the real safety constraint for mobile proxies. mypy not run locally (not in dev group) — annotations are trivial (`source: str | None`); worth a CI mypy pass if applicable.

lekss361 added 1 commit 2026-06-18 06:04:06 +00:00

feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1)

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

b9eb478c8f

Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.

Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.

BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).