feat(browser): concurrent page pool (BROWSER_CONCURRENCY, default 4) + raise mem_limit to 5g #1733

Merged

lekss361 merged 2 commits from feat/browser-concurrent-pool into main

2026-06-17 21:07:32 +00:00

lekss361 commented

2026-06-17 20:56:17 +00:00

Owner

Summary

The tradein-browser camoufox service serialized every fetch through a single _fetch_lock (one page.goto at a time, ~10s/fetch queued) — the throughput bottleneck for ALL scrapers. This makes it process up to N concurrent fetches via a page pool, N = env BROWSER_CONCURRENCY (default 4, ops-overridable).

Design (safety-first — single point of failure for all scraping)

Split the old _fetch_lock's two jobs: _launch_lock guards ONLY (re)launch (serialized + idempotent); _fetch_sem = Semaphore(BROWSER_CONCURRENCY) bounds concurrent fetches.
Per-fetch isolation: each /fetch opens its OWN browser.new_page(), goto+wait+content, closes in finally. No page sharing. Nav timeout / hydration wait / proxy+geoip kwargs preserved. Response shape identical (BrowserFetcher untouched).
_safe_relaunch (recycle + crash recovery) runs under _launch_lock only when _inflight<=1 → never tears the browser out from under live sibling pages; else deferred (logged). Dead browser → lazy relaunch on next request → 503 if proxy down (existing caller-retry path intact).
BROWSER_CONCURRENCY parsed at startup (default 4, clamp≥1).
docker-compose.prod.yml: tradein-browser mem_limit 2.5g→5g (N firefox pages; ~6.9G free on VPS) + BROWSER_CONCURRENCY: ${BROWSER_CONCURRENCY:-4}.

Test plan

12 passed (config default/override/clamp/garbage; semaphore caps to N; per-page open/close; launch serialized under concurrent callers; 503/health). AsyncCamoufox/page mocked.
ruff clean; server.py parses
Post-deploy smoke: single /fetch works; a sweep runs faster (concurrent fetches); monitor tradein-browser RSS before ops raises BROWSER_CONCURRENCY to 8 via .env.runtime

Risks (flagged)

Memory: N=8 needs headroom — default stays 4 until ops opts in; restart unless-stopped catches OOM.
Recycle best-effort under sustained load (crash-recovery still protects correctness).
All N pages share one browser fingerprint/IP → higher request rate; watch avito ban rate after ramp.

Refs #796

## Summary The tradein-browser camoufox service serialized every fetch through a single `_fetch_lock` (one page.goto at a time, ~10s/fetch queued) — the throughput bottleneck for ALL scrapers. This makes it process up to N concurrent fetches via a page pool, N = env `BROWSER_CONCURRENCY` (default 4, ops-overridable). ## Design (safety-first — single point of failure for all scraping) - Split the old `_fetch_lock`'s two jobs: `_launch_lock` guards ONLY (re)launch (serialized + idempotent); `_fetch_sem = Semaphore(BROWSER_CONCURRENCY)` bounds concurrent fetches. - Per-fetch isolation: each /fetch opens its OWN `browser.new_page()`, goto+wait+content, closes in `finally`. No page sharing. Nav timeout / hydration wait / proxy+geoip kwargs preserved. Response shape identical (BrowserFetcher untouched). - `_safe_relaunch` (recycle + crash recovery) runs under `_launch_lock` only when `_inflight<=1` → never tears the browser out from under live sibling pages; else deferred (logged). Dead browser → lazy relaunch on next request → 503 if proxy down (existing caller-retry path intact). - `BROWSER_CONCURRENCY` parsed at startup (default 4, clamp≥1). - docker-compose.prod.yml: tradein-browser `mem_limit 2.5g→5g` (N firefox pages; ~6.9G free on VPS) + `BROWSER_CONCURRENCY: ${BROWSER_CONCURRENCY:-4}`. ## Test plan - [x] 12 passed (config default/override/clamp/garbage; semaphore caps to N; per-page open/close; launch serialized under concurrent callers; 503/health). AsyncCamoufox/page mocked. - [x] ruff clean; server.py parses - [ ] Post-deploy smoke: single /fetch works; a sweep runs faster (concurrent fetches); monitor tradein-browser RSS before ops raises BROWSER_CONCURRENCY to 8 via .env.runtime ## Risks (flagged) - Memory: N=8 needs headroom — default stays 4 until ops opts in; restart unless-stopped catches OOM. - Recycle best-effort under sustained load (crash-recovery still protects correctness). - All N pages share one browser fingerprint/IP → higher request rate; watch avito ban rate after ramp. Refs #796

lekss361 added 1 commit 2026-06-17 20:56:17 +00:00