Add an in-page fetch() path so backend code can run a same-origin fetch from
inside the warm camoufox page (real fingerprint + context cookies + per-provider
proxy) and get back {status, body}. Purely additive: /fetch, _fetch_once,
_do_fetch, fetch_handler and BrowserFetcher.fetch are untouched. Nothing in the
prod flow calls it yet — a later PR wires avito IMV to it.
Sidecar: _fetch_json_once navigates to origin (anchor page, short 500ms settle
instead of full BROWSER_WAIT_MS), runs fetch via page.evaluate, mirrors the
_fetch_once page lifecycle + recycle/relaunch. _do_fetch_json mirrors _do_fetch
crash-retry. fetch_json_handler mirrors fetch_handler (400/503/500 handling).
Client: BrowserFetcher.fetch_json posts to /fetch-json with the same
single-retry-on-HTTPError/TransportError pattern as fetch.
Tests: browser/test_server_fetch_json.py (handler happy path, origin navigation,
POST payload passthrough, 400 missing url, 400 bad json, 503 unavailable, 500 on
evaluate error) and backend fetch_json client tests (payload, defaults, retry,
raise-after-two-errors).
Refs #915
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.
Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
(one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).
Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.
Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884
