fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) #909

Merged

lekss361 merged 1 commit from fix/905-browser-http-wrapper into main

2026-05-31 15:27:11 +00:00

bot-backend commented

2026-05-31 15:25:31 +00:00

Collaborator

Problem

The tradein-browser container from #907 crash-loops in prod. Root-caused by running the built image: camoufox 0.4.11's launch_server spawns a Node launchServer.js that require()s playwright/driver/package/lib/browserServerImpl.js — a file that no longer exists in any playwright 1.45–1.60 (verified by probing each). So camoufox.server.launch_server is fundamentally incompatible with current playwright → MODULE_NOT_FOUND → RuntimeError: Server process terminated unexpectedly, restart loop.

The deploy that shipped #907/#908 left tradein-browser crash-looping — harmless to prod (browser is dormant, scraper_fetch_mode=curl_cffi), but the container never comes up.

Fix — don't use the playwright protocol between containers

Verified that local AsyncCamoufox works fine in the image (launches Firefox, drives a page). So the browser container now runs camoufox locally and exposes plain HTTP; the backend talks to it over httpx — no fragile playwright WS server.

browser/server.py — aiohttp service holding a local AsyncCamoufox (headless, os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} → new_page → goto → content → close → {html}; GET /health. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
browser/Dockerfile — +aiohttp (the #908 camoufox-fetch-as-root block is untouched).
backend BrowserFetcher — playwright.firefox.connect → httpx.post(browser_http_endpoint + "/fetch"), one retry on transport/HTTP error; all playwright imports dropped.
config — browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).

Verified by a REAL local image build + run

Image builds (exit 0); camoufox fetch passes (#908 fix intact).
Container stays Up, Restarts=0 — no crash-loop (vs the WS version's Restarting (1)).
GET /health → 200 {"status":"ok"}.
POST /fetch {"url":"https://example.com"} → 500 Page.goto: NS_ERROR_UNKNOWN_HOST — i.e. the request reached the server, camoufox launched Firefox and began navigation, failing only on DNS (no egress in the local build env). This proves the full HTTP→camoufox→Firefox chain; a real page fetch validates on prod (network + proxy).
7 unit tests (mocked httpx) + ruff clean.

Risk

Dormant — scraper_fetch_mode stays curl_cffi, no runtime behavior change. Replaces the crash-looping container so build-browser/deploy go green and tradein-browser actually comes up. Refs #905, #883, #884

## Problem The `tradein-browser` container from #907 **crash-loops** in prod. Root-caused by running the built image: camoufox 0.4.11's `launch_server` spawns a Node `launchServer.js` that `require()`s `playwright/driver/package/lib/browserServerImpl.js` — a file that **no longer exists in any playwright 1.45–1.60** (verified by probing each). So `camoufox.server.launch_server` is fundamentally incompatible with current playwright → `MODULE_NOT_FOUND` → `RuntimeError: Server process terminated unexpectedly`, restart loop. The deploy that shipped #907/#908 left `tradein-browser` crash-looping — harmless to prod (browser is dormant, `scraper_fetch_mode=curl_cffi`), but the container never comes up. ## Fix — don't use the playwright protocol between containers Verified that **local** `AsyncCamoufox` works fine in the image (launches Firefox, drives a page). So the browser container now runs camoufox **locally** and exposes plain **HTTP**; the backend talks to it over httpx — no fragile playwright WS server. - **`browser/server.py`** — aiohttp service holding a local `AsyncCamoufox` (headless, os/locale/geoip/humanize, proxy from `SCRAPER_PROXY_URL`). `POST /fetch {url}` → new_page → goto → content → close → `{html}`; `GET /health`. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via `asyncio.Lock`. - **`browser/Dockerfile`** — `+aiohttp` (the #908 camoufox-fetch-as-root block is untouched). - **backend `BrowserFetcher`** — `playwright.firefox.connect` → `httpx.post(browser_http_endpoint + "/fetch")`, one retry on transport/HTTP error; all playwright imports dropped. - **config** — `browser_ws_endpoint` → `browser_http_endpoint` (`http://tradein-browser:3000`). ## Verified by a REAL local image build + run - Image builds (exit 0); `camoufox fetch` passes (#908 fix intact). - Container stays **Up, Restarts=0** — no crash-loop (vs the WS version's `Restarting (1)`). - `GET /health` → **200** `{"status":"ok"}`. - `POST /fetch {"url":"https://example.com"}` → **500** `Page.goto: NS_ERROR_UNKNOWN_HOST` — i.e. the request reached the server, camoufox **launched Firefox and began navigation**, failing only on DNS (no egress in the local build env). This proves the full **HTTP→camoufox→Firefox** chain; a real page fetch validates on prod (network + proxy). - 7 unit tests (mocked httpx) + ruff clean. ## Risk Dormant — `scraper_fetch_mode` stays `curl_cffi`, no runtime behavior change. Replaces the crash-looping container so `build-browser`/deploy go green and `tradein-browser` actually comes up. Refs #905, #883, #884

bot-backend added 1 commit 2026-05-31 15:25:31 +00:00

fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905 ) 6044a1fe3c

The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.

Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
  os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
  new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
  crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
  (one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).

Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.

Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884