feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905) #907

Merged

lekss361 merged 2 commits from feat/884-browser-separate-container into main

2026-05-31 14:44:00 +00:00

bot-backend commented

2026-05-31 14:30:28 +00:00

Collaborator

Summary

Revises #884's in-backend camoufox into a dedicated tradein-browser service (your call — browser shouldn't live in the API container). Confirmed feasible via camoufox docs: camoufox.server.launch_server(...) exposes a Playwright WS endpoint with fingerprint/geoip/proxy configured server-side; the backend connects remotely with plain playwright and no longer ships Firefox.

Architecture

Container	Carries	Image
tradein-browser (new)	camoufox + Firefox build + proxy + fingerprint, `launch_server` on :3000	heavy, `mem_limit: 2.5g` moved here
tradein-backend (slimmed)	`playwright` client → `connect(ws)` only — Firefox libs + camoufox dropped	back to lean, `mem_limit` removed

Changes

NEW tradein-mvp/browser/{server.py,Dockerfile} — camoufox WS server; Firefox fetch under app user + ENV HOME (carries the #899 cache-path fix so it doesn't repeat the root-vs-app bug).
browser_fetcher.py rewritten: launch-in-process → playwright.firefox.connect(settings.browser_ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) preserved.
backend Dockerfile slimmed (drop 5 Firefox libs + camoufox fetch layer); pyproject drops camoufox[geoip], promotes playwright to main deps.
docker-compose.prod.yml — add browser service, remove mem_limit from backend, backend depends_on browser.
config — browser_ws_endpoint default ws://tradein-browser:3000/playwright.
deploy-tradein.yml — new build-browser job + browser path-filter + IMAGE_BROWSER env, wired into deploy needs/gate (so the image exists before compose pulls it — otherwise deploy would break on a missing image).

Verified

launch_server signature checked (**kwargs→launch_options; port passes through; no host param — Playwright binds all interfaces by default).
playwright.firefox.connect needs no camoufox/Firefox on the client.
ruff clean; tests/scrapers/test_browser_fetcher.py rewritten for connect-mode → 6 pass (lifecycle, recycle, reconnect, counter-reset, crash-retry, error-propagation). import browser_fetcher works without camoufox installed.

Risk / dormancy

scraper_fetch_mode stays curl_cffi → no prod runtime behavior change. The browser is never exercised at runtime until Phase 1 (#901) flips a source to browser-mode. Merging does change the deploy topology (new container, slimmer backend image) — the build-browser job + depends_on handle ordering. Docker images not built locally (no Docker in the worker env) — CI build-browser is the real verification; recommend confirming that job goes green before/at merge.

Supersedes the in-backend approach of #884. Refs #905, #883, #884, #899, #901

## Summary Revises #884's in-backend camoufox into a dedicated **`tradein-browser`** service (your call — browser shouldn't live in the API container). Confirmed feasible via camoufox docs: `camoufox.server.launch_server(...)` exposes a Playwright WS endpoint with fingerprint/geoip/proxy configured server-side; the backend connects remotely with plain `playwright` and no longer ships Firefox. ### Architecture | Container | Carries | Image | |---|---|---| | **tradein-browser** (new) | camoufox + Firefox build + proxy + fingerprint, `launch_server` on :3000 | heavy, `mem_limit: 2.5g` moved here | | **tradein-backend** (slimmed) | `playwright` client → `connect(ws)` only — Firefox libs + camoufox dropped | back to lean, `mem_limit` removed | ### Changes - **NEW `tradein-mvp/browser/{server.py,Dockerfile}`** — camoufox WS server; Firefox fetch under `app` user + `ENV HOME` (carries the #899 cache-path fix so it doesn't repeat the root-vs-app bug). - **`browser_fetcher.py`** rewritten: launch-in-process → `playwright.firefox.connect(settings.browser_ws_endpoint)`; recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) preserved. - **backend Dockerfile** slimmed (drop 5 Firefox libs + camoufox fetch layer); **pyproject** drops `camoufox[geoip]`, promotes `playwright` to main deps. - **docker-compose.prod.yml** — add `browser` service, remove `mem_limit` from backend, `backend depends_on browser`. - **config** — `browser_ws_endpoint` default `ws://tradein-browser:3000/playwright`. - **deploy-tradein.yml** — new `build-browser` job + `browser` path-filter + `IMAGE_BROWSER` env, wired into `deploy` needs/gate (so the image exists before compose pulls it — otherwise deploy would break on a missing image). ### Verified - `launch_server` signature checked (`**kwargs`→`launch_options`; `port` passes through; no `host` param — Playwright binds all interfaces by default). - `playwright.firefox.connect` needs no camoufox/Firefox on the client. - ruff clean; `tests/scrapers/test_browser_fetcher.py` rewritten for connect-mode → 6 pass (lifecycle, recycle, reconnect, counter-reset, crash-retry, error-propagation). `import browser_fetcher` works without camoufox installed. ## Risk / dormancy `scraper_fetch_mode` stays `curl_cffi` → **no prod runtime behavior change**. The browser is never exercised at runtime until Phase 1 (#901) flips a source to browser-mode. Merging does change the deploy topology (new container, slimmer backend image) — the `build-browser` job + `depends_on` handle ordering. **Docker images not built locally** (no Docker in the worker env) — CI build-browser is the real verification; recommend confirming that job goes green before/at merge. Supersedes the in-backend approach of #884. Refs #905, #883, #884, #899, #901

bot-backend added 2 commits 2026-05-31 14:30:29 +00:00

feat(browser): split camoufox into tradein-browser container, connect-mode (#905 ) 5fb367295a

Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service.
camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/
proxy configured server-side); the backend connects remotely via playwright, so it
no longer ships Firefox.

- NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000;
  Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix)
- BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint);
  recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry)
- backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer
- pyproject: remove camoufox[geoip] from backend, promote playwright to main deps
- docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove
  mem_limit from backend, backend depends_on browser (service_started)
- config: browser_ws_endpoint default ws://tradein-browser:3000/playwright
- tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass)

Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change.
launch_server signature verified (**kwargs→launch_options, port passes through).

Refs #905, #883, #884, #899

ci(tradein): build+push tradein-browser image in deploy workflow (#905 ) 0566fed5a2

The new browser service references ghcr.io/lekss361/gendesign-tradein-browser;
without a build job the deploy's compose-up would fail pulling a missing image.
Adds build-browser job (mirrors build-backend, context ./tradein-mvp/browser),
a browser path-filter (tradein-mvp/browser/**), IMAGE_BROWSER env, and wires
build-browser into deploy needs + the no-failure gate.

Refs #905

lekss361 reviewed 2026-05-31 14:42:42 +00:00

lekss361 left a comment

Owner

Manual review (code-reviewer subagent) — ⚠️ MINOR

Корректность/безопасность — без блокеров. Deploy-risk низкий (browser dormant: scraper_fetch_mode остаётся curl_cffi, runtime не меняется до Phase 1).

👍 Сильно

Deploy-ordering верный: build-browser в needs deploy-job + gate на result != 'failure', параллельно с frontend. Образ окажется в GHCR до compose pull.
Path-filter покрывает browser: + infra: — нет false-negative для нового сервиса.
#899-фикс сохранён: USER app + ENV HOME перед camoufox fetch (Firefox под app-юзером).
_close()/_reconnect() без утечек (_pw.stop() в finally, page-counter сброс на close); crash-recovery тест проверяет connect.await_count == 2.
Dormancy-claim подтверждён — ничто в диффе не флипает scraper_fetch_mode=browser. Backend Dockerfile слиммлен корректно (5 Firefox-либ убраны, есть в browser/). pyproject: camoufox[geoip] убран, playwright промоутнут.

🔧 Перед merge / до Phase 1

docker-compose.prod.yml depends_on browser: service_started не гарантирует готовность WS (Firefox init 5-15с). При Phase-1 флипе backend упадёт на connect(). Нужен healthcheck на browser + condition: service_healthy. Сейчас безопасно (dormant) — можно отложить, но до флипа обязательно.
deploy-tradein.yml cleanup-loop не чистит browser-образ (~1GB) → SHA-тегнутые старые образы копятся на VPS. Добавить ghcr.io/lekss361/gendesign-tradein-browser в for-loop.
⚠️ server.py launch_server(port=...) без host= — проверить, что биндит 0.0.0.0, не loopback (иначе backend-контейнер не достучится). Smoke на первом деплое: docker exec tradein-backend curl -sv ws://tradein-browser:3000. Если default не 0.0.0.0 — передать host="0.0.0.0" явно.

Post-merge polish

test_browser_fetcher.py:27,34 — proxy_url/scraper_proxy_url в mock-settings dead (proxy теперь server-side); убрать.

Перед первым деплоем: df -h /var/lib/docker (первый compose up тянет ~1GB образ).

Это ручной advisory-review (main-сессия). Финальный verdict + merge — за auto-code-reviewer.

## Manual review (code-reviewer subagent) — ⚠️ MINOR Корректность/безопасность — без блокеров. **Deploy-risk низкий** (browser dormant: `scraper_fetch_mode` остаётся `curl_cffi`, runtime не меняется до Phase 1). **👍 Сильно** - Deploy-ordering **верный**: `build-browser` в `needs` deploy-job + gate на `result != 'failure'`, параллельно с frontend. Образ окажется в GHCR до `compose pull`. - Path-filter покрывает `browser:` + `infra:` — нет false-negative для нового сервиса. - #899-фикс сохранён: `USER app` + `ENV HOME` перед `camoufox fetch` (Firefox под app-юзером). - `_close()/_reconnect()` без утечек (`_pw.stop()` в `finally`, page-counter сброс на close); crash-recovery тест проверяет `connect.await_count == 2`. - Dormancy-claim **подтверждён** — ничто в диффе не флипает `scraper_fetch_mode=browser`. Backend Dockerfile слиммлен корректно (5 Firefox-либ убраны, есть в browser/). pyproject: `camoufox[geoip]` убран, `playwright` промоутнут. **🔧 Перед merge / до Phase 1** - **`docker-compose.prod.yml` `depends_on browser: service_started` не гарантирует готовность WS** (Firefox init 5-15с). При Phase-1 флипе backend упадёт на `connect()`. Нужен `healthcheck` на browser + `condition: service_healthy`. Сейчас безопасно (dormant) — можно отложить, но до флипа обязательно. - **`deploy-tradein.yml` cleanup-loop не чистит browser-образ** (~1GB) → SHA-тегнутые старые образы копятся на VPS. Добавить `ghcr.io/lekss361/gendesign-tradein-browser` в for-loop. - ⚠️ **`server.py` `launch_server(port=...)` без `host=`** — проверить, что биндит `0.0.0.0`, не loopback (иначе backend-контейнер не достучится). Smoke на первом деплое: `docker exec tradein-backend curl -sv ws://tradein-browser:3000`. Если default не `0.0.0.0` — передать `host="0.0.0.0"` явно. **Post-merge polish** - `test_browser_fetcher.py:27,34` — `proxy_url`/`scraper_proxy_url` в mock-settings dead (proxy теперь server-side); убрать. **Перед первым деплоем:** `df -h /var/lib/docker` (первый `compose up` тянет ~1GB образ). _Это ручной advisory-review (main-сессия). Финальный verdict + merge — за auto-code-reviewer._