gendesign

lekss361/gendesign

Fork 0

Commit graph

Author	SHA1	Message	Date
bot-backend	393ecea458	fix(browser): run camoufox fetch as root so geoip mmdb write succeeds (#905 ) CI build-browser (and the deploy) went red on #907's browser image: PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.12/site-packages/camoufox/GeoLite2-City.mmdb' camoufox fetch with geoip=True downloads TWO things: the Firefox build (→ $HOME/.cache, correctly routed to /home/app via ENV HOME, the #899 fix) AND the GeoLite2 mmdb, which it writes INTO the camoufox package dir in site-packages. Since the browser image pip-installs camoufox as root, that dir is root-owned, so running fetch as the app user can't write the mmdb → build fails. Fix: run fetch as root (writes both the mmdb and — via ENV HOME — Firefox into /home/app/.cache), then chown the Firefox cache to app. The mmdb stays root-owned in site-packages but world-readable, which is all the runtime app user needs. Verified by a real local image build (camoufox fetch now completes, image exports). The deploy was gated off by the failed build-browser job (not partially applied), so prod was never broken. Refs #905, #883, #899	2026-05-31 17:51:43 +03:00
bot-backend	5fb367295a	feat(browser): split camoufox into tradein-browser container, connect-mode (#905 ) Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service. camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/ proxy configured server-side); the backend connects remotely via playwright, so it no longer ships Firefox. - NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000; Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix) - BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) - backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer - pyproject: remove camoufox[geoip] from backend, promote playwright to main deps - docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove mem_limit from backend, backend depends_on browser (service_started) - config: browser_ws_endpoint default ws://tradein-browser:3000/playwright - tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass) Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change. launch_server signature verified (**kwargs→launch_options, port passes through). Refs #905, #883, #884, #899	2026-05-31 17:24:35 +03:00

Author

SHA1

Message

Date

bot-backend

393ecea458

fix(browser): run camoufox fetch as root so geoip mmdb write succeeds (#905 )

CI build-browser (and the deploy) went red on #907's browser image:
  PermissionError: [Errno 13] Permission denied:
  '/usr/local/lib/python3.12/site-packages/camoufox/GeoLite2-City.mmdb'

camoufox fetch with geoip=True downloads TWO things: the Firefox build (→
$HOME/.cache, correctly routed to /home/app via ENV HOME, the #899 fix) AND the
GeoLite2 mmdb, which it writes INTO the camoufox package dir in site-packages.
Since the browser image pip-installs camoufox as root, that dir is root-owned, so
running fetch as the app user can't write the mmdb → build fails.

Fix: run fetch as root (writes both the mmdb and — via ENV HOME — Firefox into
/home/app/.cache), then chown the Firefox cache to app. The mmdb stays root-owned
in site-packages but world-readable, which is all the runtime app user needs.

Verified by a real local image build (camoufox fetch now completes, image exports).
The deploy was gated off by the failed build-browser job (not partially applied),
so prod was never broken.

Refs #905, #883, #899

2026-05-31 17:51:43 +03:00

bot-backend

5fb367295a

feat(browser): split camoufox into tradein-browser container, connect-mode (#905 )

Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service.
camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/
proxy configured server-side); the backend connects remotely via playwright, so it
no longer ships Firefox.

- NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000;
  Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix)
- BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint);
  recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry)
- backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer
- pyproject: remove camoufox[geoip] from backend, promote playwright to main deps
- docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove
  mem_limit from backend, backend depends_on browser (service_started)
- config: browser_ws_endpoint default ws://tradein-browser:3000/playwright
- tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass)

Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change.
launch_server signature verified (**kwargs→launch_options, port passes through).

Refs #905, #883, #884, #899

2026-05-31 17:24:35 +03:00

2 commits