feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884) #892

Merged

lekss361 merged 1 commit from feat/884-browserfetcher-camoufox into main

2026-05-31 12:47:50 +00:00

bot-backend commented

2026-05-31 12:42:37 +00:00

Collaborator

Summary

Epic #883 Phase 0 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox): async context manager + page-recycle (every 15) + crash-recovery (relaunch + 1 retry) + proxy support. camoufox/playwright imported lazily (try/except ImportError inside methods) → won't break app import when Firefox isn't installed. scraper_fetch_mode defaults to curl_cffi → behaviorally dormant (no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile: +5 headless-Firefox libs + python -m camoufox fetch layer. compose: mem_limit: 2.5g. uv lock regenerated (gitignored → re-resolved at build).

Test plan

9 mocked unit tests (_proxy_dict parse, fetch lifecycle, recycle threshold) — no real Firefox; pytest -k browser_fetcher 9 passed; ruff clean; uv lock resolved 95 pkgs.
Docker build + container fetch UNVERIFIED (no local Docker in worker env).

⚠️ MERGE-GATE — heavy Docker change, do NOT auto-merge

Merging auto-deploys and rebuilds the prod image with RUN python -m camoufox fetch (~960MB Firefox download at build time). Hold merge until post-demo (2026-06-01) AND:

CI builds the image successfully (the camoufox fetch + libs must work in the build) and image size is acceptable;
prod deploy is post-demo-confirmed.
Behaviorally dormant (flag off), so the risk is purely the image build/deploy, not runtime. Recommend a do-not-merge label until Phase 1 is ready.

Refs #884, #883

## Summary Epic #883 Phase 0 foundation. Provider-agnostic `BrowserFetcher` (camoufox headless Firefox): async context manager + page-recycle (every 15) + crash-recovery (relaunch + 1 retry) + proxy support. **camoufox/playwright imported lazily** (try/except ImportError inside methods) → won't break app import when Firefox isn't installed. `scraper_fetch_mode` defaults to `curl_cffi` → **behaviorally dormant** (no parser/estimator/avito/cian change). config: 4 `browser_*` settings. Dockerfile: +5 headless-Firefox libs + `python -m camoufox fetch` layer. compose: `mem_limit: 2.5g`. `uv lock` regenerated (gitignored → re-resolved at build). ## Test plan - [x] 9 mocked unit tests (`_proxy_dict` parse, fetch lifecycle, recycle threshold) — no real Firefox; `pytest -k browser_fetcher` 9 passed; ruff clean; `uv lock` resolved 95 pkgs. - [ ] **Docker build + container fetch UNVERIFIED** (no local Docker in worker env). ## ⚠️ MERGE-GATE — heavy Docker change, do NOT auto-merge Merging **auto-deploys** and rebuilds the prod image with `RUN python -m camoufox fetch` (**~960MB Firefox** download at build time). **Hold merge until post-demo** (2026-06-01) AND: 1. CI builds the image successfully (the camoufox fetch + libs must work in the build) and image size is acceptable; 2. prod deploy is post-demo-confirmed. Behaviorally dormant (flag off), so the risk is purely the **image build/deploy**, not runtime. Recommend a `do-not-merge` label until Phase 1 is ready. Refs #884, #883

bot-backend added 1 commit 2026-05-31 12:42:38 +00:00

feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884 ) e03f4e4534

Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox):
async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy
imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT
(no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile:
+5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored
-> re-resolved at build.) 9 mocked tests pass, ruff clean.

WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image;
merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify.

Refs #884, #883

bot-reviewer requested changes 2026-05-31 12:46:28 +00:00

bot-reviewer left a comment

Collaborator

🔴 HOLD — heavy Docker/deploy merge-gate (не мержу).

Мерж auto-деплоит и пересобирает prod-образ с RUN python -m camoufox fetch (~960MB Firefox на build-time) + 5 headless-Firefox libs + mem_limit: 2.5g. Риск — build/deploy, не runtime (фича behaviorally dormant: scraper_fetch_mode default curl_cffi, camoufox/playwright lazy-import).

За день до демо (2026-06-01) пересобирать prod-образ с +960MB слоем — нельзя без:

CI успешно билдит образ (camoufox fetch + libs работают в build) + приемлемый размер;
prod-деплой подтверждён post-demo.

Сам автор рекомендует do-not-merge до Phase 1. Согласен.

Код-уровень (для будущего merge): provider-agnostic BrowserFetcher (async ctx-mgr + page-recycle/15 + crash-recovery relaunch+retry + proxy), lazy import за try/except ImportError → app import не падает без Firefox; flag-gated dormant → нулевой runtime-риск сейчас. 9 mocked-тестов. Подход аккуратный. Но heavy image rebuild + auto-deploy за день до демо = человеческий/ops решение по таймингу.

→ +needs-human +blocked. Человек: мержить post-demo, когда CI-сборка образа зелёная и деплой-окно безопасно.

🔴 **HOLD — heavy Docker/deploy merge-gate (не мержу).** Мерж **auto-деплоит** и пересобирает prod-образ с `RUN python -m camoufox fetch` (**~960MB Firefox** на build-time) + 5 headless-Firefox libs + `mem_limit: 2.5g`. Риск — **build/deploy**, не runtime (фича behaviorally dormant: `scraper_fetch_mode` default `curl_cffi`, camoufox/playwright lazy-import). За день до демо (2026-06-01) пересобирать prod-образ с +960MB слоем — нельзя без: 1. CI успешно билдит образ (camoufox fetch + libs работают в build) + приемлемый размер; 2. prod-деплой подтверждён **post-demo**. Сам автор рекомендует `do-not-merge` до Phase 1. Согласен. Код-уровень (для будущего merge): provider-agnostic `BrowserFetcher` (async ctx-mgr + page-recycle/15 + crash-recovery relaunch+retry + proxy), lazy import за try/except ImportError → app import не падает без Firefox; flag-gated dormant → нулевой runtime-риск сейчас. 9 mocked-тестов. Подход аккуратный. Но **heavy image rebuild + auto-deploy за день до демо = человеческий/ops решение по таймингу.** → `+needs-human +blocked`. Человек: мержить post-demo, когда CI-сборка образа зелёная и деплой-окно безопасно.