Commit graph

5 commits

Author SHA1 Message Date
683cb50245 fix(tradein): wire validated Cian login selectors + 2-step flow + review fixes (#639)
Провалидировано вживую: email+пароль без SMS. pre_click один->список
(LoginButton->AnotherAccountBtn->SwitchToEmailAuthBtn), 2-step «Введите пароль»
retry, реальные селекторы (input[name=username|password], ContinueAuthBtn).
M1: 502 detail без page_url. M4: BrowserFetcher импорт наверх.
2026-05-31 21:17:09 +03:00
1f18a2ccdc feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639)
Variant B: headless camoufox логинится email+паролем (без SMS), извлекает
auth-cookies → verify → save_session. Browser-сервис получает generic POST /login
(url/selectors/success_cookie в body), Cian-специфика — в config (env-overridable).
Авто-refresh в scheduler — follow-up PR после ручной валидации логина на проде.
2026-05-31 20:57:39 +03:00
6044a1fe3c fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905)
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.

Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
  os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
  new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
  crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
  (one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).

Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.

Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884
2026-05-31 18:25:00 +03:00
5fb367295a feat(browser): split camoufox into tradein-browser container, connect-mode (#905)
Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service.
camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/
proxy configured server-side); the backend connects remotely via playwright, so it
no longer ships Firefox.

- NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000;
  Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix)
- BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint);
  recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry)
- backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer
- pyproject: remove camoufox[geoip] from backend, promote playwright to main deps
- docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove
  mem_limit from backend, backend depends_on browser (service_started)
- config: browser_ws_endpoint default ws://tradein-browser:3000/playwright
- tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass)

Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change.
launch_server signature verified (**kwargs→launch_options, port passes through).

Refs #905, #883, #884, #899
2026-05-31 17:24:35 +03:00
e03f4e4534 feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884)
Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox):
async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy
imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT
(no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile:
+5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored
-> re-resolved at build.) 9 mocked tests pass, ruff clean.

WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image;
merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify.

Refs #884, #883
2026-05-31 15:41:56 +03:00