gendesign/tradein-mvp/backend/app/services/scrapers
bot-backend 6044a1fe3c fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905)
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.

Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
  os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
  new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
  crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
  (one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).

Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.

Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884
2026-05-31 18:25:00 +03:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py feat(tradein): avito citywide + byrooms sweep mode (T6) (#848) 2026-05-31 06:13:19 +00:00
avito_detail.py fix(tradein): avito house-link из detail SSR — SERP-карточка JS-only (#871) (#873) 2026-05-31 08:27:47 +00:00
avito_exceptions.py fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) 2026-05-30 16:28:23 +00:00
avito_houses.py fix(tradein): decode Avito houses URL-encoded __preloadedState__ (#566) 2026-05-28 15:58:19 +05:00
avito_imv.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00
base.py refactor(tradein): unify kadastr_num -> cadastral_number (#732) 2026-05-31 15:41:52 +03:00
browser_fetcher.py fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) 2026-05-31 18:25:00 +03:00
cian.py fix(tradein): cian SERP warm-up GET для seed cookies (_CIAN_GK/_yasc) (T2) (#846) 2026-05-31 06:01:30 +00:00
cian_detail.py feat(scrapers): route cian sessions through mobile proxy (Refs #806) (#818) 2026-05-30 19:10:36 +00:00
cian_newbuilding.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00
cian_state_parser.py fix(tradein): cian _cianConfig concat-формат + auth из header-frontend (#639) (#819) 2026-05-30 19:08:01 +00:00
cian_valuation.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00
domclick.py fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup) 2026-05-31 17:21:30 +03:00
domclick_exceptions.py fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup) 2026-05-31 17:21:30 +03:00
n1.py feat(scrapers/n1): извлекать floor/total_floors из ведущего N/M-токена (Refs #795) (#802) 2026-05-30 17:31:13 +00:00
repair_state_normalizer.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
snapshot_writer.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
yandex_detail.py refactor(tradein): drop dead schema — sale_type_text column + audit_log (#731) (#893) 2026-05-31 13:13:43 +00:00
yandex_helpers.py fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602) 2026-05-27 13:11:22 +00:00
yandex_newbuilding.py feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484) 2026-05-23 15:28:34 +00:00
yandex_realty.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00
yandex_valuation.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00