bot-backend
6044a1fe3c
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.
Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
(one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).
Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.
Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884
104 lines
4.6 KiB
Python
104 lines
4.6 KiB
Python
"""browser_fetcher.py — HTTP-клиент к tradein-browser сервису (#884/#905).
|
||
|
||
Тонкий клиент над tradein-browser контейнером, который запускает AsyncCamoufox
|
||
локально и экспонирует HTTP API (POST /fetch).
|
||
|
||
Публичный интерфейс не изменился:
|
||
|
||
async with BrowserFetcher() as fetcher:
|
||
html = await fetcher.fetch("https://example.com")
|
||
|
||
Внутреннее устройство: httpx.AsyncClient + POST к settings.browser_http_endpoint.
|
||
Recycle, crash-recovery и управление браузером живут на стороне сервера (server.py).
|
||
При HTTPError / ConnectError делает одну повторную попытку после короткой паузы,
|
||
затем пробрасывает исключение.
|
||
|
||
Архитектура выбрана потому, что playwright WS-сервер (launch_server) несовместим
|
||
с playwright >=1.45: ``browserServerImpl.js`` отсутствует → MODULE_NOT_FOUND.
|
||
Локальный AsyncCamoufox + HTTP — работающая альтернатива.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import asyncio
|
||
import logging
|
||
|
||
import httpx
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
_RETRY_SLEEP_S: float = 1.0
|
||
_HTTP_TIMEOUT_S: float = 120.0 # навигация медленная → щедрый таймаут
|
||
|
||
|
||
class BrowserFetcher:
|
||
"""Async context manager: HTTP-клиент к tradein-browser HTTP-сервису.
|
||
|
||
Использование::
|
||
|
||
async with BrowserFetcher() as fetcher:
|
||
html = await fetcher.fetch("https://example.com")
|
||
"""
|
||
|
||
def __init__(self) -> None:
|
||
self._client: httpx.AsyncClient | None = None
|
||
self._endpoint: str | None = None
|
||
|
||
# ── lifecycle ──────────────────────────────────────────────────────────────
|
||
|
||
async def __aenter__(self) -> BrowserFetcher:
|
||
from app.core.config import settings
|
||
|
||
self._endpoint = settings.browser_http_endpoint
|
||
self._client = httpx.AsyncClient(timeout=_HTTP_TIMEOUT_S)
|
||
logger.info("BrowserFetcher: клиент создан, endpoint=%s", self._endpoint)
|
||
return self
|
||
|
||
async def __aexit__(self, *_: object) -> None:
|
||
if self._client is not None:
|
||
await self._client.aclose()
|
||
self._client = None
|
||
logger.debug("BrowserFetcher: клиент закрыт")
|
||
|
||
# ── public API ─────────────────────────────────────────────────────────────
|
||
|
||
async def fetch(self, url: str) -> str:
|
||
"""Запрашивает HTML страницы через tradein-browser HTTP-сервис.
|
||
|
||
При HTTPError или ConnectError делает одну повторную попытку после
|
||
короткой паузы. Остальные исключения всплывают к вызывающему коду.
|
||
|
||
Returns:
|
||
Полный HTML-контент страницы.
|
||
"""
|
||
assert self._client is not None, "BrowserFetcher: используй как async context manager"
|
||
assert self._endpoint is not None, "BrowserFetcher: endpoint не задан"
|
||
|
||
try:
|
||
return await self._post_fetch(url)
|
||
except (httpx.HTTPError, httpx.TransportError) as exc:
|
||
logger.warning(
|
||
"BrowserFetcher: ошибка запроса (%s), retry через %.1fs: %s",
|
||
type(exc).__name__,
|
||
_RETRY_SLEEP_S,
|
||
url,
|
||
)
|
||
await asyncio.sleep(_RETRY_SLEEP_S)
|
||
return await self._post_fetch(url)
|
||
|
||
# ── internal ───────────────────────────────────────────────────────────────
|
||
|
||
async def _post_fetch(self, url: str) -> str:
|
||
"""Один HTTP POST к /fetch эндпоинту сервиса."""
|
||
assert self._client is not None
|
||
assert self._endpoint is not None
|
||
|
||
resp = await self._client.post(
|
||
f"{self._endpoint}/fetch",
|
||
json={"url": url},
|
||
)
|
||
resp.raise_for_status()
|
||
data: dict[str, str] = resp.json()
|
||
html = data["html"]
|
||
logger.debug("BrowserFetcher: fetch OK url=%r html_len=%d", url, len(html))
|
||
return html
|