First XHR after goto(origin) intermittently hit "NetworkError when
attempting to fetch resource" (page net stack/anti-bot not ready),
healed only by the outer re-navigation retry (~30-45s/house in the
house_imv_backfill, #562).
_fetch_json_once now:
- settles FETCH_JSON_SETTLE_MS (default 1200, was hardcoded 500) — fewer
first-fails;
- wraps the in-page fetch() in a JS retry loop (FETCH_JSON_INPAGE_RETRIES
default 1, FETCH_JSON_RETRY_DELAY_MS default 800) that retries ONLY on
network throw, never on HTTP status (4xx/5xx short-circuit, caller
decides). An in-page retry costs ~retryDelayMs vs the ~30-45s outer
re-navigation. Last error re-thrown — outer crash-retry contract intact.
/fetch (SERP) path untouched. +2 tests (settle ms, retry params).
Refs #1917, #562
- browser/server.py: GET /pacing и PUT /pacing — read/write _MIN_PAGE_INTERVAL_BY_PROVIDER
in-memory; валидация source in PROVIDERS, interval_s >= 0; логирует изменения; сбрасывается
к env-дефолту на рестарте by design
- backend/admin.py: GET /scraper/pacing — прокси к browser /pacing, PacingResponse pydantic;
PUT /scraper/pacing/{source} — прокси PUT с валидацией ge=0 le=120; 502/503 при недоступности
- backend/admin.py: GET /scraper/data-quality — single-pass FILTER-агрегаты по listings WHERE
is_active GROUP BY source (description/photo_urls/address/lat/lon/kitchen_area_m2/living_area_m2/
ceiling_height/ceiling_height_m/metro_stations); houses (total/avito_validated_at%/rating_score%/
house_type%); house_reviews count
- browser/test_server_pacing.py: 10 новых тестов GET+PUT /pacing (16 total, все зелёные)
- backend/tests/test_scraper_admin_apis.py: тесты pacing-прокси + data-quality shape/pct-range
(21 total, все зелёные)
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning
a 274KB generic avito shell with 0 listings — NOT a block/captcha (no
captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates
the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms
wait the page.content() snapshot still had only the generic shell. At 5-6s it's
the full SERP: 3.2MB, priceDetailed=50.
Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py
browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works —
this was the last blocker. NOT headless detection (headless=True returns real
data); the abandoned xvfb/headless=virtual change is unnecessary.
Refs #905, #883
Drive-by: browser/server.py was never linted (pre-commit ruff is scoped to
backend/), so #909 left an unused noqa + a 101-char line. Cleaned up while
editing this file for the proxy fix.
Prod smoke: tradein-browser fetched avito but got the ~400KB soft-block stub
(preloadedState present but priceDetailed=0, firewall=0) — because camoufox ran
with NO proxy (datacenter IP). Root cause: server.py read SCRAPER_PROXY_URL, but
the proxy in backend/.env.runtime is named AVITO_PROXY_URL (mobileproxy, #623) —
the same var curl_cffi-avito uses. So the var was empty in the browser container.
Fix: read AVITO_PROXY_URL first, SCRAPER_PROXY_URL as a generic fallback. One-line
env-name swap; the value is the working mobileproxy egress already on the host.
After deploy, re-run tradein-mvp/scripts/browser-smoke.sh — expect real data
(priceDetailed>0, ~1.4MB, firewall 0). IP rotation (AVITO_PROXY_ROTATE_URL) stays
backend-side and keeps working (same proxy endpoint, changing egress IP).
Refs #905, #883, #623
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.
Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
(one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000).
Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.
Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884
