fix(browser): bump fetch wait 2500→6000ms so avito listings hydrate (#905) — migration proven #912

Merged

lekss361 merged 1 commit from fix/905-browser-wait-ms into main

2026-05-31 16:20:42 +00:00

bot-backend commented

2026-05-31 16:17:23 +00:00

Collaborator

The actual root cause (found by prod SSH debug)

The camoufox fetch returned a 274KB generic avito shell with 0 listings — but it was not a block: status 200, correct final_url, no captcha/datadome/firewall markers, proxy active. A direct camoufox run in the prod browser container (headless=True + mobileproxy) with a 5s wait returned the full SERP: 3.2MB, priceDetailed=50.

Avito hydrates the SERP listings into the DOM client-side after domcontentloaded. At the old BROWSER_WAIT_MS=2500 the page.content() snapshot caught only the pre-hydration shell → 0 listings. 5–6s → full data.

Fix

BROWSER_WAIT_MS default 2500 → 6000 (5s proven full + 1s margin); config.py browser_wait_ms 2500 → 6000 to match. Tunable via env BROWSER_WAIT_MS.

This is the last blocker — the whole camoufox + mobileproxy migration works end-to-end (real avito data, beating the curl_cffi soft-block). NOT headless detection (headless=True returns real data), so the earlier xvfb/headless=virtual idea is unnecessary and abandoned.

Verify after deploy

docker exec -i tradein-backend sh < tradein-mvp/scripts/browser-smoke.sh

Expect: priceDetailed ~50, bytes ~3.2MB, firewall 0.

Dormant — scraper_fetch_mode still curl_cffi. Refs #905, #883

## The actual root cause (found by prod SSH debug) The camoufox fetch returned a 274KB generic avito shell with **0 listings** — but it was **not** a block: status 200, correct `final_url`, no captcha/datadome/firewall markers, proxy active. A direct camoufox run in the prod browser container (headless=True + mobileproxy) with a **5s wait** returned the **full SERP: 3.2MB, priceDetailed=50**. Avito hydrates the SERP listings into the DOM **client-side after `domcontentloaded`**. At the old `BROWSER_WAIT_MS=2500` the `page.content()` snapshot caught only the pre-hydration shell → 0 listings. 5–6s → full data. ## Fix `BROWSER_WAIT_MS` default 2500 → **6000** (5s proven full + 1s margin); `config.py browser_wait_ms` 2500 → 6000 to match. Tunable via env `BROWSER_WAIT_MS`. This is the **last blocker** — the whole camoufox + mobileproxy migration works end-to-end (real avito data, beating the curl_cffi soft-block). NOT headless detection (headless=True returns real data), so the earlier xvfb/`headless=virtual` idea is unnecessary and abandoned. ## Verify after deploy ```sh docker exec -i tradein-backend sh < tradein-mvp/scripts/browser-smoke.sh ``` Expect: `priceDetailed` ~50, `bytes` ~3.2MB, `firewall 0`. Dormant — `scraper_fetch_mode` still `curl_cffi`. Refs #905, #883

bot-backend added 1 commit 2026-05-31 16:17:24 +00:00

fix(browser): bump fetch wait 2500→6000ms so avito listings hydrate (#905 ) f09d9737ce

Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning
a 274KB generic avito shell with 0 listings — NOT a block/captcha (no
captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates
the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms
wait the page.content() snapshot still had only the generic shell. At 5-6s it's
the full SERP: 3.2MB, priceDetailed=50.

Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py
browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works —
this was the last blocker. NOT headless detection (headless=True returns real
data); the abandoned xvfb/headless=virtual change is unnecessary.

Refs #905, #883