fix(browser): camoufox reads AVITO_PROXY_URL → real avito data not soft-block stub (#905) #911

Merged

lekss361 merged 2 commits from fix/905-browser-proxy-env into main

2026-05-31 15:51:44 +00:00

bot-backend commented

2026-05-31 15:50:56 +00:00

Collaborator

Problem (found by prod smoke)

tradein-browser is up and camoufox fetches avito (health ok, no firewall), but it returned the ~400KB soft-block stub — preloadedState present but priceDetailed=0 (zero listings). Root cause: camoufox ran with no proxy (datacenter IP), and avito serves the stub to datacenter IPs.

server.py read SCRAPER_PROXY_URL, but the proxy in backend/.env.runtime is named AVITO_PROXY_URL (mobileproxy, #623) — the same var curl_cffi-avito already uses. So the var was empty in the browser container → no proxy → stub.

Fix

server.py now reads AVITO_PROXY_URL first, SCRAPER_PROXY_URL as a generic fallback. One-line env-name swap; the value is the working mobileproxy egress already on the host. Also fixed 2 pre-existing ruff lints in this file (browser/ was never linted — pre-commit ruff is scoped to backend/).

IP rotation stays backend-side: AVITO_PROXY_ROTATE_URL is hit from avito's _rotate_ip (Phase 1) and changes the egress IP behind the same proxy endpoint, so the browser needs no reconfig.

Verify after deploy

cd /opt/gendesign && git pull
docker exec -i tradein-backend sh < tradein-mvp/scripts/browser-smoke.sh

Expect real data: priceDetailed >0 (~50), bytes ~1.4MB, firewall 0. (Before this fix: priceDetailed=0, ~400KB stub.)

Dormant — scraper_fetch_mode still curl_cffi; this only affects the browser service's proxy. Refs #905, #883, #623

## Problem (found by prod smoke) `tradein-browser` is up and camoufox fetches avito (health ok, no firewall), but it returned the **~400KB soft-block stub** — `preloadedState` present but `priceDetailed=0` (zero listings). Root cause: camoufox ran with **no proxy** (datacenter IP), and avito serves the stub to datacenter IPs. `server.py` read `SCRAPER_PROXY_URL`, but the proxy in `backend/.env.runtime` is named **`AVITO_PROXY_URL`** (mobileproxy, #623) — the same var curl_cffi-avito already uses. So the var was empty in the browser container → no proxy → stub. ## Fix `server.py` now reads `AVITO_PROXY_URL` first, `SCRAPER_PROXY_URL` as a generic fallback. One-line env-name swap; the value is the working mobileproxy egress already on the host. Also fixed 2 pre-existing ruff lints in this file (browser/ was never linted — pre-commit ruff is scoped to backend/). IP rotation stays backend-side: `AVITO_PROXY_ROTATE_URL` is hit from avito's `_rotate_ip` (Phase 1) and changes the egress IP behind the same proxy endpoint, so the browser needs no reconfig. ## Verify after deploy ```sh cd /opt/gendesign && git pull docker exec -i tradein-backend sh < tradein-mvp/scripts/browser-smoke.sh ``` Expect **real data**: `priceDetailed` >0 (~50), `bytes` ~1.4MB, `firewall 0`. (Before this fix: priceDetailed=0, ~400KB stub.) Dormant — `scraper_fetch_mode` still `curl_cffi`; this only affects the browser service's proxy. Refs #905, #883, #623

bot-backend added 2 commits 2026-05-31 15:50:56 +00:00

fix(browser): read AVITO_PROXY_URL for camoufox proxy (#905 ) 57acec215f

Prod smoke: tradein-browser fetched avito but got the ~400KB soft-block stub
(preloadedState present but priceDetailed=0, firewall=0) — because camoufox ran
with NO proxy (datacenter IP). Root cause: server.py read SCRAPER_PROXY_URL, but
the proxy in backend/.env.runtime is named AVITO_PROXY_URL (mobileproxy, #623) —
the same var curl_cffi-avito uses. So the var was empty in the browser container.

Fix: read AVITO_PROXY_URL first, SCRAPER_PROXY_URL as a generic fallback. One-line
env-name swap; the value is the working mobileproxy egress already on the host.

After deploy, re-run tradein-mvp/scripts/browser-smoke.sh — expect real data
(priceDetailed>0, ~1.4MB, firewall 0). IP rotation (AVITO_PROXY_ROTATE_URL) stays
backend-side and keeps working (same proxy endpoint, changing egress IP).

Refs #905, #883, #623

style(browser): fix pre-existing ruff lints in server.py (RUF100, E501) 3527a6bc8d

Drive-by: browser/server.py was never linted (pre-commit ruff is scoped to
backend/), so #909 left an unused noqa + a 101-char line. Cleaned up while
editing this file for the proxy fix.