house_type-нормализация = реальный фикс estimator soft-penalty: yandex SERP
писал building.buildingType как есть (SCREAMING — MONOLIT/BRICK/PANEL/...),
а estimator штрафует аналог с house_type != target. 'MONOLIT' никогда не равен
каноничному 'monolith' → ~70% yandex-аналогов получали ложный штраф и фактически
выпадали из скоринга. Новый модуль house_type_normalizer.normalize_house_type
приводит yandex SCREAMING и cian camelCase (reuse в #2008) к канону
{panel, brick, monolith, monolith_brick, block, wood}; unknown/'other'/'' → None
(NULL нейтрально для estimator, не штрафуется). Backfill существующих yandex-строк
миграцией 140 (идемпотентный UPDATE под тот же маппинг).
kitchen_area_m2 / ceiling_height_m promote из raw_payload в колонки — для
coverage-дашборда и matching, НЕ valuation-фича (estimator их пока не читает —
отдельный follow-up). ceiling пишется в ОБЕ колонки (ceiling_height 019 — её
читает coverage-дашборд admin.py + yandex_detail/cian_detail; ceiling_height_m
111 — живая avito-колонка) через COALESCE в ON CONFLICT, чтобы SERP re-scrape
без полей не затирал detail-enrichment (avito поведение сохранено).
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice,
not predictedPrice.predictedPrice (that key does not exist). _predicted_prices
read the wrong outer key, so predicted_price_rub/_min/_max were None for every
offer (prod: pp=0 across all 6189 yandex rows). Verified live against the
gate-API 2026-06-18.
The rich-field test fixture used the same wrong wrapper, so the test passed
vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice
shape — with the old code the assertions now fail, confirming non-vacuity.
Replace the system-curl SOCKS5 subprocess transport with the shared
BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not
tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl
got the proxy IP flagged.
One BrowserFetcher is opened per scraper session in __aenter__ and reused
across all page fetches. _http_get now fetches via the browser, extracts the
gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body
of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so
all existing callers and the tarpit retry/rotate logic keep working unchanged.
URL building, _parse_gate_json, pagination, combos and field mapping are
untouched. Removed the dead curl subprocess constants/method.
Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a
loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected,
call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.)
return [] immediately without rotation. Mirrors the existing exhaustive-path
pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success,
captcha→rotate→success, exhausted retries→empty, 404→no-rotate.
