Avito leaks addresses with house number glued to district marker without
separator («…Савкова, 29р-н Академический»), breaking house-number extraction
and same-house matching (41 unmatched avito listings on one street).
_clean_address now inserts ", " between a house-number token (digits +
optional Cyrillic letter + optional /N|сN|кN|корпус|стр) and a following
marker (р-н, мкр, район, г., пос, снт, …).
- avito.py: idempotent _deglue_house_marker step (CSS/tail-strip kept)
- tests: de-glue positive/negative/idempotency (24 passed)
- data/sql/124_deglue_avito_addresses.sql: idempotent backfill for stored rows
normalize.py НЕ тронут (общий fingerprint матчинга, #1534).
Закрывает avito-часть #1773 (cian-без-номера + дедуп — отдельно, зона cian).
Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched
because the detail fetcher only had a same-session 429 short-retry (no reconnect).
429s are persistent: keep-alive reuses one backconnect exit IP across sequential
detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769):
widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP)
on 429 exhaustion before raising. Caller session never touched (as with 403 path).
Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36
(deep pagination, backconnect conn-limit). Widen the 429 retry budget + one
reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip
a blocked bucket and continue rather than aborting the run; only N consecutive
blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.
fetch_detail raised on the first 403/429, aborting the detail backfill (run 191
enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active.
Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy,
rebuild an ephemeral session (fresh exit IP) and retry up to N before raising;
short-retry 429. Never close/rebuild the caller-provided shared session. Also
add the missing HTTP-200 firewall-interstitial check on the curl detail path.
Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and
stored on every scrape. Skips the redundant per-day listings_snapshots write
when a listing's card is unchanged. listings.last_seen_at/is_active still
bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh
gating (C2).
Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate
sequentially newest-first and stop once cards fall below the watermark, skipping
deep pagination (and the 403 clusters it triggers). since=None keeps the exact
exhaustive bisection walk. No scheduler/schedule change here — wiring follows
in a separate PR.
PR #1764's reconnect-retry was dead code in prod: the gate excluded configs
with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL
(what curl_cffi egresses through) AND a leftover changeip URL for the separate
auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403
still aborted the sweep (run 212). The curl session always uses scraper_proxy_url,
so session-rebuild rotates the exit IP regardless of changeip config.
Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or
firewall interstitial aborted the entire avito_full_load sweep via
AvitoBlockedError -> mark_banned. Each new connection through backconnect
gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to
_AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip
rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice,
not predictedPrice.predictedPrice (that key does not exist). _predicted_prices
read the wrong outer key, so predicted_price_rub/_min/_max were None for every
offer (prod: pp=0 across all 6189 yandex rows). Verified live against the
gate-API 2026-06-18.
The rich-field test fixture used the same wrong wrapper, so the test passed
vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice
shape — with the old code the assertions now fail, confirming non-vacuity.
Replace the system-curl SOCKS5 subprocess transport with the shared
BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not
tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl
got the proxy IP flagged.
One BrowserFetcher is opened per scraper session in __aenter__ and reused
across all page fetches. _http_get now fetches via the browser, extracts the
gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body
of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so
all existing callers and the tarpit retry/rotate logic keep working unchanged.
URL building, _parse_gate_json, pagination, combos and field mapping are
untouched. Removed the dead curl subprocess constants/method.
Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a
loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected,
call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.)
return [] immediately without rotation. Mirrors the existing exhaustive-path
pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success,
captcha→rotate→success, exhausted retries→empty, 404→no-rotate.
Добавлено поле kitchen_area_m2 в DetailEnrichment dataclass,
извлечение через _parse_float(offer.get("kitchenArea")) в fetch_detail,
и COALESCE(CAST(:ka AS double precision), kitchen_area_m2) в UPDATE listings
внутри save_detail_enrichment.
Имя поля kitchenArea — стандартное cian offer API (live-verify не делался,
прокси недоступен в dev-окружении).
Closes#1553
Корень проблемы: avito_houses.py обходил канонический matching-путь через
прямой INSERT ON CONFLICT(source, ext_house_id), создавая дубли для физически
одного дома (~33% таблицы houses = ~4479 лишних строк).
Forward fix: upsert_house/upsert_house_with_url → _persist_house(db, h, url),
который вызывает match_or_create_house (3-tier: source_exact→fingerprint→geo→new,
с pg_advisory_xact_lock) и затем UPDATE с enrichment-полями. avito-авторитетные
поля (rating, developer, технические хар-ки) перезаписываются напрямую;
address/lat/lon/year_built — через COALESCE чтобы не затирать canonical-данные.
Merge-миграция 108: cluster_key по cadastral/address/geo → canonical (max
не-NULL полей, geom присутствует, min(id)) → репойнт всех 11 зависимых таблиц
с обработкой UNIQUE-коллизий → DELETE дублей → backfill house_sources/aliases.
Валидационный yandex_newbuilding_sweep (run 80) заполнял ratings_count /
text_reviews_count / developer_name / lat-lon, но rating оставался NULL для всех
ЖК. Live-диагностика (прод-браузер, ЖК Татлин/Рио 2026-06-15): рейтинг рендерится
как «4.3из 5» / «4,5из 5» — БЕЗ пробела перед «из» (nbsp внутри «из 5»), а
RE_RATING требовал \s+ перед «из» → match=None.
- RE_RATING: (\d[.,]\d)\s+из\s+5 → (\d[.,]\d)\s*из\s+5 (пробел перед «из» опционален).
\s уже матчит nbsp, поэтому «из\xa05» обрабатывался; ломал именно отсутствующий
разделитель перед «из».
- test_parse_rating_no_space_before_iz: регрессия на обе дрейфовые формы.
Существующие rating-тесты («4,3 из 5» с пробелом) остаются зелёными (\s* ⊇ \s+).
Refs #974
Cian переименовал MFE ЖК-карточки и переместил reliability-контейнер, из-за чего
fetch_newbuilding получал initialState=None → failed_fetch для всех домов
(verified live 2026-06-15 на прод-браузере через мобильный egress):
- MFE: 'newbuilding-card-desktop-fichering-frontend' → 'newbuilding-card-desktop-frontend'
(старое имя на странице отсутствует → extract_state=None → backfill succeeded=0)
- reliability: initialState.reliability → initialState.reliabilityState.reliability
(inner dict сохранил форму {checkStatus, details[], banner, actions})
Проверено на ЖК «Светлый» (nb 17556): newbuilding.id, realtyValuation chart (27 точек),
reliabilityState.reliability.checkStatus + 2 details — всё извлекается.
test_fetch_newbuilding_parses_state: fixture обновлён под reliabilityState.reliability.
Полный suite зелёный (1779 passed; 2 несвязанных env-only локальных падения —
cian_valuation cache / search_api auth — в CI проходят).
Refs #972
curl_cffi получает zhk-*.cian.ru без initialState; tradein-browser (camoufox) тянет
страницу с 1.17MB initialState. Заменяем транспорт только для page-fetch;
session= param сохранён для backward-compat вызывающих. resolve_cian_zhk_url_via_search
остаётся на curl_cffi SERP. Тесты: 21 passed.
The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the
318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL)
unfetchable -> the newbuilding-enrichment backfill matched 0 houses.
Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP
and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the
<h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries
promo/recommendation zhk-* links before the title that would otherwise resolve
the wrong ЖК and silently corrupt enrichment). Validated against the real
2.81MB prod SERP + an adversarial poisoned-recommendation test.
Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR
cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching,
rate-limited + resumable + idempotent. Keep the old resolver (deprecated).
Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully
enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode
anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy
(mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) +
resolver hardened against wrong-ЖК. 19 tests green.
Refs #972.
