curl_cffi получает zhk-*.cian.ru без initialState; tradein-browser (camoufox) тянет
страницу с 1.17MB initialState. Заменяем транспорт только для page-fetch;
session= param сохранён для backward-compat вызывающих. resolve_cian_zhk_url_via_search
остаётся на curl_cffi SERP. Тесты: 21 passed.
The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the
318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL)
unfetchable -> the newbuilding-enrichment backfill matched 0 houses.
Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP
and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the
<h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries
promo/recommendation zhk-* links before the title that would otherwise resolve
the wrong ЖК and silently corrupt enrichment). Validated against the real
2.81MB prod SERP + an adversarial poisoned-recommendation test.
Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR
cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching,
rate-limited + resumable + idempotent. Keep the old resolver (deprecated).
Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully
enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode
anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy
(mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) +
resolver hardened against wrong-ЖК. 19 tests green.
Refs #972.
950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to
ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing
fetch_newbuilding + save_newbuilding_enrichment over each, populating
houses_price_dynamics + house_reliability_checks (+ house_reviews when present).
- Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe
(price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC
tiebreaker, reviews ON CONFLICT (source,ext_review_id)).
- Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house
(one captcha/failure logs + continues, never aborts the batch).
- limit= for bounded proof runs; sizing counters (total/fetchable/pending).
Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate
drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on
LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None
(bool excluded before int). Fixes the live enrichment crash, not just backfill.
house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК
initialState — extending fetch_newbuilding for them is a documented follow-up.
Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1
reliability row, idempotency proven across re-runs. Refs #972.
