Cian secondary-market detail (bti_data) и Valuation Calculator (house_info/
managementCompany/houseId) парсились, но выбрасывались — комментарий "это
задача Stage 6 (houses)" так и не был выполнен.
- cian/detail.py::save_detail_enrichment принимает инжектируемый matcher
(HouseMatcher, optional) → новый _persist_cian_bti_house резолвит дом через
match_or_create_house (address/geo листинга, mirror avito/houses.py::
_persist_house) и пишет BTI-эксклюзивные колонки из 020_houses_alter_cian.sql
(series_name/entrances/flat_count/is_emergency/heat_supply_type/
gas_supply_type/overlap_type) через COALESCE(new, existing).
- cian/valuation.py::_save_to_cache получает уже резолвленный house_id (read-only
match_house_readonly, estimator.py) → новый _persist_cian_valuation_house пишет
management_company_id (UPSERT management_companies) + cian_internal_house_id
(filters.houseId) COALESCE(new, existing), плюс houseInfo.items-производные
поля (год/тип/этажность/газ/отопление/перекрытия/подъезды/квартиры/
аварийность/детская площадка/лифты) COALESCE(existing, new) — валюация не
авторитетный источник для них (conflict_resolution.HOUSE_FIELD_PRIORITY).
- Оба пути best-effort: SAVEPOINT (db.begin_nested) изолирует сбой резолва/
записи дома от основной save-транзакции; house_id=None / matcher=None /
безномерный адрес (P1 no_house_number) — graceful no-op, без исключений.
- matcher прокинут в реальные call sites: pipeline.py (cian city-sweep +
full-load — уже был в scope), cian_history_backfill.py, cian_price_history.py,
admin.py ad-hoc endpoint.
Gap (нет чистого маппинга на существующую колонку houses — не создавали новых
колонок): bti.houseData.{demolishedInMoscowProgramm, heatIndex,
houseOverhaulFundType, lifts (недифференцированный total)}; houseInfo.items
{Мусоропровод, Реновация, Спортивная площадка, Фонд капремонта}.
10 новых тестов (tests/test_cian_bti_house_persist.py), MagicMock db, без
реальной БД — mirror test_snapshot_writer.py / test_extval_house_id_write_path.py.
Migrates legacy app.services.scrapers.* imports to scraper_kit equivalents for
house_imv_backfill.py, avito_detail_backfill.py, cian_history_backfill.py,
ekb_geoportal_ingest.py, and yandex_detail_backfill.py, proving parity via
tests/support/parity.assert_parity per the epic's gate (#2304).
newbuilding_enrich_backfill.py and yandex_newbuilding_sweep.py are left fully
on legacy imports: both call into scraper_kit.providers.{cian,yandex}.newbuilding,
which construct BrowserFetcher(source=...) without the now-mandatory endpoint=
kwarg (issue #2322, verified still open against the actual provider source, not
just issue status) -- no caller-side fix is possible, matching Group A's (#2305)
precedent for the same bug in admin.py.
Config-gated kit function footguns found and fixed (Group B #2306 pattern):
- avito fetch_detail's backconnect-on-403 retry silently drops when config=
is omitted -- now passes config=RealScraperConfig() explicitly, with a
regression test proving the gate.
- kit AvitoScraper's constructor now requires ScraperConfig positionally --
wired via RealScraperConfig(), which _rotate_ip() reads for
avito_proxy_rotate_url.
- BrowserFetcher(source=...) call sites (house_imv_backfill, avito/cian
detail backfills) now pass the mandatory endpoint=settings.browser_http_endpoint.
New footgun discovered (NOT #2322, flagged for follow-up): kit's
build_warmed_session() builds its curl_cffi session via _build_detail_session()
with no config parameter at all, unlike fetch_detail -- migrating it would
silently drop the sticky MGTS-proxy egress on avito_detail_backfill's
warm-batch path (the prod default). Left build_warmed_session/
_AVITO_WARM_SEARCH_URL on legacy imports, documented inline.
Refs #2310
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
