Group F4d (epic #2277, parent #2352). Migrated the ad-hoc BrowserFetcher
construction in providers/domclick/serp.py::fetch_city to the shared
build_browser_fetcher(config, source) helper from Foundation (#2358).
Scope actually touched:
- serp.py: replaced the local BrowserFetcher(source="domclick", endpoint=...)
construction with build_browser_fetcher(self._config, "domclick").
- detail.py: NO changes needed. Grepped for BrowserFetcher(/AsyncSession(/
curl_cffi and found none - fetch_detail() receives an already-constructed
browser_fetcher from the caller (orchestrator owns the session lifecycle,
per its own module docstring), so there was nothing to migrate here.
Caveat verified (same class as avito, per issue instructions): the old
call site omitted use_pool, so it implicitly defaulted to False via the
BrowserFetcher class default. build_browser_fetcher(config, source) reads
use_pool=config.use_proxy_pool_browser instead - a real difference in the
value passed in. However this call site does not pass proxy_provider
(stays the default None), and BrowserFetcher._pool_proxy computes
`use_pool and proxy_provider is not None` before ever using the pool - so
with proxy_provider=None the effective behavior is identical regardless
of what use_pool evaluates to (confirmed by reading browser_fetcher.py
directly, matching the build_browser_fetcher docstring's own note that
domclick is one of the providers not yet wired to the proxy pool, #2160
P4). No observable behavior change. Documented inline at the call site.
Tests: domclick-specific suite (91 tests: golden parity, sweep, admin
ingest parity, detail + detail kit parity) all green, plus cross-provider
test_scraper_kit_pipeline_parity2.py (8 tests). Full backend suite:
3278 passed, 6 skipped, 1 pre-existing flake (test_search_cache_hit,
#2208) - no new failures.
code-reviewer отметил: httpx>=0.28 больше не принимает dict в proxies= (только
одиночный proxy= строкой). Docstring ошибочно называл эту функцию
"httpx-совместимой" — уточнено что это только для curl_cffi, чтобы будущий
мигратор F4a-d не словил TypeError.
Найдено при code-review этого же PR (F1, #2349) и независимо подтверждено
F3-аудитом (#2351): run_cian_city_sweep's houses-фаза звала
fetch_newbuilding(zhk_url) без config=, хотя config — mandatory параметр
enclosing-функции, тривиально в scope. Сейчас dormant (USE_KIT_SCHEDULER=False +
cian_city_sweep.enabled=false в SQL seed), но при включении флага/расписания
каждый house молча падал бы AssertionError (перехватывается except → houses_failed,
sweep не крашится целиком, но newbuilding-enrichment был бы 100% failing silently).
code-reviewer отметил: формулировка "PROD DEFAULT" в докстрингах могла ввести в
заблуждение — прод-путь сегодня всё ещё легаси (avito_detail_backfill.py
импортирует build_warmed_session из legacy-модуля), этот фикс — advance-prep,
не изменение живого прод-поведения.
Group D of the scraper_kit migration epic (#2277). Both legacy modules had no
kit-side equivalent yet (per audit Scraper_Kit_Legacy_Dependency_Audit_0703),
unlike the other 24/26 files which were already strangler-duplicated.
- scraper_kit/providers/domclick/detail.py — Layer B detail-card enrichment
(parse_detail_html/fetch_detail/save_detail_enrichment), no ScraperConfig
injection needed (fetch_detail takes an already-open browser_fetcher).
- scraper_kit/providers/ekb_geoportal/{__init__,client}.py — new subpackage,
first non-listing-site provider in scraper_kit. Legacy module had zero
app.* coupling (only httpx), so the port is near-verbatim.
Parity proven via tests/support/parity.py (assert_parity) against the exact
legacy functions on realistic fixtures (live-card SSR JSON, GeoJSON
FeatureCollection).
Callers of both legacy modules are unchanged (out of scope — ekb_geoportal_ingest
switch is #2310; the only other domclick_detail caller is
scripts/ingest_domclick_jsonl.py, also untouched).
- listings.phones: миграция 166 обнуляет данные; cian.py + scraper-kit
serp.py (golden-parity зеркало) перестают читать offer.get("phones") на
входе — write-path выключен у источника, колонка/поле в модели остаются.
Мёртвая запись "phones" в LISTING_FIELD_PRIORITY убрана.
- trade_in_estimates.client_name/client_phone: миграция 167 дропает колонки;
синхронно убраны из TradeInEstimateInput, обоих INSERT в estimator.py
(основной путь + _empty_estimate), frontend TradeInEstimateInput и
CRM-блока EstimateForm (поля "Имя клиента"/"Телефон").
- sentry_scrub.py и management_companies.phones не тронуты — вне scope.
Fixed ANCHOR_TIMEOUT_SEC=240 guillotined every cian anchor when running via
USE_PROXY_POOL_BROWSER=true: each SERP fetch goes through camoufox with relaunch
on proxy swap (13-45s/page), and an anchor = 4 room-buckets x pages_per_anchor
SERP fetches + detail_top_n details + houses-enrich >> 240s. save_listings never
ran, lots_fetched=0, and after 3 consecutive skips the run was mark_banned.
Introduces _cian_anchor_timeout_s() mirroring the avito/yandex/domclick scaling
pattern: max(ANCHOR_TIMEOUT_SEC, buckets*pages*(delay+per_serp) + details*per_detail
+ houses_budget). Defaults -> 1580s worst-case watchdog (hang guard, not expected
duration). Applied to both the scraper-kit copy (strangler target) and the active
app.services.scrape_pipeline copy that run 581 actually hit.
