scraper_kit.providers.cian.valuation.estimate_via_cian_valuation (and
session.load_session) already existed with mandatory config: ScraperConfig
(no default) — this was ground prepared ahead of Group E2, not new work here.
This change proves it's safe to switch estimator.py's import to it (#2337):
- Extends the existing _parse_valuation_state golden-parity test with a
full-pipeline parity test (cache-miss -> load_session -> curl_cffi fetch ->
extract_state -> auth-check -> parse -> sanity-check), wired with
config=RealScraperConfig() per the #2306 convention
(app/services/cian_price_history.py).
- Adds a regression guard proving estimate_via_cian_valuation raises TypeError
when config= is omitted, and documents why that must stay mandatory:
estimator.py's call site (line ~3143) sits inside a blanket
`except Exception` that only does `logger.warning(..., exc)` (no exc_info,
no logger.exception) -- below GlitchTip's ERROR-level capture threshold
(see LoggingIntegration(event_level=logging.ERROR) in app/main.py), so a
TypeError from a missing config= would be logged but invisible in
GlitchTip/Sentry, indistinguishable from ordinary graceful degradation.
- Documents an asymmetry vs. the task's initial assumption: kit's
load_session requires config= mandatorily, but mark_session_invalid does
NOT take a config parameter at all (confirmed by inspection, not a
footgun -- just noting the actual contract).
- Confirms app/services/estimator.py is not the only real caller:
app/tasks/cian_history_backfill.py also imports estimate_via_cian_valuation
directly and deliberately stays on the legacy import for now (see its own
comment referencing issue #2308) -- out of scope for this change, flagged
for whoever picks up cian_history_backfill's migration.
Full backend test suite: 3211 passed, 6 skipped, 1 pre-existing unrelated
failure (test_search_api.py::test_search_cache_hit, 401 vs 200 auth issue,
reproduces identically on main without this change).
Group E3 recon confirmed `scraper_kit.providers.yandex.valuation` is already a
strangler-copy (#2133) and already used via admin.py's debug route (#2305,
Group A) with the established config=RealScraperConfig()/delay_provider=
get_scraper_delay convention. estimator.py (the real caller, #2337's scope)
still imports the legacy module directly with no arguments -- not touched here.
Adds regression coverage for the two confirmed footguns before #2337 switches
that call site:
- mandatory `config: ScraperConfig` raises TypeError when omitted (same
discipline as the cian_valuation footgun, #2335)
- optional `delay_provider` silently falls back to hardcoded 5.0s when not
wired, discarding admin-tuned anti-ban throttling (scraper_settings.
get_scraper_delay)
Also extends the existing `.parse()` golden-parity test (test_scraper_kit_
yandex_golden_parity.py, SimpleNamespace config) with an assert_parity-based
proof over the REAL production construction path (RealScraperConfig() +
get_scraper_delay), plus a harness-catches-divergence sanity check.
No production code changed -- app/services/scrapers/yandex_valuation.py and
scraper_kit/providers/yandex/valuation.py are both pre-existing.
Group E1 of the scraper_kit migration epic (#2277 -> #2308 -> #2334): prepares the
kit side of app/services/scrapers/avito_imv.py -> scraper_kit.providers.avito.imv
WITHOUT switching any real call site (estimator.py is #2337, a separate final step;
house_imv_backfill.py is deliberately deferred too, see report).
- Golden-parity: full evaluate_via_imv() async flow (geocode A/B + evaluate C) proven
byte-identical between legacy and kit on REAL live-captured Avito API responses
(tests/fixtures/avito_imv_geo_position.json + avito_imv_getdata.json), routed through
the browser_fetcher transport to stay offline/deterministic. Previously this function
had zero parity coverage (only _parse_price was covered, via admin.py Group A).
Added cheap pure-function parity for compute_imv_cache_key/_parse_geo_position/
_parse_placement_history/_parse_suggestions too.
- Config-footgun regression: kit evaluate_via_imv only reads config.scraper_proxy_url
when config= is explicitly passed; called bare it silently builds a proxy-less
curl_cffi session even with a real proxy configured. Two tests lock this down
(without config= -> proxies=None, with config=RealScraperConfig() -> proxies wired) —
manually verified by breaking scraper_kit/providers/avito/imv.py's config threading,
watching the "with config" test fail on the exact proxies assertion, then reverting.
No production import switched in this commit — see PR/issue report for the full
caller audit and the house_imv_backfill.py deferral rationale.
Code-review follow-up: the Group C regression tests for the fetch_detail
backconnect footgun and AvitoScraper's config requirement proved the KIT
LIBRARY behavior in isolation, but nothing asserted the actual call sites in
avito_detail_backfill.py still pass config=RealScraperConfig() at fetch_detail
and AvitoScraper construction. Mirrors #2306's test_backfill_wave2.py:282-286
pattern -- isinstance-checks the captured call_args instead of a bare
assert_called().
Verified the new assertions actually catch the regression: temporarily
removed both config=RealScraperConfig() call-site args from
avito_detail_backfill.py, confirmed test_backfill_processes_snapshot_to_completion
fails exactly on the new assertion line, then restored (git diff was empty
afterward).
Refs #2310
Migrates legacy app.services.scrapers.* imports to scraper_kit equivalents for
house_imv_backfill.py, avito_detail_backfill.py, cian_history_backfill.py,
ekb_geoportal_ingest.py, and yandex_detail_backfill.py, proving parity via
tests/support/parity.assert_parity per the epic's gate (#2304).
newbuilding_enrich_backfill.py and yandex_newbuilding_sweep.py are left fully
on legacy imports: both call into scraper_kit.providers.{cian,yandex}.newbuilding,
which construct BrowserFetcher(source=...) without the now-mandatory endpoint=
kwarg (issue #2322, verified still open against the actual provider source, not
just issue status) -- no caller-side fix is possible, matching Group A's (#2305)
precedent for the same bug in admin.py.
Config-gated kit function footguns found and fixed (Group B #2306 pattern):
- avito fetch_detail's backconnect-on-403 retry silently drops when config=
is omitted -- now passes config=RealScraperConfig() explicitly, with a
regression test proving the gate.
- kit AvitoScraper's constructor now requires ScraperConfig positionally --
wired via RealScraperConfig(), which _rotate_ip() reads for
avito_proxy_rotate_url.
- BrowserFetcher(source=...) call sites (house_imv_backfill, avito/cian
detail backfills) now pass the mandatory endpoint=settings.browser_http_endpoint.
New footgun discovered (NOT #2322, flagged for follow-up): kit's
build_warmed_session() builds its curl_cffi session via _build_detail_session()
with no config parameter at all, unlike fetch_detail -- migrating it would
silently drop the sticky MGTS-proxy egress on avito_detail_backfill's
warm-batch path (the prod default). Left build_warmed_session/
_AVITO_WARM_SEARCH_URL on legacy imports, documented inline.
Refs #2310
code-reviewer flagged that neither the new parity test nor
test_backfill_wave2.py's 3 cian_price_history tests would catch a
regression if config=RealScraperConfig() were accidentally dropped from
the fetch_detail call in cian_price_history.py (the proxy-footgun fix
from the #2306 migration) — mock_fetch.assert_called_once() doesn't
check kwargs.
Strengthens all 3 tests (saves_changes / skips_no_changes /
handles_fetch_none) to assert isinstance(call_kwargs["config"],
RealScraperConfig). Manually verified the guard actually catches the
regression: temporarily removed config= from cian_price_history.py:110,
confirmed all 3 tests fail, restored the fix, confirmed they pass again.
Group A of the scraper_kit migration epic (#2277). Switches admin.py's manual
"run parser" debug endpoints and scripts/ingest_domclick_jsonl.py from direct
app.services.scrapers.* imports to their scraper_kit.providers.* equivalents,
using the DI adapters (RealScraperConfig/RealMatcherAdapter/RealProxyProvider,
app.services.scraper_settings.get_scraper_delay) already established by
app.scheduler_main._run_kit_scheduler.
Migrated: /scrape (AvitoScraper/CianScraper/YandexRealtyScraper + save_listings),
scrape_avito_house, scrape_avito_detail, scrape_avito_imv, scrape_yandex_detail,
scrape_yandex_valuation, scrape_cian_detail, cian_auto_login's BrowserFetcher.
scripts/ingest_domclick_jsonl.py: ScrapedLot/save_listings + (now that #2307/
Group D ported a kit equivalent while this was in flight) DomClickDetailEnrichment/
save_detail_enrichment.
Deliberately NOT migrated (documented in admin.py): scrape_yandex_newbuilding /
scrape_cian_newbuilding — their kit equivalents
(providers/{yandex,cian}/newbuilding.py) construct an internal BrowserFetcher(...)
without the mandatory `endpoint` kwarg, so any call crashes/silently-fails
regardless of caller-side DI. Bug lives in scraper_kit provider code, out of
scope here (only consuming, not touching provider logic) — flagged as follow-up.
Parity proven via tests/support/parity.py (assert_parity) against the exact
functions each debug route now calls, on offline fixtures — no live network/DB.
Refs #2305
Group B of the scraper_kit migration epic (#2277): switch the three files'
internal dependency imports from legacy app/services/scrapers/* to their
byte/structurally-identical scraper_kit equivalents, proven via the new
parity harness (tests/support/parity.assert_parity, #2304). Public API of
all three files is unchanged, so no callers needed updating.
- cian_price_history.py: fetch_detail/save_detail_enrichment now from
scraper_kit.providers.cian.detail. Wires config=RealScraperConfig() at
the call site — kit's fetch_detail only reads cian_proxy_url when an
explicit config is passed, unlike legacy's always-on settings read;
without this the admin-triggered backfill would silently drop its
datacenter-IP proxy (#806) after the import swap.
- cian_session.py: extract_state now from scraper_kit.cian_state_parser
(byte-identical to legacy).
- yandex_price_history.py: ScrapedLot now from scraper_kit.base
(byte-identical fields/methods; record_yandex_price_history only reads
lot.* via duck-typing, no isinstance checks).
New tests/test_scraper_kit_pricehistory_session_parity.py proves the
migration delta specifically (extract_state, ScrapedLot.model_dump(),
fetch_detail+DetailEnrichment with config injection, and the own-session
proxy-kwarg wiring). Full backend suite passes (3145 passed, 6 skipped,
1 pre-existing unrelated failure confirmed on unmodified forgejo/main).
Group D of the scraper_kit migration epic (#2277). Both legacy modules had no
kit-side equivalent yet (per audit Scraper_Kit_Legacy_Dependency_Audit_0703),
unlike the other 24/26 files which were already strangler-duplicated.
- scraper_kit/providers/domclick/detail.py — Layer B detail-card enrichment
(parse_detail_html/fetch_detail/save_detail_enrichment), no ScraperConfig
injection needed (fetch_detail takes an already-open browser_fetcher).
- scraper_kit/providers/ekb_geoportal/{__init__,client}.py — new subpackage,
first non-listing-site provider in scraper_kit. Legacy module had zero
app.* coupling (only httpx), so the port is near-verbatim.
Parity proven via tests/support/parity.py (assert_parity) against the exact
legacy functions on realistic fixtures (live-card SSR JSON, GeoJSON
FeatureCollection).
Callers of both legacy modules are unchanged (out of scope — ekb_geoportal_ingest
switch is #2310; the only other domclick_detail caller is
scripts/ingest_domclick_jsonl.py, also untouched).
Code-reviewer follow-up on the parity harness (aaaf8179):
- parity.py: plain `==` in the scalar-equality branch let `True == 1` /
`False == 0` silently pass (Python bool is an int subclass). A future
migration bug turning a `bool | None` field into a raw 0/1 would slip
through undetected. Now any type mismatch where exactly one side is a
bool is reported as a diff, regardless of numeric equality.
- test_parity.py: unit test for the new bool-vs-int branch.
- test_avito_detail_kit_parity.py: the existing smoke test only proved the
harness reports "no diff" on two genuinely-identical real dataclass
instances — it never proved the harness catches a real divergence on
this same 30+-field shape (only the toy fixtures in test_parity.py did).
Added a test that mutates `price_rub` via dataclasses.replace() on the
real kit DetailEnrichment and asserts assert_parity raises
ParityMismatchError naming that field.
Stage 0 of the scraper_kit migration epic (#2277): shared test tool for
issues #2305-#2310, which each need to prove their kit-path importer
produces the same output as the legacy path on the same input.
- tests/support/parity.py: assert_parity()/compare_outputs() normalize
dataclass/pydantic outputs to dict/list/scalar before comparing, since
legacy vs kit dataclasses (e.g. DetailEnrichment) are different classes
and dataclass __eq__ always returns False across classes even when all
field values match. Supports ignore_fields (drop non-deterministic
fields like latency_ms/fetched_at) and numeric tolerance (math.isclose)
for float fields, with an assertion listing every differing field
(path + legacy value + kit value) on mismatch.
- tests/support/test_parity.py: unit tests for the harness itself
(identical outputs pass, differing outputs raise with informative
diff, tolerance/ignore_fields options, cross-class dataclass parity).
- tests/scrapers/test_avito_detail_kit_parity.py: end-to-end smoke proof
against real code — app.services.scrapers.avito_detail.parse_detail_html
(legacy, reached via admin.py's scrape_avito_detail debug endpoint
through fetch_detail) vs scraper_kit.providers.avito.detail's copy, on
a fixed HTML fixture.
- tests/support/README.md: usage note for #2305-#2310 migration PRs.
Found while implementing: tests/test_scraper_kit_*_parity.py (9 files,
~3400 lines) already do ad-hoc `dataclasses.asdict(old) == asdict(new)`
parity checks for the already-migrated SERP scraper modules (avito/cian/
domclick/yandex/base/scheduler/pipeline) — this harness generalizes that
repeated pattern for the remaining 12 non-scraper importers, adding
ignore_fields/tolerance which those ad-hoc checks don't have.
Финальный PR issue #2045 (BE-3): GET /api/v1/trade-in/location-coef для
LocationDrawer. FDW foreign table -> локальное зеркало osm_poi_ekb_local
(TRUNCATE+INSERT, тот же паттерн что cad_buildings_local/cadastral_geo_match,
избегает ~1.16s/row FDW round-trip) -> straight-line POI-скоринг, портированный
из Site Finder poi_score.py::compute_poi_weighted_top7 (CATEGORY_WEIGHTS as-is,
радиус 1200м для квартир вместо Ptica 2000м для участков). score->coef -
новая MVP-эвристика (0.95..1.05, не откалибрована на реальных дельтах).
Graceful fallback (не 500, не фабрикуем факторы): пустая/не отрефрешенная
osm_poi_ekb_local или отсутствие lat/lon у оценки -> coef=1.0, factors=[],
geo_source="unavailable".
Scheduler: source=osm_poi_ekb_refresh, daily, зарегистрирован и в боевом
dispatch (scheduler.py), и в kit-registry (product_handlers.py) - иначе
test_kit_registry_completeness падает на ship-dark инварианте (#2192).
Frontend wiring (mappers.ts/LocationDrawer.tsx) - вне scope, отдельная задача
после проверки endpoint'а curl'ом на деплое.
- listings.phones: миграция 166 обнуляет данные; cian.py + scraper-kit
serp.py (golden-parity зеркало) перестают читать offer.get("phones") на
входе — write-path выключен у источника, колонка/поле в модели остаются.
Мёртвая запись "phones" в LISTING_FIELD_PRIORITY убрана.
- trade_in_estimates.client_name/client_phone: миграция 167 дропает колонки;
синхронно убраны из TradeInEstimateInput, обоих INSERT в estimator.py
(основной путь + _empty_estimate), frontend TradeInEstimateInput и
CRM-блока EstimateForm (поля "Имя клиента"/"Телефон").
- sentry_scrub.py и management_companies.phones не тронуты — вне scope.
/me теперь отдаёт display_name/org/email (kopylov -> "Копылов", остальные None).
TopNav использует их вместо фабрикации username-as-name; org/email фолбэк
остаётся прежним для юзеров без известного профиля.
Audit of /trade-in/v2 found the FE-3/FE-4 overlay wiring (mapAnalytics/mapSources
feeding AnalyticsView/SourcesView from useEstimateHouseAnalytics,
useEstimateSellTimeSensitivity, analogs[]/actual_deals[]) already shipped in
267585e3. The remaining gap was BE-1 (#2043, merged 2026-07-02) landing real
cv/source_counts/created_at on AggregatedEstimate AFTER that wiring pass, so
v2/mappers.ts was still FE-approximating cv and duplicating a hardcoded source
label map instead of the single source-registry (#2211).
- mapSources' marketAds.kpi.cv (#2041) now prefers the real estimate.cv
(0..1 ratio from BE-1) and only falls back to the FE coefficient-of-variation
of analog prices for pre-#2043 cached estimates where cv is absent.
- adRows/dealRows source badges (#2041) now resolve via the shared
src/lib/source-registry sourceLabel() instead of a local SOURCE_LABEL map,
picking up etagi/canonical labels and staying in sync with the #2211 roster.
- AnalyticsView (#2040) KPIs/sell-time/price-history/scatterDetail were already
wired to real data with no gaps found against the issue spec.
Fixed ANCHOR_TIMEOUT_SEC=240 guillotined every cian anchor when running via
USE_PROXY_POOL_BROWSER=true: each SERP fetch goes through camoufox with relaunch
on proxy swap (13-45s/page), and an anchor = 4 room-buckets x pages_per_anchor
SERP fetches + detail_top_n details + houses-enrich >> 240s. save_listings never
ran, lots_fetched=0, and after 3 consecutive skips the run was mark_banned.
Introduces _cian_anchor_timeout_s() mirroring the avito/yandex/domclick scaling
pattern: max(ANCHOR_TIMEOUT_SEC, buckets*pages*(delay+per_serp) + details*per_detail
+ houses_budget). Defaults -> 1580s worst-case watchdog (hang guard, not expected
duration). Applied to both the scraper-kit copy (strangler target) and the active
app.services.scrape_pipeline copy that run 581 actually hit.
