Удаляет весь `app/services/scrapers/` (16 файлов, ~7100 строк) — Part D (D1-D5)
убрал всех внешних вызывающих, 0 runtime importers подтверждено grep'ом на main.
Заодно:
- удалены 7 осиротевших локальных probe/sweep-скриптов (tradein-mvp/scripts/),
импортировавших уже-удалённые или удаляемые сейчас legacy-модули
- тест-хирургия по 65+ файлам: DELETE прямых legacy-юнит-тестов, RETARGET
тестов, тестирующих ещё живую бизнес-логику (переключены на scraper_kit.*
эквиваленты, включая quality-gate #781/#753/#754/#755/#773/#740), partial-delete
golden-parity тестов, потерявших legacy-оракл
- kit save_listings/AvitoScraper/etc. требуют инжектируемые matcher/config —
ретаргетированные тесты обновлены под новую сигнатуру (RealScraperConfig(),
MagicMock HouseMatcher, region_code=66)
Полный pytest suite зелёный (2255 passed, 6 skipped) кроме известного флейка
#2208 (test_search_cache_hit, не связан со scrapers).
code-reviewer отметил: формулировка "PROD DEFAULT" в докстрингах могла ввести в
заблуждение — прод-путь сегодня всё ещё легаси (avito_detail_backfill.py
импортирует build_warmed_session из legacy-модуля), этот фикс — advance-prep,
не изменение живого прод-поведения.
Group E3 recon confirmed `scraper_kit.providers.yandex.valuation` is already a
strangler-copy (#2133) and already used via admin.py's debug route (#2305,
Group A) with the established config=RealScraperConfig()/delay_provider=
get_scraper_delay convention. estimator.py (the real caller, #2337's scope)
still imports the legacy module directly with no arguments -- not touched here.
Adds regression coverage for the two confirmed footguns before #2337 switches
that call site:
- mandatory `config: ScraperConfig` raises TypeError when omitted (same
discipline as the cian_valuation footgun, #2335)
- optional `delay_provider` silently falls back to hardcoded 5.0s when not
wired, discarding admin-tuned anti-ban throttling (scraper_settings.
get_scraper_delay)
Also extends the existing `.parse()` golden-parity test (test_scraper_kit_
yandex_golden_parity.py, SimpleNamespace config) with an assert_parity-based
proof over the REAL production construction path (RealScraperConfig() +
get_scraper_delay), plus a harness-catches-divergence sanity check.
No production code changed -- app/services/scrapers/yandex_valuation.py and
scraper_kit/providers/yandex/valuation.py are both pre-existing.
Group E1 of the scraper_kit migration epic (#2277 -> #2308 -> #2334): prepares the
kit side of app/services/scrapers/avito_imv.py -> scraper_kit.providers.avito.imv
WITHOUT switching any real call site (estimator.py is #2337, a separate final step;
house_imv_backfill.py is deliberately deferred too, see report).
- Golden-parity: full evaluate_via_imv() async flow (geocode A/B + evaluate C) proven
byte-identical between legacy and kit on REAL live-captured Avito API responses
(tests/fixtures/avito_imv_geo_position.json + avito_imv_getdata.json), routed through
the browser_fetcher transport to stay offline/deterministic. Previously this function
had zero parity coverage (only _parse_price was covered, via admin.py Group A).
Added cheap pure-function parity for compute_imv_cache_key/_parse_geo_position/
_parse_placement_history/_parse_suggestions too.
- Config-footgun regression: kit evaluate_via_imv only reads config.scraper_proxy_url
when config= is explicitly passed; called bare it silently builds a proxy-less
curl_cffi session even with a real proxy configured. Two tests lock this down
(without config= -> proxies=None, with config=RealScraperConfig() -> proxies wired) —
manually verified by breaking scraper_kit/providers/avito/imv.py's config threading,
watching the "with config" test fail on the exact proxies assertion, then reverting.
No production import switched in this commit — see PR/issue report for the full
caller audit and the house_imv_backfill.py deferral rationale.
Group A of the scraper_kit migration epic (#2277). Switches admin.py's manual
"run parser" debug endpoints and scripts/ingest_domclick_jsonl.py from direct
app.services.scrapers.* imports to their scraper_kit.providers.* equivalents,
using the DI adapters (RealScraperConfig/RealMatcherAdapter/RealProxyProvider,
app.services.scraper_settings.get_scraper_delay) already established by
app.scheduler_main._run_kit_scheduler.
Migrated: /scrape (AvitoScraper/CianScraper/YandexRealtyScraper + save_listings),
scrape_avito_house, scrape_avito_detail, scrape_avito_imv, scrape_yandex_detail,
scrape_yandex_valuation, scrape_cian_detail, cian_auto_login's BrowserFetcher.
scripts/ingest_domclick_jsonl.py: ScrapedLot/save_listings + (now that #2307/
Group D ported a kit equivalent while this was in flight) DomClickDetailEnrichment/
save_detail_enrichment.
Deliberately NOT migrated (documented in admin.py): scrape_yandex_newbuilding /
scrape_cian_newbuilding — their kit equivalents
(providers/{yandex,cian}/newbuilding.py) construct an internal BrowserFetcher(...)
without the mandatory `endpoint` kwarg, so any call crashes/silently-fails
regardless of caller-side DI. Bug lives in scraper_kit provider code, out of
scope here (only consuming, not touching provider logic) — flagged as follow-up.
Parity proven via tests/support/parity.py (assert_parity) against the exact
functions each debug route now calls, on offline fixtures — no live network/DB.
Refs #2305
Group D of the scraper_kit migration epic (#2277). Both legacy modules had no
kit-side equivalent yet (per audit Scraper_Kit_Legacy_Dependency_Audit_0703),
unlike the other 24/26 files which were already strangler-duplicated.
- scraper_kit/providers/domclick/detail.py — Layer B detail-card enrichment
(parse_detail_html/fetch_detail/save_detail_enrichment), no ScraperConfig
injection needed (fetch_detail takes an already-open browser_fetcher).
- scraper_kit/providers/ekb_geoportal/{__init__,client}.py — new subpackage,
first non-listing-site provider in scraper_kit. Legacy module had zero
app.* coupling (only httpx), so the port is near-verbatim.
Parity proven via tests/support/parity.py (assert_parity) against the exact
legacy functions on realistic fixtures (live-card SSR JSON, GeoJSON
FeatureCollection).
Callers of both legacy modules are unchanged (out of scope — ekb_geoportal_ingest
switch is #2310; the only other domclick_detail caller is
scripts/ingest_domclick_jsonl.py, also untouched).
Code-reviewer follow-up on the parity harness (aaaf8179):
- parity.py: plain `==` in the scalar-equality branch let `True == 1` /
`False == 0` silently pass (Python bool is an int subclass). A future
migration bug turning a `bool | None` field into a raw 0/1 would slip
through undetected. Now any type mismatch where exactly one side is a
bool is reported as a diff, regardless of numeric equality.
- test_parity.py: unit test for the new bool-vs-int branch.
- test_avito_detail_kit_parity.py: the existing smoke test only proved the
harness reports "no diff" on two genuinely-identical real dataclass
instances — it never proved the harness catches a real divergence on
this same 30+-field shape (only the toy fixtures in test_parity.py did).
Added a test that mutates `price_rub` via dataclasses.replace() on the
real kit DetailEnrichment and asserts assert_parity raises
ParityMismatchError naming that field.
Stage 0 of the scraper_kit migration epic (#2277): shared test tool for
issues #2305-#2310, which each need to prove their kit-path importer
produces the same output as the legacy path on the same input.
- tests/support/parity.py: assert_parity()/compare_outputs() normalize
dataclass/pydantic outputs to dict/list/scalar before comparing, since
legacy vs kit dataclasses (e.g. DetailEnrichment) are different classes
and dataclass __eq__ always returns False across classes even when all
field values match. Supports ignore_fields (drop non-deterministic
fields like latency_ms/fetched_at) and numeric tolerance (math.isclose)
for float fields, with an assertion listing every differing field
(path + legacy value + kit value) on mismatch.
- tests/support/test_parity.py: unit tests for the harness itself
(identical outputs pass, differing outputs raise with informative
diff, tolerance/ignore_fields options, cross-class dataclass parity).
- tests/scrapers/test_avito_detail_kit_parity.py: end-to-end smoke proof
against real code — app.services.scrapers.avito_detail.parse_detail_html
(legacy, reached via admin.py's scrape_avito_detail debug endpoint
through fetch_detail) vs scraper_kit.providers.avito.detail's copy, on
a fixed HTML fixture.
- tests/support/README.md: usage note for #2305-#2310 migration PRs.
Found while implementing: tests/test_scraper_kit_*_parity.py (9 files,
~3400 lines) already do ad-hoc `dataclasses.asdict(old) == asdict(new)`
parity checks for the already-migrated SERP scraper modules (avito/cian/
domclick/yandex/base/scheduler/pipeline) — this harness generalizes that
repeated pattern for the remaining 12 non-scraper importers, adding
ignore_fields/tolerance which those ad-hoc checks don't have.
avito_detail_backfill (mode=browser) aborted prod run 458 with
attempted=5 enriched=0 blocked=5: the lat-null pending queue is dominated
by DEAD listings (404 — that's why they are stale coord-holes). In browser
mode a dead listing's 404 page ("Ошибка 404. Страница не найдена") has no
item-view, so _is_detail_soft_block returned True → AvitoBlockedError →
backfill counted it as `blocked` and the consecutive-block breaker aborted
the run before reaching live listings.
- New AvitoListingGoneError (subclass of common AvitoError, NOT of
AvitoBlockedError/AvitoRateLimitedError) so the backfill block-trap
does not catch it.
- New _is_detail_not_found() detector + browser-mode branch in fetch_detail,
checked BEFORE firewall/soft-block (keyed on the 404 title so it does not
swallow the generic soft-block deflect). Curl path unchanged (404 → 302/
non-200 → ValueError → failed, as before).
- Backfill: new `gone` counter; dedicated except AvitoListingGoneError branch
that is neutral to the consecutive-block breaker and marks the listing
is_active=FALSE (SAVEPOINT) so it leaves the snapshot scope.
Tests: unit for _is_detail_not_found (404 True; soft-block/item-view/empty
False), browser-mode fetch_detail raising AvitoListingGoneError (not Blocked),
and backfill gone-row marks inactive without aborting the breaker.
Soft-block (HTTP 200 generic витрина без item-view) теперь детектится как блок → существующая 403/firewall reconnect-машинерия → ротация exit-IP вместо silent failed (blocked=0, enriched→0 каскад). + snapshot ORDER BY (lat IS NULL) DESC для приоритизации coord-holes (#1967, in-scope 321). Tests: test_avito_detail_soft_block.py. Full tradein suite 2591 passed.
Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched
because the detail fetcher only had a same-session 429 short-retry (no reconnect).
429s are persistent: keep-alive reuses one backconnect exit IP across sequential
detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769):
widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP)
on 429 exhaustion before raising. Caller session never touched (as with 403 path).
Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36
(deep pagination, backconnect conn-limit). Widen the 429 retry budget + one
reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip
a blocked bucket and continue rather than aborting the run; only N consecutive
blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.
fetch_detail raised on the first 403/429, aborting the detail backfill (run 191
enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active.
Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy,
rebuild an ephemeral session (fresh exit IP) and retry up to N before raising;
short-retry 429. Never close/rebuild the caller-provided shared session. Also
add the missing HTTP-200 firewall-interstitial check on the curl detail path.
Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate
sequentially newest-first and stop once cards fall below the watermark, skipping
deep pagination (and the 403 clusters it triggers). since=None keeps the exact
exhaustive bisection walk. No scheduler/schedule change here — wiring follows
in a separate PR.
PR #1764's reconnect-retry was dead code in prod: the gate excluded configs
with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL
(what curl_cffi egresses through) AND a leftover changeip URL for the separate
auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403
still aborted the sweep (run 212). The curl session always uses scraper_proxy_url,
so session-rebuild rotates the exit IP regardless of changeip config.
Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or
firewall interstitial aborted the entire avito_full_load sweep via
AvitoBlockedError -> mark_banned. Each new connection through backconnect
gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to
_AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip
rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
