4a6820b8f7
All checks were successful
CI / changes (pull_request) Successful in 9s
CI Trade-In / changes (pull_request) Successful in 9s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m48s
Group E4 (final, highest-risk step of scraper_kit migration epic #2277): estimator.py and house_imv_backfill.py's avito_imv/cian_valuation/ yandex_valuation call sites now import from scraper_kit.providers.* instead of app.services.scrapers.*, following the exact wiring proven safe by E1/E2/E3 (#2334/#2335/#2336): - estimator.py's avito IMV (_get_or_fetch_imv_cached, both call sites) and house_imv_backfill.py's _process_one_house: add config=RealScraperConfig() — kit's evaluate_via_imv silently drops the configured proxy without it. - estimator.py's cian valuation (Stage 9): add config=RealScraperConfig() — mandatory kwarg on the kit function (TypeError if omitted). - estimator.py's yandex valuation: add config=RealScraperConfig() (mandatory) and delay_provider=get_scraper_delay — without it kit silently falls back to a hardcoded 5.0s throttle instead of the DB-configured anti-ban delay. - All exception classes imported consistently from the same kit module as evaluate_via_imv (not just the function) — mixing legacy/kit exception classes would break `except IMVAddressNotFoundError` etc. via identity mismatch (caught by an existing test that assumed the legacy class, fixed alongside). Observability: both cian_valuation and yandex_valuation graceful-degradation except-blocks upgraded from logger.warning to logger.exception. GlitchTip's LoggingIntegration listens at event_level=ERROR (main.py/scheduler_main.py) — a WARNING never reaches GlitchTip as an event regardless of exc_info, so a future config-wiring mistake at these call sites needs ERROR level to be visible in monitoring. house_imv_backfill.py: RealScraperConfig is imported lazily inside _process_one_house (not at module level) to avoid a circular import — app.services.scraper_adapters imports backfill_house_imv/ process_houses_imv_batch from this module at module level. Verified via direct import in both orders plus a full `app.main` import. Also fixes a stale docstring claiming process_houses_imv_batch is "not wired into scheduler" — it is, via scrape_pipeline.py's run_avito_city_sweep. Test updates: 2 pre-existing tests (test_backfill_wave2.py) mocked the legacy IMVAddressNotFoundError/IMVEvaluation/IMVGeo classes, now updated to import from scraper_kit to match the production exception identity. Added config=/delay_provider= regression-guard asserts to the relevant estimator and backfill tests, mirroring the existing #2306 cian_price_history pattern. Legacy app/services/scrapers/{avito_imv,cian_valuation,yandex_valuation}.py are untouched and still imported by cian_history_backfill.py's valuation block (separately scoped, not touched here) — revert is a clean single-commit revert, no schema/data migration involved. |
||
|---|---|---|
| .. | ||
| exporters | ||
| matching | ||
| scrapers | ||
| __init__.py | ||
| account_quota.py | ||
| brand.py | ||
| buildings_query.py | ||
| cache.py | ||
| cian_price_history.py | ||
| cian_session.py | ||
| dadata.py | ||
| estimator.py | ||
| gar_flats_loader.py | ||
| geocoder.py | ||
| house_dedup_merge.py | ||
| house_imv_backfill.py | ||
| house_metadata.py | ||
| image_sanitizer.py | ||
| location_coef.py | ||
| product_handlers.py | ||
| proxy_pool.py | ||
| rosreestr_poll.py | ||
| sber_index.py | ||
| scheduler.py | ||
| scrape_pipeline.py | ||
| scrape_runs.py | ||
| scraper_adapters.py | ||
| scraper_settings.py | ||
| search_query.py | ||
| yandex_address_backfill.py | ||
| yandex_price_history.py | ||
| zhkh_flats_loader.py | ||