5 commits

Author	SHA1	Message	Date
bot-backend	575c233e51	refactor(tradein/scrapers): удалить 6 осиротевших legacy scraper-модулей + тест-хирургия (#2397 Part E2) ... app.services.scrapers.{cian,cian_detail,yandex_realty,domclick,domclick_exceptions, avito_houses} имели 0 runtime-импортёров после E1 (scrape_pipeline.py удалён, #2405) — admin.py/cian_history_backfill.py уже на scraper_kit.providers.*. Удаление: - 6 модулей целиком (avito_houses.py, cian.py, cian_detail.py, domclick.py, domclick_exceptions.py, yandex_realty.py) - 3 осиротевших dev-sweep скрипта (local-sweep-ekb{,-cian,-yandex}.py) — эксклюзивно драйвили удалённые легаси-скраперы, kit-путь их заменяет Тесты (18 файлов DELETE — эксклюзивно тестировали удалённые модули или были чистым legacy-vs-kit parity, чей легаси-эталон исчез): test_cian_detail.py, test_cian_serp_scraper.py, tests/scrapers/test_cian_exhaustive.py, test_yandex_realty_serp.py, tests/scrapers/test_domclick.py, test_avito_houses_dedup.py, test_avito_houses_parse.py, test_houses_rating_filter_entries_1789.py, test_scraper_kit_domclick_golden_parity.py, tests/scrapers/test_admin_{avito,cian,yandex}_kit_parity.py. CONVERT (файл покрывает СМЕШАННЫЙ live+legacy набор — вырезаны только секции удалённых модулей, live-покрытие сохранено): - test_scraper_kit_cian_golden_parity.py — SERP/`_format_address`/detail (cian.py/cian_detail.py) убраны; extract_state (cian_state_parser, live)/ valuation (cian_valuation, live)/newbuilding (cian_newbuilding, live)/ no-app-imports guard — остались - test_scraper_kit_yandex_golden_parity.py — SERP `_entity_to_lot`/ `_parse_gate_json`/`_build_url` (yandex_realty) убраны; normalize_house_type (live)/newbuilding parse (yandex_newbuilding, live)/no-app-imports guard — остались - test_scraper_kit_bisection_parity.py — cian/domclick/yandex-ветки убраны; avito-ветка (live, единственная оставшаяся легаси-бисекция) — осталась полностью, включая full-seed-grid - test_scraper_kit_group_c_backfill_kit_parity.py — cian fetch_detail browser_fetcher parity-тест убран; avito/house_imv_backfill/ cian_history_backfill/ekb_geoportal тесты — остались - test_scraper_kit_pricehistory_session_parity.py — fetch_detail config-injection/proxy-wiring тесты (cian_detail) убраны; extract_state (cian_state_parser, live)/ScrapedLot (base, live) parity — остались - test_yandex_scrapers_delay_wiring.py — yandex_realty delay-тест убран (kit-эквивалент уже покрыт test_kit_serp_proxy_pool.py); yandex_newbuilding (live)/yandex_detail/yandex_valuation (kit) тесты — остались - test_scraper_proxy.py — yandex_realty BrowserFetcher-тест убран (redundant с test_kit_serp_proxy_pool.py::test_yandex_serp_no_pool_by_default) RETARGET на kit (символ живёт, только источник поменялся): - test_domclick_sweep.py — `_map_item` → scraper_kit.providers.domclick.serp - test_yandex_city_sweep.py — DEFAULT_PRICE_RANGES/ROOM_PATH → scraper_kit.providers.yandex.serp - test_pipeline_browser_routing.py — fetch_house_catalog → scraper_kit.providers.avito.houses (+ KitAvitoBlockedError для firewall-теста) - test_snapshot_writer.py — DetailEnrichment/save_detail_enrichment → scraper_kit.providers.cian.detail (byte-identical strangler-копия) - test_backfill_wave2.py — DetailEnrichment → scraper_kit.providers.cian.detail - tests/scrapers/test_avito_unix_date_tz_consistency.py — avito_houses → scraper_kit.providers.avito.houses (houses-ветка _unix_to_date/RUS_MONTHS consolidation-проверки) Stale-комментарии поправлены в cian_price_history.py, tests/scrapers/ test_domclick_detail.py (описывали ещё-живой domclick.py/domclick_exceptions.py). grep -rn scrapers\.\(cian\|cian_detail\|yandex_realty\|domclick\|domclick_exceptions\| avito_houses\) app/ scripts/ packages/ — только historical footnotes ("strangler-копия app.services.scrapers.X", past-tense "удалён/deleted"). Backend suite: 2811 passed, 6 skipped, 1 failed (known flake test_search_cache_hit #2208, не регрессия). ruff check чист на всех изменённых файлах.	2026-07-04 14:37:19 +03:00
bot-backend	4a6820b8f7	feat(tradein/estimator): switch core valuation call sites to scraper_kit (#2337) ... Group E4 (final, highest-risk step of scraper_kit migration epic #2277): estimator.py and house_imv_backfill.py's avito_imv/cian_valuation/ yandex_valuation call sites now import from scraper_kit.providers.* instead of app.services.scrapers.*, following the exact wiring proven safe by E1/E2/E3 (#2334/#2335/#2336): - estimator.py's avito IMV (_get_or_fetch_imv_cached, both call sites) and house_imv_backfill.py's _process_one_house: add config=RealScraperConfig() — kit's evaluate_via_imv silently drops the configured proxy without it. - estimator.py's cian valuation (Stage 9): add config=RealScraperConfig() — mandatory kwarg on the kit function (TypeError if omitted). - estimator.py's yandex valuation: add config=RealScraperConfig() (mandatory) and delay_provider=get_scraper_delay — without it kit silently falls back to a hardcoded 5.0s throttle instead of the DB-configured anti-ban delay. - All exception classes imported consistently from the same kit module as evaluate_via_imv (not just the function) — mixing legacy/kit exception classes would break `except IMVAddressNotFoundError` etc. via identity mismatch (caught by an existing test that assumed the legacy class, fixed alongside). Observability: both cian_valuation and yandex_valuation graceful-degradation except-blocks upgraded from logger.warning to logger.exception. GlitchTip's LoggingIntegration listens at event_level=ERROR (main.py/scheduler_main.py) — a WARNING never reaches GlitchTip as an event regardless of exc_info, so a future config-wiring mistake at these call sites needs ERROR level to be visible in monitoring. house_imv_backfill.py: RealScraperConfig is imported lazily inside _process_one_house (not at module level) to avoid a circular import — app.services.scraper_adapters imports backfill_house_imv/ process_houses_imv_batch from this module at module level. Verified via direct import in both orders plus a full `app.main` import. Also fixes a stale docstring claiming process_houses_imv_batch is "not wired into scheduler" — it is, via scrape_pipeline.py's run_avito_city_sweep. Test updates: 2 pre-existing tests (test_backfill_wave2.py) mocked the legacy IMVAddressNotFoundError/IMVEvaluation/IMVGeo classes, now updated to import from scraper_kit to match the production exception identity. Added config=/delay_provider= regression-guard asserts to the relevant estimator and backfill tests, mirroring the existing #2306 cian_price_history pattern. Legacy app/services/scrapers/{avito_imv,cian_valuation,yandex_valuation}.py are untouched and still imported by cian_history_backfill.py's valuation block (separately scoped, not touched here) — revert is a clean single-commit revert, no schema/data migration involved.	2026-07-04 02:40:34 +03:00
bot-backend	88b93435fc	test(tradein): assert config kwarg on cian price-history fetch_detail call (#2306) ... code-reviewer flagged that neither the new parity test nor test_backfill_wave2.py's 3 cian_price_history tests would catch a regression if config=RealScraperConfig() were accidentally dropped from the fetch_detail call in cian_price_history.py (the proxy-footgun fix from the #2306 migration) — mock_fetch.assert_called_once() doesn't check kwargs. Strengthens all 3 tests (saves_changes / skips_no_changes / handles_fetch_none) to assert isinstance(call_kwargs["config"], RealScraperConfig). Manually verified the guard actually catches the regression: temporarily removed config= from cian_price_history.py:110, confirmed all 3 tests fail, restored the fix, confirmed they pass again.	2026-07-04 00:27:08 +03:00
bot-backend	55049046b1	fix(tradein): починить 3 теста-регрессии из #1543 (sber rollback mock, matching tier0, cian price history) — deploy gate green	2026-06-16 09:03:27 +03:00
bot-backend	5373f5db63	feat(tradein): admin backfill endpoints — cian price-history (T8a) + yandex address (T10) + house IMV (T7) (#856) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:37:10 +00:00