2 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
bot-backend
|
575c233e51 |
refactor(tradein/scrapers): удалить 6 осиротевших legacy scraper-модулей + тест-хирургия (#2397 Part E2)
All checks were successful
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m2s
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 8s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
app.services.scrapers.{cian,cian_detail,yandex_realty,domclick,domclick_exceptions,
avito_houses} имели 0 runtime-импортёров после E1 (scrape_pipeline.py удалён,
#2405) — admin.py/cian_history_backfill.py уже на scraper_kit.providers.*.
Удаление:
- 6 модулей целиком (avito_houses.py, cian.py, cian_detail.py, domclick.py,
domclick_exceptions.py, yandex_realty.py)
- 3 осиротевших dev-sweep скрипта (local-sweep-ekb{,-cian,-yandex}.py) —
эксклюзивно драйвили удалённые легаси-скраперы, kit-путь их заменяет
Тесты (18 файлов DELETE — эксклюзивно тестировали удалённые модули или были
чистым legacy-vs-kit parity, чей легаси-эталон исчез):
test_cian_detail.py, test_cian_serp_scraper.py, tests/scrapers/test_cian_exhaustive.py,
test_yandex_realty_serp.py, tests/scrapers/test_domclick.py,
test_avito_houses_dedup.py, test_avito_houses_parse.py,
test_houses_rating_filter_entries_1789.py, test_scraper_kit_domclick_golden_parity.py,
tests/scrapers/test_admin_{avito,cian,yandex}_kit_parity.py.
CONVERT (файл покрывает СМЕШАННЫЙ live+legacy набор — вырезаны только
секции удалённых модулей, live-покрытие сохранено):
- test_scraper_kit_cian_golden_parity.py — SERP/`_format_address`/detail
(cian.py/cian_detail.py) убраны; extract_state (cian_state_parser, live)/
valuation (cian_valuation, live)/newbuilding (cian_newbuilding, live)/
no-app-imports guard — остались
- test_scraper_kit_yandex_golden_parity.py — SERP `_entity_to_lot`/
`_parse_gate_json`/`_build_url` (yandex_realty) убраны; normalize_house_type
(live)/newbuilding parse (yandex_newbuilding, live)/no-app-imports guard —
остались
- test_scraper_kit_bisection_parity.py — cian/domclick/yandex-ветки убраны;
avito-ветка (live, единственная оставшаяся легаси-бисекция) — осталась
полностью, включая full-seed-grid
- test_scraper_kit_group_c_backfill_kit_parity.py — cian fetch_detail
browser_fetcher parity-тест убран; avito/house_imv_backfill/
cian_history_backfill/ekb_geoportal тесты — остались
- test_scraper_kit_pricehistory_session_parity.py — fetch_detail
config-injection/proxy-wiring тесты (cian_detail) убраны; extract_state
(cian_state_parser, live)/ScrapedLot (base, live) parity — остались
- test_yandex_scrapers_delay_wiring.py — yandex_realty delay-тест убран
(kit-эквивалент уже покрыт test_kit_serp_proxy_pool.py); yandex_newbuilding
(live)/yandex_detail/yandex_valuation (kit) тесты — остались
- test_scraper_proxy.py — yandex_realty BrowserFetcher-тест убран (redundant
с test_kit_serp_proxy_pool.py::test_yandex_serp_no_pool_by_default)
RETARGET на kit (символ живёт, только источник поменялся):
- test_domclick_sweep.py — `_map_item` → scraper_kit.providers.domclick.serp
- test_yandex_city_sweep.py — DEFAULT_PRICE_RANGES/ROOM_PATH →
scraper_kit.providers.yandex.serp
- test_pipeline_browser_routing.py — fetch_house_catalog →
scraper_kit.providers.avito.houses (+ KitAvitoBlockedError для firewall-теста)
- test_snapshot_writer.py — DetailEnrichment/save_detail_enrichment →
scraper_kit.providers.cian.detail (byte-identical strangler-копия)
- test_backfill_wave2.py — DetailEnrichment → scraper_kit.providers.cian.detail
- tests/scrapers/test_avito_unix_date_tz_consistency.py — avito_houses →
scraper_kit.providers.avito.houses (houses-ветка _unix_to_date/RUS_MONTHS
consolidation-проверки)
Stale-комментарии поправлены в cian_price_history.py, tests/scrapers/
test_domclick_detail.py (описывали ещё-живой domclick.py/domclick_exceptions.py).
grep -rn scrapers\.\(cian\|cian_detail\|yandex_realty\|domclick\|domclick_exceptions\|
avito_houses\) app/ scripts/ packages/ — только historical footnotes
("strangler-копия app.services.scrapers.X", past-tense "удалён/deleted").
Backend suite: 2811 passed, 6 skipped, 1 failed (known flake test_search_cache_hit
#2208, не регрессия). ruff check чист на всех изменённых файлах.
|
||
| 62fa1f732a |
refactor(scraper-kit): unify price bisection engine across providers + dedup (#2134)
Some checks failed
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 14s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Has been cancelled
Deploy Trade-In / test (push) Has been cancelled
|
