bot-backend
|
79a77200d7
|
refactor(tradein/scrapers): удалить legacy scrape_pipeline.py + тест-хирургия (#2397 Part E1)
CI Trade-In / changes (pull_request) Successful in 10s
CI / changes (pull_request) Successful in 9s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m43s
app.services.scrape_pipeline (3734 строк) имел 0 runtime-импортёров — Part C убрал
последнего вызывающего (admin.py, scheduler_main.py уже на
scraper_kit.orchestration.pipeline). Удаление:
- backend/app/services/scrape_pipeline.py — удалён целиком
Тесты (18 файлов, импортировавших scrape_pipeline):
DELETE (чистая legacy-оркестрация, kit-эквивалент покрыт в
test_scraper_kit_pipeline_parity{,2}.py):
- test_scrape_pipeline.py, test_scrape_pipeline_proxy_pool.py (proxy-pool DB-lease
вытеснен RealProxyProvider DI-адаптером, покрытым test_scraper_adapters_contracts.py)
- test_anchor_watchdog.py, test_ban_rotation_1790.py, test_provider_rotation_1848.py
- test_scraper_resilience_1949_1950.py, test_sweep_drain.py
CONVERT (parity-сравнение убрано, kit-only regression оставлен/усилен):
- test_scraper_kit_pipeline_parity.py, test_scraper_kit_pipeline_parity2.py —
_drive_old убран, _drive_new стал единственной веткой; ассёрты на конкретные
значения counters сохранены (это теперь основное regression-покрытие kit
orchestration)
- test_sweep_imv_phase.py — 6 IMV-phase тестов (enrich_imv true/false/no-touched/
crash/failed-counter/cancel) переведены на run_avito_city_sweep (kit) с DI
(config/matcher/enrichment); enrichment.process_houses_imv_batch — инжектируемый
Protocol-метод вместо прямого импорта app.services.house_imv_backfill
PARTIAL-EDIT (легаси sweep-behavior вырезан, admin/kit-DI/несвязанные тесты
оставлены):
- test_city_sweep.py — только retarget 4 импортов на scraper_kit.orchestration.pipeline
- test_cian_city_sweep.py, test_yandex_city_sweep.py — sweep-behavior секции удалены,
admin endpoint тесты (уже kit-DI) + anchor-timeout/watchdog формулы (retarget import)
оставлены
- test_domclick_sweep.py — sweep-wiring тесты удалены, Counters + _map_item оставлены
- test_avito_newbuilding_sweep.py — sweep-wiring тест #5 удалён, retarget
NewbuildingSweepCounters
- test_pipeline_browser_routing.py — 3 run_avito_pipeline теста удалены, browser-routing
на уровне fetch_detail/fetch_house_catalog/AvitoScraper не тронут
- test_scraper_proxy.py, tests/scrapers/test_avito_anti_bot.py — _avito_proxies()
переведён на kit (DI-параметр вместо settings-patch); pipeline-level тесты удалены
Stale-комментарии (scrape_pipeline.py как текущая реальность) поправлены в
admin.py, avito_detail_backfill.py, house_imv_backfill.py, yandex_price_history.py,
scrapers/avito_detail.py + kit-пакете (orchestration/__init__.py, orchestration
/pipeline.py, providers/avito/detail.py — strangler завершён, легаси удалён).
grep -rn scrape_pipeline app/ scripts/ packages/ — только past-tense упоминания
("удалён в Part E1", "перед его удалением") + один historical footnote
(contracts.py:138, provenance-заметка "собраны grep'ом по ... .py").
Backend suite: 3088 passed, 6 skipped (известный flake test_search_cache_hit #2208 —
не регрессия). ruff check чист на всех изменённых файлах.
|
2026-07-04 13:56:06 +03:00 |
|
|
|
52fc920507
|
fix(tradein/domclick): rewrite dead scraper to BFF JSON API + honest status (#1846, #1968)
Old HTML scraper dead: /search?city_id retired in DomClick redesign + VPS IP
QRATOR-banned (prod domclick_city_sweep reported done/0 every run). New path:
- GET bff-search-web.domclick.ru/api/offers/v1 + count/v1 via
BrowserFetcher(source="domclick") -> generic provider -> shared mobile proxy
-> bypasses QRATOR (prod-verified live: 20 items returned, no block).
- Sweeps 6 room buckets (st/1/2/3/4/5+); recursive binary price-split for
oversized buckets (rooms=2 ~2215 > offset cap 2000); count/v1 sizes buckets.
- Maps SERP JSON -> ScrapedLot with lat/lon at ~100% (closes avito 54% NULL-lat
gap); per-card EKB geo-guard.
- Honest status (#1968): QRATOR block OR fetch errors with 0 lots -> mark_failed,
not mark_done. Parse failures are per-page resilient (break bucket, keep partial
lots) so a transient bad page never aborts the whole sweep.
- Schedule shipped DORMANT (enabled=false, pages=100) via
138_domclick_bff_rewrite_schedule.sql; operator enables after prod smoke.
42 scraper tests (price-split boundaries, parse-error resilience, honest status);
full backend suite green; ruff clean. Layer B (detail renovation/wallType/
priceHistory) deferred to #1846 follow-up.
|
2026-06-27 13:57:07 +03:00 |
|
