lekss361
|
0154c2914a
|
feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail)
Stage 2e of AvitoScraper_v2.
- run_avito_pipeline(db, lat, lon, radius_m, *, enrich_houses, enrich_detail_top_n)
- 5-step flow:
1. SEARCH: AvitoScraper().fetch_around (uses Stage 2a refactored search)
2. SAVE: save_listings (inserts + tracks house_ext_id from Stage 2a)
3. GROUP: dedupe house path → unique set (path-only для consistency с fetch_house_catalog)
4. ENRICH_HOUSES: fetch_house_catalog + save_house_catalog_enrichment (Stage 2c) per unique house
5. ENRICH_DETAIL: top-N priority listings (detail_enriched_at IS NULL within radius)
→ fetch_detail + save_detail_enrichment (Stage 2b)
- PipelineCounters dataclass: lots_fetched/inserted/updated, unique_houses, houses_enriched/failed,
detail_attempted/enriched/failed, errors[]
- Graceful degradation: per-house and per-detail try/except — single failure не валит pipeline
- IMV ОТСУТСТВУЕТ — он on-demand (Stage 3 estimator integration), не cron
- pytest offline smoke (3 tests: counters default + search failure graceful + group_by_house dedupe)
- Add pytest-asyncio>=0.24.0 to dev deps + asyncio_mode=auto в pyproject.toml
Refs: AvitoScraper_v2_Implementation_Plan Stage 2e.
|
2026-05-23 15:45:34 +03:00 |
|
