0154c2914a
Stage 2e of AvitoScraper_v2.
- run_avito_pipeline(db, lat, lon, radius_m, *, enrich_houses, enrich_detail_top_n)
- 5-step flow:
1. SEARCH: AvitoScraper().fetch_around (uses Stage 2a refactored search)
2. SAVE: save_listings (inserts + tracks house_ext_id from Stage 2a)
3. GROUP: dedupe house path → unique set (path-only для consistency с fetch_house_catalog)
4. ENRICH_HOUSES: fetch_house_catalog + save_house_catalog_enrichment (Stage 2c) per unique house
5. ENRICH_DETAIL: top-N priority listings (detail_enriched_at IS NULL within radius)
→ fetch_detail + save_detail_enrichment (Stage 2b)
- PipelineCounters dataclass: lots_fetched/inserted/updated, unique_houses, houses_enriched/failed,
detail_attempted/enriched/failed, errors[]
- Graceful degradation: per-house and per-detail try/except — single failure не валит pipeline
- IMV ОТСУТСТВУЕТ — он on-demand (Stage 3 estimator integration), не cron
- pytest offline smoke (3 tests: counters default + search failure graceful + group_by_house dedupe)
- Add pytest-asyncio>=0.24.0 to dev deps + asyncio_mode=auto в pyproject.toml
Refs: AvitoScraper_v2_Implementation_Plan Stage 2e.
44 lines
1.4 KiB
TOML
44 lines
1.4 KiB
TOML
[project]
|
||
name = "tradein-mvp-backend"
|
||
version = "0.1.0"
|
||
description = "Trade-In Estimator MVP — standalone fork of gendesign trade-in feature"
|
||
requires-python = ">=3.12"
|
||
dependencies = [
|
||
"fastapi>=0.115.0",
|
||
"uvicorn[standard]>=0.30.0",
|
||
"sqlalchemy>=2.0.30",
|
||
"geoalchemy2>=0.15.0", # PostGIS support для SQLAlchemy
|
||
"pydantic>=2.7.0",
|
||
"pydantic-settings>=2.3.0",
|
||
"psycopg[binary]>=3.2.0",
|
||
"weasyprint>=62.0",
|
||
"jinja2>=3.1.0",
|
||
"httpx>=0.27.0", # для geocoder + scrapers
|
||
"tenacity>=9.0.0", # retry с exp backoff
|
||
"selectolax>=0.3.0", # быстрый HTML парсинг для scrapers
|
||
"segno>=1.6.0", # QR-код для PDF shareable URL
|
||
"curl-cffi>=0.7.0", # impersonate=chrome120 для Cian/Avito (TLS fingerprint)
|
||
"python-multipart>=0.0.9", # FastAPI UploadFile — загрузка фото квартиры (#394)
|
||
"sentry-sdk>=2.0.0", # мониторинг ошибок → GlitchTip (#396)
|
||
]
|
||
|
||
[dependency-groups]
|
||
dev = [
|
||
"pytest>=8.0.0",
|
||
"pytest-asyncio>=0.24.0",
|
||
"ruff>=0.5.0",
|
||
]
|
||
|
||
[tool.pytest.ini_options]
|
||
asyncio_mode = "auto"
|
||
|
||
[tool.setuptools.packages.find]
|
||
include = ["app*"]
|
||
|
||
[tool.ruff]
|
||
target-version = "py312"
|
||
line-length = 100
|
||
|
||
[tool.ruff.lint]
|
||
select = ["E", "F", "I", "B", "UP", "N", "RUF"]
|
||
ignore = ["RUF001", "RUF002", "RUF003"]
|