Commit graph

4 commits

Author SHA1 Message Date
bot-backend
c65a73ca9a fix(tradein/security): admin.py — generic detail + error-id вместо сырого текста исключений (#2234) 2026-07-03 09:16:34 +03:00
b6634ea40c feat(scrapers): live pacing-регулятор (GET/PUT /scraper/pacing) + data-quality coverage endpoint
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- browser/server.py: GET /pacing и PUT /pacing — read/write _MIN_PAGE_INTERVAL_BY_PROVIDER
  in-memory; валидация source in PROVIDERS, interval_s >= 0; логирует изменения; сбрасывается
  к env-дефолту на рестарте by design
- backend/admin.py: GET /scraper/pacing — прокси к browser /pacing, PacingResponse pydantic;
  PUT /scraper/pacing/{source} — прокси PUT с валидацией ge=0 le=120; 502/503 при недоступности
- backend/admin.py: GET /scraper/data-quality — single-pass FILTER-агрегаты по listings WHERE
  is_active GROUP BY source (description/photo_urls/address/lat/lon/kitchen_area_m2/living_area_m2/
  ceiling_height/ceiling_height_m/metro_stations); houses (total/avito_validated_at%/rating_score%/
  house_type%); house_reviews count
- browser/test_server_pacing.py: 10 новых тестов GET+PUT /pacing (16 total, все зелёные)
- backend/tests/test_scraper_admin_apis.py: тесты pacing-прокси + data-quality shape/pct-range
  (21 total, все зелёные)
2026-06-20 12:47:03 +03:00
8361961a44 fix(scrapers/backend): socks5 health-probe (httpx[socks]) + status Literal whitelist + #1820 ext_id=0 dedup-гард
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- pyproject.toml: httpx[socks]>=0.27.0 — тянет socksio, socks5:// прокси в _probe_current_ip работает прозрачно
- admin.py list_scrape_runs_unified: status Literal["done","running","banned","zombie","failed","cancelled"] вместо str|None — невалидные значения → 422
- avito_houses.py save_house_catalog_enrichment: гард ext_id=0 + нет адреса → skip without DB touch; два id-less novostroyka URL не схлопываются в bogus avito:0 в house_sources
- тесты: test_unified_runs_invalid_status_422 + test_unified_runs_valid_statuses_200; test_save_house_catalog_enrichment_skips_zero_ext_id_no_address + *_with_address_persists
2026-06-20 10:17:37 +03:00
2f88a71915 feat(admin): unified scrape/runs + scraper/health + rotate-ip API для единой scrapers-страницы
All checks were successful
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-20 09:25:57 +03:00