gendesign/tradein-mvp/backend/app/services/scrapers
lekss361 147c36d448 feat(tradein): Avito mobile proxy egress + firewall detect + MSK date (#623, #726)
- avito.py: AVITO_PROXY_URL → proxies в curl_cffi AsyncSession; firewall-детект
  «Доступ ограничен: проблема с IP» на HTTP 200 → AvitoBlockedError (раньше бан
  проходил молча как 0-cards); ротация мобильного IP через changeip-ссылку
  (_rotate_ip) с retry до avito_proxy_max_rotations; transient-retry на сетевых
  обрывах мобильного канала.
- config.py: avito_proxy_url / avito_proxy_rotate_url / avito_proxy_max_rotations.
- #726: sortTimeStamp → дата в MSK (UTC+3 fixed offset), а не UTC — совпадает с
  датой на сайте; off-by-one для постов 00:00–03:00 MSK устранён.
- tests: firewall-детект (HTTP 200), ротация+повтор, exhausted-rotation; serp_date
  переведён на MSK.

Live-verified с прод-сервера: mobile proxy снимает firewall (0→48 карточек),
ротация IP 188.162.65.168→94.25.229.234. Без env AVITO_PROXY_URL поведение не
меняется (dev). Требует deploy (#659) + env AVITO_PROXY_URL/ROTATE_URL на VPS.
2026-05-30 19:49:09 +03:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py feat(tradein): Avito mobile proxy egress + firewall detect + MSK date (#623, #726) 2026-05-30 19:49:09 +03:00
avito_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
avito_exceptions.py fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) 2026-05-30 16:28:23 +00:00
avito_houses.py fix(tradein): decode Avito houses URL-encoded __preloadedState__ (#566) 2026-05-28 15:58:19 +05:00
avito_imv.py fix(tradein-imv): harden Avito IMV anti-bot path + offline contract tests (#565) 2026-05-28 16:29:05 +05:00
base.py fix(scrapers): stable dedup_hash — drop price_rub + strip URL query (Refs #753) (#761) 2026-05-30 16:10:27 +00:00
cian.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
cian_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
cian_newbuilding.py feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627) 2026-05-28 13:57:43 +00:00
cian_state_parser.py feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447) 2026-05-23 12:35:01 +00:00
cian_valuation.py fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523) 2026-05-24 13:44:39 +00:00
n1.py feat(tradein): global scraper delay setting (applies across all scrapers) (#485) 2026-05-23 15:43:28 +00:00
repair_state_normalizer.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
snapshot_writer.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
yandex_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
yandex_helpers.py fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602) 2026-05-27 13:11:22 +00:00
yandex_newbuilding.py feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484) 2026-05-23 15:28:34 +00:00
yandex_realty.py fix(scrapers): yandex area из embedded-state JSON (Refs #775) (#783) 2026-05-30 16:38:12 +00:00
yandex_valuation.py fix(yandex-valuation): off-by-one cell index in _parse_row_cells (#552) 2026-05-24 19:57:17 +00:00