lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/tradein-mvp/backend/app/services/scrapers
History
bot-backend e5f4e37415 feat(scrapers): route cian sessions through mobile proxy (Refs #806) 
Снимает egress-часть #639: cian-скрейперы ходили с datacenter-IP (без proxies)
→ даже валидные куки не фетчили с прод-бокса. Прокидываем тот же mobile-proxy,
что у avito (#623/#805).

- config: generic scraper_proxy_url (env SCRAPER_PROXY_URL → fallback
  AVITO_PROXY_URL → None/direct). Прод .env.runtime с AVITO_PROXY_URL работает
  без изменений env.
- avito.py + scrape_pipeline._avito_proxies(): на settings.scraper_proxy_url
  (поведение идентично).
- cian.py / cian_detail.fetch_detail / cian_session.verify_session: proxies=
  {http,https} из settings.scraper_proxy_url; пусто → direct (dev no-op).

Cookie-refresh DMIR_AUTH (#639) — НЕ тронут (needs-human). 7 тестов
(precedence + dict-shape + fallback), ruff clean.
2026-05-30 22:01:21 +03:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py feat(scrapers): route cian sessions through mobile proxy (Refs #806) 2026-05-30 22:01:21 +03:00
avito_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
avito_exceptions.py fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) 2026-05-30 16:28:23 +00:00
avito_houses.py fix(tradein): decode Avito houses URL-encoded __preloadedState__ (#566) 2026-05-28 15:58:19 +05:00
avito_imv.py fix(tradein-imv): harden Avito IMV anti-bot path + offline contract tests (#565) 2026-05-28 16:29:05 +05:00
base.py feat(estimator): exclude city-centroid listings from radius analogs (Refs #769 Part E) 2026-05-30 21:22:21 +03:00
cian.py feat(scrapers): route cian sessions through mobile proxy (Refs #806) 2026-05-30 22:01:21 +03:00
cian_detail.py feat(scrapers): route cian sessions through mobile proxy (Refs #806) 2026-05-30 22:01:21 +03:00
cian_newbuilding.py feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627) 2026-05-28 13:57:43 +00:00
cian_state_parser.py fix(tradein): word-boundary street match + cian unescape robustness (Refs #769 A1+D) (#798) 2026-05-30 17:18:33 +00:00
cian_valuation.py fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523) 2026-05-24 13:44:39 +00:00
n1.py feat(scrapers/n1): извлекать floor/total_floors из ведущего N/M-токена (Refs #795) (#802) 2026-05-30 17:31:13 +00:00
repair_state_normalizer.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
snapshot_writer.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
yandex_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
yandex_helpers.py fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602) 2026-05-27 13:11:22 +00:00
yandex_newbuilding.py feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484) 2026-05-23 15:28:34 +00:00
yandex_realty.py fix(scrapers): yandex area из embedded-state JSON (Refs #775) (#783) 2026-05-30 16:38:12 +00:00
yandex_valuation.py fix(yandex-valuation): off-by-one cell index in _parse_row_cells (#552) 2026-05-24 19:57:17 +00:00