lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/tradein-mvp/backend/app/services
History
bot-backend e5f4e37415 feat(scrapers): route cian sessions through mobile proxy (Refs #806) 
Снимает egress-часть #639: cian-скрейперы ходили с datacenter-IP (без proxies)
→ даже валидные куки не фетчили с прод-бокса. Прокидываем тот же mobile-proxy,
что у avito (#623/#805).

- config: generic scraper_proxy_url (env SCRAPER_PROXY_URL → fallback
  AVITO_PROXY_URL → None/direct). Прод .env.runtime с AVITO_PROXY_URL работает
  без изменений env.
- avito.py + scrape_pipeline._avito_proxies(): на settings.scraper_proxy_url
  (поведение идентично).
- cian.py / cian_detail.fetch_detail / cian_session.verify_session: proxies=
  {http,https} из settings.scraper_proxy_url; пусто → direct (dev no-op).

Cookie-refresh DMIR_AUTH (#639) — НЕ тронут (needs-human). 7 тестов
(precedence + dict-shape + fallback), ruff clean.
2026-05-30 22:01:21 +03:00
..
exporters fix(pdf): brand-spoof + median=0 empty-state + SSRF + дисклеймер + sources (Refs #772) (#789) 2026-05-30 16:55:56 +00:00
matching feat(tradein): same-building матчинг аналогов по house_id (#6) 2026-05-28 13:55:51 +05:00
scrapers feat(scrapers): route cian sessions through mobile proxy (Refs #806) 2026-05-30 22:01:21 +03:00
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
account_quota.py fix(tradein): atomic conditional quota increment — close TOCTOU (#747) (#749) 2026-05-30 15:52:34 +00:00
brand.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
cache.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00
cian_session.py feat(scrapers): route cian sessions through mobile proxy (Refs #806) 2026-05-30 22:01:21 +03:00
dadata.py feat(tradein): DaData /suggest tier в /suggest endpoint (PR Q2) 2026-05-27 19:08:58 +05:00
estimator.py feat(estimator): exclude city-centroid listings from radius analogs (Refs #769 Part E) 2026-05-30 21:22:21 +03:00
geocoder.py feat(tradein): DaData /suggest tier в /suggest endpoint (PR Q2) 2026-05-27 19:08:58 +05:00
house_metadata.py feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) 2026-05-24 08:57:30 +00:00
image_sanitizer.py fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511) 2026-05-24 11:32:04 +00:00
scheduler.py fix(tradein): data-layer tail A2+B+C — trgm index, FDW options, matview refresh (Refs #769) (#804) 2026-05-30 17:40:54 +00:00
scrape_pipeline.py feat(scrapers): route cian sessions through mobile proxy (Refs #806) 2026-05-30 22:01:21 +03:00
scrape_runs.py fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487) 2026-05-23 20:08:40 +00:00
scraper_settings.py feat(tradein): global scraper delay setting (applies across all scrapers) (#485) 2026-05-23 15:43:28 +00:00
search_query.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00