Commit graph

3 commits

Author SHA1 Message Date
a07fe55db9 fix(tradein): route yandex/valuation/imv/newbuilding scrapers through scraper_proxy_url
Yandex SERP was being banned (datacenter IP) and on-demand scrapers (cian_valuation,
cian_newbuilding, avito_imv, avito_detail) silently returned None due to same cause.
Mobile proxy was already wired for cian.py/cian_detail.py/avito.py but missing elsewhere.

- yandex_realty: proxies= added to _CurlCffiSession in __aenter__
- yandex_valuation: proxies= added to _CurlCffiSession in __aenter__
- cian_valuation: proxies= added to AsyncSession (async with ... fetch block)
- cian_newbuilding: proxies= added to both own-session paths (fetch_newbuilding +
  resolve_cian_zhk_url); shared-session callers unaffected
- avito_imv: proxies= added to own-session path only (shared CffiAsyncSession from
  AvitoScraper already carries proxy)
- avito_detail: proxies= added to own-session path only (scrape_pipeline passes shared
  session with proxy; admin endpoint creates own-session -> now proxied)

All use pattern: _proxy = settings.scraper_proxy_url; proxies={"http":_proxy,"https":_proxy}
if _proxy else None -- matching cian.py (#806 reference). proxy=None = direct connection (dev).

Tests: 9 new proxy-wiring unit tests added to test_scraper_proxy.py (20 total, all pass).
2026-05-31 09:57:17 +03:00
1731704ddc feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 51s
Deploy Trade-In / deploy (push) Successful in 36s
2026-05-28 13:57:43 +00:00
d28bf4e66a feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 54s
Deploy Trade-In / deploy (push) Successful in 25s
Stage 6 / Wave 4 Worker C of CianScraper v1.

Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn).

Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (cross-source benchmark для Stage 9 estimator).

Files:
- app/services/scrapers/cian_newbuilding.py (NEW, 548 LOC)
- tests/test_cian_newbuilding.py (NEW, 357 LOC, 20 tests)

Reviewed: schema match (migrations 020/021/025/029), ON CONFLICT constraint name verified (houses_price_dynamics_dim_key), graceful chart extraction (empty/None/fallback shapes), atomic transaction.

Known minor follow-ups (non-blocking):
- management_companies UPSERT: ext_id=NULL → potential duplicates (Postgres NULL!=NULL); bounded, document in vault
- offers items[] fast path: no dict-only filter / no room_count injection
- houses_price_dynamics primary shape stores total price into price_per_sqm column with prices_type='price' discriminator — Stage 9 estimator must respect prices_type filter
- house_reliability_checks no UNIQUE → unbounded growth on rescrape (deferred to future migration 030+)

Verdict: APPROVE (deep-code-reviewer).
2026-05-23 13:23:11 +00:00