Commit graph

3 commits

Author SHA1 Message Date
cfa24c3bcf fix(tradein): SAVEPOINT-isolate houses backfill + ruff/shell fixes (#568) 2026-05-28 18:54:27 +05:00
c47a42c0a5 feat(tradein): houses.cian_zhk_url column + backfill (#568)
Add TEXT column cian_zhk_url to houses (migration 071) with partial index.
This unblocks fetch_newbuilding() which requires a ZHK slug URL — the
existing cian_internal_house_id (bigint) is not sufficient for the API call.

Activate the do_houses block in backfill_cian_history(): query houses with
cian_zhk_url set + no price_dynamics rows, then fetch + save per house.

Add resolve_cian_zhk_url() in cian_newbuilding.py: follows Cian redirect
from https://cian.ru/zhk/<id>/ to canonical slug URL, for use by the new
backfill script (scripts/backfill_cian_zhk_url.py).

Store cian_zhk_url into NewbuildingEnrichment and persist it via
save_newbuilding_enrichment UPDATE (COALESCE — never overwrites non-NULL).

Remove stale TODO comments referencing #525 from cian_history_backfill.py
and admin.py docstring. Enable DO_HOUSES=true by default in cian_backfill_all.sh.

Post-deploy required: apply 071 migration, then run
scripts/backfill_cian_zhk_url.py to populate existing rows.
2026-05-28 18:45:42 +05:00
d28bf4e66a feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 54s
Deploy Trade-In / deploy (push) Successful in 25s
Stage 6 / Wave 4 Worker C of CianScraper v1.

Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn).

Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (cross-source benchmark для Stage 9 estimator).

Files:
- app/services/scrapers/cian_newbuilding.py (NEW, 548 LOC)
- tests/test_cian_newbuilding.py (NEW, 357 LOC, 20 tests)

Reviewed: schema match (migrations 020/021/025/029), ON CONFLICT constraint name verified (houses_price_dynamics_dim_key), graceful chart extraction (empty/None/fallback shapes), atomic transaction.

Known minor follow-ups (non-blocking):
- management_companies UPSERT: ext_id=NULL → potential duplicates (Postgres NULL!=NULL); bounded, document in vault
- offers items[] fast path: no dict-only filter / no room_count injection
- houses_price_dynamics primary shape stores total price into price_per_sqm column with prices_type='price' discriminator — Stage 9 estimator must respect prices_type filter
- house_reliability_checks no UNIQUE → unbounded growth on rescrape (deferred to future migration 030+)

Verdict: APPROVE (deep-code-reviewer).
2026-05-23 13:23:11 +00:00