lekss361/gendesign

Fork 0

feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) #459

Merged

lekss361 merged 1 commit from feat/tradein-cian-newbuilding-parser into main

2026-05-23 13:23:11 +00:00

lekss361 commented

2026-05-23 13:18:17 +00:00

Owner

Summary

Stage 6 / Wave 4 of CianScraper v1. Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn).

🎯 Main Stage 6 deliverable: 7-month price dynamics chart → houses_price_dynamics table (Cross-source benchmark для Stage 9 estimator).

Files

app/services/scrapers/cian_newbuilding.py (NEW) — fetch_newbuilding() + save_newbuilding_enrichment() + NewbuildingEnrichment dataclass
tests/test_cian_newbuilding.py (NEW) — 20 tests

State extraction

URL pattern: https://zhk-<slug>-<city>-i.cian.ru/
MFE: newbuilding-card-desktop-fichering-frontend
Key: initialState (sister containers live as top-level keys, не separate push entries)

Sister containers parsed

Container	Shape	Target table
`realtyValuation`	`data.priceDynamics.chart.data.{labels[], values[]}` + fallback flat `chart[]`	`houses_price_dynamics` (UPSERT on `houses_price_dynamics_dim_key`)
`reliability`	`{checkStatus, details[]}` + fallback `checks[]`	`house_reliability_checks`
`reviews`	`reviews.items[]`	(stash в raw_sister_states; existing `house_reviews` table из migration 014)
`offers`	`data.total.fromDeveloperRooms[].layouts[]` + `items[]` fast path	nested_offers (used by future matching)
`builders` / `banks` / `housesByTurn`	top-level `newbuilding.*` jsonb arrays	`houses.builders/banks/houses_by_turn`

Persistence (`save_newbuilding_enrichment`)

UPSERT management_companies (ON CONFLICT ext_source, ext_id, name) → get mc_id
UPDATE houses SET cian_internal_house_id, year_built, management_company_id, transport_accessibility_rate, advantages, banks, builders, houses_by_turn
INSERT INTO houses_price_dynamics ... ON CONFLICT ON CONSTRAINT houses_price_dynamics_dim_key (post-029 multi-dimension)
INSERT INTO house_reliability_checks (no UNIQUE — каждая scrape inserts; см. TBD)

Schema verified

migration 020 — houses.cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn jsonb
migration 021 — management_companies UNIQUE (ext_source, name, ext_id)
migration 029 — houses_price_dynamics_dim_key (house_id, source, room_count, prices_type, period, month_date) — exact ON CONFLICT match

Tests

20/20 pass + 48/48 full Cian regression suite clean
Coverage: chart normalization, reliability extraction, address/deadline polymorphism, dataclass defaults, async mocked fetch_newbuilding (no-state graceful)

Ruff

Clean (fixed N806 uppercase var, E501 line length, B905 zip strict=).

Anti-bot

curl_cffi.AsyncSession(impersonate="chrome120") — consistent with cian.py / cian_detail.py.

TBD / known limitations

realtyValuation per-room-type data — доступна только через XHR при switch filter tab. Initial page state даёт aggregate only (room_count='all'). Per-room chart требует separate XHR scraper (Stage 7+ scope).
house_reliability_checks — no UNIQUE constraint в migration 025 → repeated scrapes INSERT new rows. Каллер должен truncate/delete existing если нужен dedup (schema decision deferred; future migration 030+ может добавить UNIQUE(house_id, source, check_name)).

Refs

decisions/CianScraper_v1_Implementation_Plan.md Stage 6
decisions/Schema_Cian_SERP_Inventory.md sec 15-19 (newbuilding state + sister containers)

Test plan

20 unit tests pass
48 Cian-related full suite clean
Ruff clean
Post-deploy live smoke: target ЖК (Екатерининский парк) → houses_price_dynamics 7 monthly points + reliability checks saved

## Summary **Stage 6 / Wave 4** of CianScraper v1. Cian.ru newbuilding (ЖК) catalog page scraper — extracts newbuilding metadata + 6 sister state containers (realtyValuation chart / reliability / reviews / offers / builders / housesByTurn). 🎯 **Main Stage 6 deliverable**: 7-month price dynamics chart → `houses_price_dynamics` table (Cross-source benchmark для Stage 9 estimator). ## Files - `app/services/scrapers/cian_newbuilding.py` (NEW) — `fetch_newbuilding()` + `save_newbuilding_enrichment()` + `NewbuildingEnrichment` dataclass - `tests/test_cian_newbuilding.py` (NEW) — 20 tests ## State extraction - **URL pattern**: `https://zhk-<slug>-<city>-i.cian.ru/` - **MFE**: `newbuilding-card-desktop-fichering-frontend` - **Key**: `initialState` (sister containers live as top-level keys, не separate push entries) ## Sister containers parsed | Container | Shape | Target table | |---|---|---| | `realtyValuation` | `data.priceDynamics.chart.data.{labels[], values[]}` + fallback flat `chart[]` | `houses_price_dynamics` (UPSERT on `houses_price_dynamics_dim_key`) | | `reliability` | `{checkStatus, details[]}` + fallback `checks[]` | `house_reliability_checks` | | `reviews` | `reviews.items[]` | (stash в raw_sister_states; existing `house_reviews` table из migration 014) | | `offers` | `data.total.fromDeveloperRooms[].layouts[]` + `items[]` fast path | nested_offers (used by future matching) | | `builders` / `banks` / `housesByTurn` | top-level `newbuilding.*` jsonb arrays | `houses.builders/banks/houses_by_turn` | ## Persistence (`save_newbuilding_enrichment`) 1. UPSERT `management_companies` (ON CONFLICT `ext_source, ext_id, name`) → get mc_id 2. `UPDATE houses SET cian_internal_house_id, year_built, management_company_id, transport_accessibility_rate, advantages, banks, builders, houses_by_turn` 3. `INSERT INTO houses_price_dynamics ... ON CONFLICT ON CONSTRAINT houses_price_dynamics_dim_key` (post-029 multi-dimension) 4. `INSERT INTO house_reliability_checks` (no UNIQUE — каждая scrape inserts; см. TBD) ## Schema verified - migration 020 — `houses.cian_internal_house_id`, `management_company_id`, `transport_accessibility_rate`, `advantages/banks/builders/houses_by_turn jsonb` - migration 021 — `management_companies UNIQUE (ext_source, name, ext_id)` - migration 029 — `houses_price_dynamics_dim_key (house_id, source, room_count, prices_type, period, month_date)` — exact ON CONFLICT match ## Tests - 20/20 pass + 48/48 full Cian regression suite clean - Coverage: chart normalization, reliability extraction, address/deadline polymorphism, dataclass defaults, async mocked fetch_newbuilding (no-state graceful) ## Ruff Clean (fixed N806 uppercase var, E501 line length, B905 zip strict=). ## Anti-bot `curl_cffi.AsyncSession(impersonate="chrome120")` — consistent with cian.py / cian_detail.py. ## TBD / known limitations 1. **realtyValuation per-room-type data** — доступна только через XHR при switch filter tab. Initial page state даёт aggregate only (`room_count='all'`). Per-room chart требует separate XHR scraper (Stage 7+ scope). 2. **`house_reliability_checks`** — no UNIQUE constraint в migration 025 → repeated scrapes INSERT new rows. Каллер должен truncate/delete existing если нужен dedup (schema decision deferred; future migration 030+ может добавить UNIQUE(house_id, source, check_name)). ## Refs - `decisions/CianScraper_v1_Implementation_Plan.md` Stage 6 - `decisions/Schema_Cian_SERP_Inventory.md` sec 15-19 (newbuilding state + sister containers) ## Test plan - [x] 20 unit tests pass - [x] 48 Cian-related full suite clean - [x] Ruff clean - [ ] Post-deploy live smoke: target ЖК (Екатерининский парк) → houses_price_dynamics 7 monthly points + reliability checks saved

lekss361 added 1 commit 2026-05-23 13:18:18 +00:00

feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) ec196339a6

lekss361 merged commit d28bf4e66a into main

2026-05-23 13:23:11 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 13:23:12 +00:00

feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459)

lekss361 commented

2026-05-23 13:23:49 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict ✅ APPROVE.

Stage 6 / Wave 4 Worker C — CianScraper v1 newbuilding (ЖК) catalog scraper. Squash-merged as d28bf4e6.

Cross-checks verified

Item	Status
`houses_price_dynamics_dim_key` constraint name (migration 029:159) matches `ON CONFLICT ON CONSTRAINT` in cian_newbuilding.py	✅ exact match
Migration 020 — `houses.cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn` all present	✅
Migration 021 — `management_companies UNIQUE (ext_source, name, ext_id)` matches `ON CONFLICT (ext_source, name, ext_id)` (PR body wrongly says `ext_source, ext_id, name` — actual SQL order is correct)	✅
Migration 025 — `house_reliability_checks (house_id, check_status, check_name, details, source, recorded_at)` columns match INSERT	✅
Chart extraction graceful: returns `[]` on empty/None/missing dict	✅ verified by `test_extract_chart_handles_empty`
Chart None values skipped in `zip(labels, values, strict=False)`	✅
`CAST(:x AS type)` everywhere — no `:x::type` (CAST trap rule)	✅
Anti-bot `curl_cffi impersonate="chrome120"` consistent with cian.py / cian_detail.py	✅
Atomic transaction: single `db.commit()` at end → UPDATE houses + INSERTs roll back together on failure	✅
Tests 20/20 + Cian regression 48/48 (per PR body)	✅

Minor notes (non-blocking, follow-up in vault / future PR)

management_companies dedup with NULL ext_id — Postgres NULL != NULL → UNIQUE (ext_source, name, ext_id) allows duplicates when ext_id IS NULL. Bounded risk (1 dup per ЖК per scrape if Cian omits id). Mitigation idea: coalesce ext_id to hash of name, or partial unique index on (ext_source, name) WHERE ext_id IS NULL.
houses_price_dynamics.price_per_sqm semantic mismatch — primary chart path stores total RUB values (e.g. 14_000_000) into column literally named price_per_sqm with prices_type='price' discriminator. PR #448 backfill set legacy rows to prices_type='priceSqm'/'allTime'/'all'. Stage 9 estimator MUST filter by prices_type to avoid mixing total/sqm price series. Acceptable trade-off per PR TBD #1.
offers items[] fast path (line ~285): returns raw list without dict-only filter or room_count injection. Downstream may get heterogeneous items. Low impact (offers consumed by future Stage 9 matching, not Stage 6 chart).
save_newbuilding_enrichment caller contract — UPDATE houses WHERE id=:hid silently no-ops if house missing; subsequent FK to houses_price_dynamics.house_id would then raise. Caller must guarantee house_id exists. Consider explicit RETURNING id check or assert cursor.rowcount == 1.
house_reliability_checks unbounded growth — no UNIQUE, every scrape inserts. Documented as TBD. Future migration 030+ should add UNIQUE (house_id, source, check_name) or per-scrape truncate.
Test gap — no integration test for save_newbuilding_enrichment (full DB orchestration) and no per-container failure isolation test. Pilot-acceptable.

Stage 9 readiness

Chart data flows into houses_price_dynamics with proper multi-dim discriminators (source='cian_realty_valuation', room_count='all', prices_type='price', period='halfYear'). Stage 9 estimator must:

Filter by source IN ('cian_realty_valuation', ...) and explicit prices_type to avoid mixing scales.
Be aware that initial-state chart is aggregate only (per-room data requires Stage 7+ XHR scraper, per TBD #1).

Post-merge expectations (per user request — NOT running qa-tester)

deploy-tradein.yml should trigger on merge to main.
CI: pytest 20/20 (cian_newbuilding) + 48/48 (Cian regression) should pass green.
Post-deploy live smoke: target Екатерининский парк ЖК URL → SELECT COUNT(*) FROM houses_price_dynamics WHERE house_id = ? AND source = 'cian_realty_valuation' AND prices_type = 'price' should return 7 monthly points.

Vault update suggestion: code/modules/tradein/Cian_Newbuilding_Parser.md documenting the prices_type='price' vs 'priceSqm' coexistence + management_companies NULL ext_id risk.

Merged via deep-code-reviewer — verdict ✅ APPROVE. Stage 6 / Wave 4 Worker C — CianScraper v1 newbuilding (ЖК) catalog scraper. Squash-merged as `d28bf4e6`. ## Cross-checks verified | Item | Status | |---|---| | `houses_price_dynamics_dim_key` constraint name (migration 029:159) matches `ON CONFLICT ON CONSTRAINT` in cian_newbuilding.py | ✅ exact match | | Migration 020 — `houses.cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn` all present | ✅ | | Migration 021 — `management_companies UNIQUE (ext_source, name, ext_id)` matches `ON CONFLICT (ext_source, name, ext_id)` (PR body wrongly says `ext_source, ext_id, name` — actual SQL order is correct) | ✅ | | Migration 025 — `house_reliability_checks (house_id, check_status, check_name, details, source, recorded_at)` columns match INSERT | ✅ | | Chart extraction graceful: returns `[]` on empty/None/missing dict | ✅ verified by `test_extract_chart_handles_empty` | | Chart None values skipped in `zip(labels, values, strict=False)` | ✅ | | `CAST(:x AS type)` everywhere — no `:x::type` (CAST trap rule) | ✅ | | Anti-bot `curl_cffi impersonate="chrome120"` consistent with cian.py / cian_detail.py | ✅ | | Atomic transaction: single `db.commit()` at end → UPDATE houses + INSERTs roll back together on failure | ✅ | | Tests 20/20 + Cian regression 48/48 (per PR body) | ✅ | ## Minor notes (non-blocking, follow-up in vault / future PR) 1. **management_companies dedup with NULL ext_id** — Postgres `NULL != NULL` → `UNIQUE (ext_source, name, ext_id)` allows duplicates when `ext_id IS NULL`. Bounded risk (1 dup per ЖК per scrape if Cian omits id). Mitigation idea: coalesce ext_id to hash of name, or partial unique index on `(ext_source, name) WHERE ext_id IS NULL`. 2. **`houses_price_dynamics.price_per_sqm` semantic mismatch** — primary chart path stores total RUB values (e.g. 14_000_000) into column literally named `price_per_sqm` with `prices_type='price'` discriminator. PR #448 backfill set legacy rows to `prices_type='priceSqm'/'allTime'/'all'`. **Stage 9 estimator MUST filter by `prices_type` to avoid mixing total/sqm price series.** Acceptable trade-off per PR TBD #1. 3. **`offers items[]` fast path** (line ~285): returns raw list without dict-only filter or `room_count` injection. Downstream may get heterogeneous items. Low impact (offers consumed by future Stage 9 matching, not Stage 6 chart). 4. **`save_newbuilding_enrichment` caller contract** — `UPDATE houses WHERE id=:hid` silently no-ops if house missing; subsequent FK to `houses_price_dynamics.house_id` would then raise. Caller must guarantee `house_id` exists. Consider explicit `RETURNING id` check or `assert cursor.rowcount == 1`. 5. **`house_reliability_checks` unbounded growth** — no UNIQUE, every scrape inserts. Documented as TBD. Future migration 030+ should add `UNIQUE (house_id, source, check_name)` or per-scrape truncate. 6. **Test gap** — no integration test for `save_newbuilding_enrichment` (full DB orchestration) and no per-container failure isolation test. Pilot-acceptable. ## Stage 9 readiness Chart data flows into `houses_price_dynamics` with proper multi-dim discriminators (`source='cian_realty_valuation', room_count='all', prices_type='price', period='halfYear'`). Stage 9 estimator must: - Filter by `source IN ('cian_realty_valuation', ...)` and explicit `prices_type` to avoid mixing scales. - Be aware that initial-state chart is aggregate only (per-room data requires Stage 7+ XHR scraper, per TBD #1). ## Post-merge expectations (per user request — NOT running qa-tester) 1. `deploy-tradein.yml` should trigger on merge to `main`. 2. CI: pytest 20/20 (cian_newbuilding) + 48/48 (Cian regression) should pass green. 3. Post-deploy live smoke: target Екатерининский парк ЖК URL → `SELECT COUNT(*) FROM houses_price_dynamics WHERE house_id = ? AND source = 'cian_realty_valuation' AND prices_type = 'price'` should return 7 monthly points. Vault update suggestion: `code/modules/tradein/Cian_Newbuilding_Parser.md` documenting the prices_type='price' vs 'priceSqm' coexistence + management_companies NULL ext_id risk.