feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) #455

Merged

lekss361 merged 1 commit from feat/tradein-base-save-listings-cian-cols into main

2026-05-23 13:05:20 +00:00

lekss361 commented

2026-05-23 12:59:37 +00:00

Owner

Summary

Stage 3.5 of CianScraper v1 — addresses PR #450 deep-code-reviewer MEDIUM follow-up. Without this fix, all Stage-2 model fields populated by _offer_to_lot are silently dropped at INSERT → Stage 8 cross-source matching cannot use description_minhash / cadastral_number / building_cadastral_number signals.

Must merge before Wave 4 (Stages 5/6) — иначе detail/newbuilding scrapers тоже будут терять данные.

Diff

base.py: +38 lines (extend INSERT + ON CONFLICT UPDATE SET)
tests/test_base_save_listings.py: NEW 260 lines, 6 tests

Fields persisted (13)

All confirmed present в ScrapedLot top-level (PR #447, base.py:97-109):

Field	Column	Type	Purpose
`living_area_m2`	`living_area_m2`	numeric	apartment dimensions
`bedrooms_count`	`bedrooms_count`	int
`balconies_count`	`balconies_count`	int
`loggias_count`	`loggias_count`	int
`description_minhash`	`description_minhash`	text	Stage 8 matching signal
`cadastral_number`	`cadastral_number`	text	Stage 8 Tier-0 matching
`building_cadastral_number`	`building_cadastral_number`	text	Stage 8 house matching
`phones`	`phones`	jsonb	seller contact
`is_homeowner`	`is_homeowner`	bool	seller type filter
`is_pro_seller`	`is_pro_seller`	bool
`bargain_allowed`	`bargain_allowed`	bool
`sale_type`	`sale_type`	text	direct / mortgage / military
`metro_stations`	`metro_stations`	jsonb	transport accessibility

ON CONFLICT

DO UPDATE SET extended — re-scrape refreshes all 13 new cols (not just inserts on first scrape).

Backward compat

test_save_listings_avito_compat_cian_cols_are_null verifies — Avito lot без Cian fields → все 13 params = None → SQL NULL. No regression in existing Avito/DomRF/other scrapers.

Fields NOT yet in ScrapedLot (deferred to Stage 5)

windows_view_type, separate_wcs_count, combined_wcs_count, ceiling_height, repair_type — это detail-page fields. Currently stored in raw_payload only. Stage 5 (cian_detail.py) или follow-up PR может promote to top-level + extend INSERT.

Tests (6)

Cian lot full populate → SELECT verifies all 13 cols persisted correctly
Avito lot empty cian fields → NULL persisted (no regression)
ON CONFLICT DO UPDATE refreshes Cian cols on re-scrape
Empty phones[] / metro_stations[] → NULL (not empty array)
SQL text inspection — INSERT statement contains all 13 col names
SQL text inspection — UPDATE SET clause contains all 13 col names

Full suite: 77/77 pass.

psycopg v3 compliance

jsonb cols passed as json.dumps(value) if value else None
No :x::type casts (CAST(:x AS type) where needed)
No import psycopg2

Refs

PR #450 deep-code-reviewer comment (full follow-up text)
decisions/Schema_Cian_SERP_Inventory.md sec 11 (field mapping)
decisions/CianScraper_v1_Implementation_Plan.md Stages 2/8

Test plan

6 new tests pass
Full backend test suite 77/77 green
Ruff clean (4 pre-existing E501 in UA strings unchanged)
Post-merge: verify next Cian scrape cycle populates new cols in prod DB

## Summary **Stage 3.5** of CianScraper v1 — addresses PR #450 deep-code-reviewer MEDIUM follow-up. Without this fix, all Stage-2 model fields populated by `_offer_to_lot` are silently dropped at INSERT → **Stage 8 cross-source matching cannot use `description_minhash` / `cadastral_number` / `building_cadastral_number` signals**. Must merge **before Wave 4** (Stages 5/6) — иначе detail/newbuilding scrapers тоже будут терять данные. ## Diff - `base.py`: +38 lines (extend INSERT + ON CONFLICT UPDATE SET) - `tests/test_base_save_listings.py`: NEW 260 lines, 6 tests ## Fields persisted (13) All confirmed present в `ScrapedLot` top-level (PR #447, base.py:97-109): | Field | Column | Type | Purpose | |---|---|---|---| | `living_area_m2` | `living_area_m2` | numeric | apartment dimensions | | `bedrooms_count` | `bedrooms_count` | int | | | `balconies_count` | `balconies_count` | int | | | `loggias_count` | `loggias_count` | int | | | `description_minhash` | `description_minhash` | text | **Stage 8 matching signal** | | `cadastral_number` | `cadastral_number` | text | **Stage 8 Tier-0 matching** | | `building_cadastral_number` | `building_cadastral_number` | text | **Stage 8 house matching** | | `phones` | `phones` | jsonb | seller contact | | `is_homeowner` | `is_homeowner` | bool | seller type filter | | `is_pro_seller` | `is_pro_seller` | bool | | | `bargain_allowed` | `bargain_allowed` | bool | | | `sale_type` | `sale_type` | text | direct / mortgage / military | | `metro_stations` | `metro_stations` | jsonb | transport accessibility | ## ON CONFLICT `DO UPDATE SET` extended — re-scrape refreshes all 13 new cols (not just inserts on first scrape). ## Backward compat `test_save_listings_avito_compat_cian_cols_are_null` verifies — Avito lot без Cian fields → все 13 params = `None` → SQL `NULL`. No regression in existing Avito/DomRF/other scrapers. ## Fields NOT yet in ScrapedLot (deferred to Stage 5) `windows_view_type`, `separate_wcs_count`, `combined_wcs_count`, `ceiling_height`, `repair_type` — это detail-page fields. Currently stored in `raw_payload` only. Stage 5 (cian_detail.py) или follow-up PR может promote to top-level + extend INSERT. ## Tests (6) 1. Cian lot full populate → SELECT verifies all 13 cols persisted correctly 2. Avito lot empty cian fields → NULL persisted (no regression) 3. ON CONFLICT DO UPDATE refreshes Cian cols on re-scrape 4. Empty phones[] / metro_stations[] → NULL (not empty array) 5. SQL text inspection — INSERT statement contains all 13 col names 6. SQL text inspection — UPDATE SET clause contains all 13 col names Full suite: 77/77 pass. ## psycopg v3 compliance - jsonb cols passed as `json.dumps(value) if value else None` - No `:x::type` casts (CAST(:x AS type) where needed) - No `import psycopg2` ## Refs - PR #450 deep-code-reviewer comment (full follow-up text) - `decisions/Schema_Cian_SERP_Inventory.md` sec 11 (field mapping) - `decisions/CianScraper_v1_Implementation_Plan.md` Stages 2/8 ## Test plan - [x] 6 new tests pass - [x] Full backend test suite 77/77 green - [x] Ruff clean (4 pre-existing E501 in UA strings unchanged) - [ ] Post-merge: verify next Cian scrape cycle populates new cols in prod DB

lekss361 added 1 commit 2026-05-23 12:59:37 +00:00

feat(tradein): extend save_listings INSERT to persist Cian-specific columns (PR #450 follow-up) 702cb2b7d0

Extends base.py save_listings() to persist all 13 Cian-specific columns
added to ScrapedLot in PR #447 and listings DDL in migration 019:
living_area_m2, bedrooms_count, balconies_count, loggias_count,
description_minhash, cadastral_number, building_cadastral_number,
phones, is_homeowner, is_pro_seller, bargain_allowed, sale_type,
metro_stations.

ON CONFLICT DO UPDATE SET extended to refresh all new columns on re-scrape.
Avito/DomRF backward compat: optional fields default None → SQL NULL.

Adds 6 unit tests (test_base_save_listings.py) covering Cian INSERT,
Avito null compat, ON CONFLICT UPDATE SQL text, empty phones/metro → NULL.

lekss361 merged commit 1b3edff6a9 into main

2026-05-23 13:05:20 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 13:05:22 +00:00

feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455)

lekss361 commented

2026-05-23 13:05:45 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Verification summary:

Column ↔ placeholder order: 13/13 aligned pair-by-pair (living_area_m2 → metro_stations)
DDL type compat (migration 019): all 13 types match (numeric / int / text / bool / jsonb)
psycopg v3 compliance: CAST(:phones AS jsonb) + CAST(:metro_stations AS jsonb) correct, no :x::type casts
Backward compat: grepped avito.py / yandex_realty.py / n1.py — no scraper passes any of 13 new fields → all default None (or [] for phones/metro_stations) → INSERT writes NULL → no regression
ON CONFLICT DO UPDATE SET: 13/13 EXCLUDED references correct, overwrite strategy consistent with PR claim
Test #4 (empty list → NULL): consistent with if lot.phones else None helper; aligned with _to_json([])="[]" avoided

Minor observations (non-blocking, для будущих PR):

Migration 011 sets metro_stations DEFAULT '[]'::jsonb; this PR writes explicit NULL when list empty. Existing Avito rows have '[]', new Avito rows have NULL — cosmetic inconsistency, Stage 8 matching should COALESCE(metro_stations, '[]') defensively.
phones/metro_stations collapse "scraper didn't fetch" (None) и "scraper got 0 results" ([]) в SQL NULL — acceptable trade-off per PR body, no caller currently distinguishes.
Deferred fields (windows_view_type, separate_wcs_count, etc) — Stage 5 should follow the same INSERT-extension pattern. Recommend: when adding ScrapedLot field, extend save_listings в same PR.

Wave 4 (Stages 5/6 cian_detail/cian_newbuilding) unblocked.

Merged via deep-code-reviewer — verdict APPROVE. **Verification summary:** - Column ↔ placeholder order: 13/13 aligned pair-by-pair (living_area_m2 → metro_stations) - DDL type compat (migration 019): all 13 types match (numeric / int / text / bool / jsonb) - psycopg v3 compliance: `CAST(:phones AS jsonb)` + `CAST(:metro_stations AS jsonb)` correct, no `:x::type` casts - Backward compat: grepped avito.py / yandex_realty.py / n1.py — no scraper passes any of 13 new fields → all default `None` (or `[]` for phones/metro_stations) → INSERT writes NULL → no regression - ON CONFLICT DO UPDATE SET: 13/13 EXCLUDED references correct, overwrite strategy consistent with PR claim - Test #4 (empty list → NULL): consistent with `if lot.phones else None` helper; aligned with `_to_json([])="[]"` avoided **Minor observations (non-blocking, для будущих PR):** 1. Migration 011 sets `metro_stations DEFAULT '[]'::jsonb`; this PR writes explicit `NULL` when list empty. Existing Avito rows have `'[]'`, new Avito rows have `NULL` — cosmetic inconsistency, Stage 8 matching should `COALESCE(metro_stations, '[]')` defensively. 2. `phones`/`metro_stations` collapse "scraper didn't fetch" (None) и "scraper got 0 results" (`[]`) в SQL NULL — acceptable trade-off per PR body, no caller currently distinguishes. 3. Deferred fields (windows_view_type, separate_wcs_count, etc) — Stage 5 should follow the same INSERT-extension pattern. Recommend: when adding ScrapedLot field, extend `save_listings` в same PR. Wave 4 (Stages 5/6 cian_detail/cian_newbuilding) unblocked.

lekss361 referenced this pull request

2026-05-23 13:08:48 +00:00

feat(tradein): integrate Avito IMV as 5th evaluation source (on-demand cached) #452