lekss361/gendesign

Fork 0

dynamics tables (PR #445 follow-up) #448

Merged

lekss361 merged 1 commit from feat/tradein-sql-029-extend-matching-valuation into main

2026-05-23 12:35:04 +00:00

lekss361 commented

2026-05-23 12:29:38 +00:00

Owner

Summary

Follow-up к PR #445 (Stage 1 SQL migrations). Addresses 3 MEDIUM follow-ups from deep-code-reviewer + 1 missing column для Tier-0 matching.

Blocks Wave 4-6 stages (cian_detail/newbuilding/valuation scrapers + matching service).

Sections

Section	Target	Purpose
A	`house_sources` ALTER	+ ext_url, raw_payload (jsonb GIN), last_seen_at (idx DESC)
B	`listing_sources` ALTER	+ price_rub, area_m2, floor, rooms_count, raw_payload (GIN), last_seen_at — для cross-source price divergence detection
C	`external_valuations` ALTER	+ sale/rent split (price + accuracy), chart_change_pct/direction, filters_hash, external_house_id
D	`houses_price_dynamics` ALTER	+ room_count, prices_type, period dimensions; DROP+REPLACE UNIQUE constraint (defensive DO block с pg_constraint exists check)
E	`houses` ALTER	+ cadastral_number (text) + BTree index — для Tier-0 matching в Stage 8

Idempotency

Section D — единственное non-trivial место. Wrapped в DO $$ ... $$ block:

Drops houses_price_dynamics_house_id_month_date_source_key if exists
Adds houses_price_dynamics_dim_key UNIQUE on (house_id, source, room_count, prices_type, period, month_date) if not exists
Backfill existing rows: room_count='all', prices_type='priceSqm', period='allTime'

Re-running migration safe.

Deviations from spec

room_count typed text not int — vault sec 16.3 specifies text enum values ('studio'/'oneRoom'/'twoRoom') matching Cian's API. Avoids mapping layer bug.
listing_sources.last_seen_at дублирует существующий last_scraped_at — kept both для backward compat (existing code reads last_scraped_at; new code uses semantic alias).
external_house_id typed bigint not text — Cian's houseId is always integer per vault sec 26.3.

Refs

decisions/Cross_Source_Matching_Strategy.md sec 2.1, 2.2
decisions/Schema_Cian_SERP_Inventory.md sec 16.3, 26
PR #445 deep-code-reviewer comments (follow-ups #1-3)

Test plan

CI green (deploy.yml psql apply)
Verify existing houses_price_dynamics rows survive UNIQUE replacement (backfill UPDATE in migration)
Verify GIN index creation succeeds on raw_payload cols

## Summary Follow-up к PR #445 (Stage 1 SQL migrations). Addresses 3 MEDIUM follow-ups from deep-code-reviewer + 1 missing column для Tier-0 matching. **Blocks Wave 4-6 stages** (cian_detail/newbuilding/valuation scrapers + matching service). ## Sections | Section | Target | Purpose | |---|---|---| | A | `house_sources` ALTER | + ext_url, raw_payload (jsonb GIN), last_seen_at (idx DESC) | | B | `listing_sources` ALTER | + price_rub, area_m2, floor, rooms_count, raw_payload (GIN), last_seen_at — для cross-source price divergence detection | | C | `external_valuations` ALTER | + sale/rent split (price + accuracy), chart_change_pct/direction, filters_hash, external_house_id | | D | `houses_price_dynamics` ALTER | + room_count, prices_type, period dimensions; **DROP+REPLACE** UNIQUE constraint (defensive DO block с pg_constraint exists check) | | E | `houses` ALTER | + cadastral_number (text) + BTree index — для Tier-0 matching в Stage 8 | ## Idempotency Section D — единственное non-trivial место. Wrapped в `DO $$ ... $$` block: - Drops `houses_price_dynamics_house_id_month_date_source_key` if exists - Adds `houses_price_dynamics_dim_key` UNIQUE on `(house_id, source, room_count, prices_type, period, month_date)` if not exists - Backfill existing rows: `room_count='all'`, `prices_type='priceSqm'`, `period='allTime'` Re-running migration safe. ## Deviations from spec 1. **`room_count` typed `text` not `int`** — vault sec 16.3 specifies text enum values (`'studio'/'oneRoom'/'twoRoom'`) matching Cian's API. Avoids mapping layer bug. 2. **`listing_sources.last_seen_at` дублирует существующий `last_scraped_at`** — kept both для backward compat (existing code reads last_scraped_at; new code uses semantic alias). 3. **`external_house_id` typed `bigint` not `text`** — Cian's `houseId` is always integer per vault sec 26.3. ## Refs - `decisions/Cross_Source_Matching_Strategy.md` sec 2.1, 2.2 - `decisions/Schema_Cian_SERP_Inventory.md` sec 16.3, 26 - PR #445 deep-code-reviewer comments (follow-ups #1-3) ## Test plan - [ ] CI green (deploy.yml psql apply) - [ ] Verify existing `houses_price_dynamics` rows survive UNIQUE replacement (backfill UPDATE in migration) - [ ] Verify GIN index creation succeeds on raw_payload cols

lekss361 added 1 commit 2026-05-23 12:29:39 +00:00

feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) 697047d17d

Addresses 3 MEDIUM follow-ups from deep-code-reviewer on PR #445:

A. house_sources: add ext_url, raw_payload (GIN), last_seen_at — Stage 8 conflict resolution
B. listing_sources: add price_rub, area_m2, floor, rooms_count, raw_payload (GIN), last_seen_at
C. external_valuations: add sale/rent split columns, chart_change_pct/dir, filters_hash, external_house_id
D. houses_price_dynamics: add room_count/prices_type/period dims + replace 3-col UNIQUE with 6-col
E. houses: add cadastral_number (BTree idx) for Tier-0 Cian Valuation house matching

All ADD COLUMN IF NOT EXISTS. DO $$ block guards UNIQUE constraint replacement idempotently.

lekss361 merged commit 7fad898c7f into main

2026-05-23 12:35:04 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 12:35:05 +00:00

feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) (#448)

lekss361 commented

2026-05-23 12:35:33 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Section D verification (UNIQUE constraint replacement):

Old constraint name houses_price_dynamics_house_id_month_date_source_key — confirmed correct (PG auto-generated from anonymous UNIQUE (house_id, month_date, source) in 024_houses_price_dynamics.sql:19)
Backfill UPDATE ordered BEFORE constraint creation — existing rows get (room_count='all', prices_type='priceSqm', period='allTime') so 6-col UNIQUE projection still unique by (house_id, month_date, source) — no dup violation
pg_constraint check uses conrelid::regclass — schema-safe
DO block idempotent (re-run safe)

Paths trigger: .forgejo/workflows/deploy-tradein.yml filter tradein-mvp/** catches tradein-mvp/backend/data/sql/029_*.sql — deploy will trigger.

Dependencies verified: tables houses_price_dynamics (024), house_sources+listing_sources (028), external_valuations (026), houses (009) all exist with expected column names. cadastral_number absent from 009/020 — new column legitimate.

Vault cross-check: decisions/Schema_Cian_SERP_Inventory.md sec 16.3 confirms exact UNIQUE shape (house_id, source, room_count, prices_type, period, month_date) and text enum room_count values — PR matches spec identically.

MINOR (non-blocking):

No CHECK constraint on room_count text enum — typo risk delegated to app-layer Pydantic enum (spec also defers)
last_seen_at vs last_scraped_at redundancy — explicit in PR body, dual-write should be enforced in scraper code

Post-deploy SQL verification (recommended):

SELECT count(*) FROM houses_price_dynamics WHERE room_count IS NULL; → 0
\d houses_price_dynamics → houses_price_dynamics_dim_key UNIQUE present, old ..._key gone
SELECT cadastral_number FROM houses LIMIT 5; → column exists

Merged via deep-code-reviewer — verdict APPROVE. **Section D verification (UNIQUE constraint replacement)**: - Old constraint name `houses_price_dynamics_house_id_month_date_source_key` — confirmed correct (PG auto-generated from anonymous `UNIQUE (house_id, month_date, source)` in `024_houses_price_dynamics.sql:19`) - Backfill UPDATE ordered BEFORE constraint creation — existing rows get `(room_count='all', prices_type='priceSqm', period='allTime')` so 6-col UNIQUE projection still unique by `(house_id, month_date, source)` — no dup violation - `pg_constraint` check uses `conrelid::regclass` — schema-safe - DO block idempotent (re-run safe) **Paths trigger**: `.forgejo/workflows/deploy-tradein.yml` filter `tradein-mvp/**` catches `tradein-mvp/backend/data/sql/029_*.sql` — deploy will trigger. **Dependencies verified**: tables `houses_price_dynamics` (024), `house_sources`+`listing_sources` (028), `external_valuations` (026), `houses` (009) all exist with expected column names. `cadastral_number` absent from 009/020 — new column legitimate. **Vault cross-check**: `decisions/Schema_Cian_SERP_Inventory.md sec 16.3` confirms exact UNIQUE shape `(house_id, source, room_count, prices_type, period, month_date)` and text enum `room_count` values — PR matches spec identically. **MINOR (non-blocking)**: - No CHECK constraint on `room_count` text enum — typo risk delegated to app-layer Pydantic enum (spec also defers) - `last_seen_at` vs `last_scraped_at` redundancy — explicit in PR body, dual-write should be enforced in scraper code **Post-deploy SQL verification (recommended)**: 1. `SELECT count(*) FROM houses_price_dynamics WHERE room_count IS NULL;` → 0 2. `\d houses_price_dynamics` → `houses_price_dynamics_dim_key` UNIQUE present, old `..._key` gone 3. `SELECT cadastral_number FROM houses LIMIT 5;` → column exists

lekss361 referenced this pull request

2026-05-23 12:38:03 +00:00

feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) #449

lekss361 referenced this pull request

2026-05-23 12:48:18 +00:00

feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) #450

lekss361 referenced this pull request from a commit

2026-05-23 12:53:19 +00:00

feat(tradein): SQL migrations 030-032 — Yandex schema extensions

lekss361 referenced this pull request

2026-05-23 12:54:11 +00:00

feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) #453