feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) #464

Merged

lekss361 merged 1 commit from feat/tradein-matching-schema into main

2026-05-23 13:30:23 +00:00

lekss361 commented

2026-05-23 13:25:23 +00:00

Owner

Motivation

Phase 1.1 of Multi-Source Integration. Foundation for matching module (Phase 1.2)

search optimization (Phase 3.1) + estimate refactor (Phase 3.3).

Gap-audit finding

Original spec proposed 7 CREATE TABLE migrations. Audit of existing SQL
(014/017/020/026/028/029/030/031) revealed 80%+ of planned schema already
in place. This PR ships thin delta only (mostly ALTER ADD COLUMN
IF NOT EXISTS + 3 views).

Files

#	File	Type	Lines
040	houses_extend	ALTER + 1 index	24
041	house_sources_noop	COMMENTs only	15
042	listing_sources_price_divergence_idx	1 index	11
043	house_reviews_extend	ALTER + 1 column	12
044	external_valuations_link	ALTER + 3 indexes	33
045	house_placement_history_extend	ALTER + 1 index	27
046	views	3 views (CREATE OR REPLACE)	110

Total: 7 files / 232 lines SQL.

Coordination

NO CREATE TABLE — all targeted tables exist (014/017/026/028)
NO column renames — vocabulary aligned via COMMENT ON COLUMN
NO touching of 030_ collision (030_avito_imv_cache_key_unique + 030_listings_alter_yandex both kept as-is)
NO Python changes — pure SQL PR
Idempotent (every ADD IF NOT EXISTS, every index IF NOT EXISTS, views OR REPLACE)

Smoke verification (post-merge on prod)

-- 3 + 1 + 4 + 2 = 10 new columns:
SELECT column_name FROM information_schema.columns
 WHERE (table_name='houses' AND column_name IN ('complex_id','avito_validated_at','cian_validated_at'))
    OR (table_name='house_reviews' AND column_name='likes')
    OR (table_name='external_valuations' AND column_name IN ('house_id','listing_id','low_price','high_price'))
    OR (table_name='house_placement_history' AND column_name IN ('source_confidence','notes'));

-- 3 new views:
SELECT table_name FROM information_schema.views
 WHERE table_name IN ('v_price_divergence','v_cross_source_health','v_data_quality');

-- Smoke select:
SELECT * FROM v_data_quality;

Reviewer

deep-code-reviewer — please verify:

No CREATE TABLE accidentally introduced
All ALTER ADD COLUMN have IF NOT EXISTS
v_data_quality references only existing columns (lat, kadastr_num, description, is_active, is_outlier, source, house_id_fk)
No conflict with PR #456 (Python-only, different directory)
044 dual-UNIQUE strategy (cache_key from 030 + new partial canonical_uniq) doesn't lock out IMV cache inserts that have NULL house_id/listing_id

Refs

Master Plan sec 1, 4-8 — decisions/MultiSource_Integration_Master_Plan.md
Cross_Source_Matching sec 2-4 — decisions/Cross_Source_Matching_Strategy.md
Coordination ranges: Avito 010-018, Cian 019-028, Yandex 029-032, Matching 040-046, Search 050+

## Motivation Phase 1.1 of Multi-Source Integration. Foundation for matching module (Phase 1.2) + search optimization (Phase 3.1) + estimate refactor (Phase 3.3). ## Gap-audit finding Original spec proposed 7 CREATE TABLE migrations. Audit of existing SQL (014/017/020/026/028/029/030/031) revealed 80%+ of planned schema already in place. This PR ships **thin delta only** (mostly ALTER ADD COLUMN IF NOT EXISTS + 3 views). ## Files | # | File | Type | Lines | |---|---|---|---| | 040 | houses_extend | ALTER + 1 index | 24 | | 041 | house_sources_noop | COMMENTs only | 15 | | 042 | listing_sources_price_divergence_idx | 1 index | 11 | | 043 | house_reviews_extend | ALTER + 1 column | 12 | | 044 | external_valuations_link | ALTER + 3 indexes | 33 | | 045 | house_placement_history_extend | ALTER + 1 index | 27 | | 046 | views | 3 views (CREATE OR REPLACE) | 110 | **Total**: 7 files / 232 lines SQL. ## Coordination - **NO** CREATE TABLE — all targeted tables exist (014/017/026/028) - **NO** column renames — vocabulary aligned via COMMENT ON COLUMN - **NO** touching of 030_ collision (`030_avito_imv_cache_key_unique` + `030_listings_alter_yandex` both kept as-is) - **NO** Python changes — pure SQL PR - Idempotent (every ADD `IF NOT EXISTS`, every index `IF NOT EXISTS`, views `OR REPLACE`) ## Smoke verification (post-merge on prod) ```sql -- 3 + 1 + 4 + 2 = 10 new columns: SELECT column_name FROM information_schema.columns WHERE (table_name='houses' AND column_name IN ('complex_id','avito_validated_at','cian_validated_at')) OR (table_name='house_reviews' AND column_name='likes') OR (table_name='external_valuations' AND column_name IN ('house_id','listing_id','low_price','high_price')) OR (table_name='house_placement_history' AND column_name IN ('source_confidence','notes')); -- 3 new views: SELECT table_name FROM information_schema.views WHERE table_name IN ('v_price_divergence','v_cross_source_health','v_data_quality'); -- Smoke select: SELECT * FROM v_data_quality; ``` ## Reviewer **deep-code-reviewer** — please verify: 1. No CREATE TABLE accidentally introduced 2. All ALTER ADD COLUMN have IF NOT EXISTS 3. v_data_quality references only existing columns (`lat`, `kadastr_num`, `description`, `is_active`, `is_outlier`, `source`, `house_id_fk`) 4. No conflict with PR #456 (Python-only, different directory) 5. 044 dual-UNIQUE strategy (cache_key from 030 + new partial canonical_uniq) doesn't lock out IMV cache inserts that have NULL house_id/listing_id ## Refs - Master Plan sec 1, 4-8 — `decisions/MultiSource_Integration_Master_Plan.md` - Cross_Source_Matching sec 2-4 — `decisions/Cross_Source_Matching_Strategy.md` - Coordination ranges: Avito 010-018, Cian 019-028, Yandex 029-032, Matching 040-046, Search 050+

lekss361 added 1 commit 2026-05-23 13:25:24 +00:00

feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) d019b5327e

Multi-Source Integration Phase 1.1 thin-delta migrations. Gap audit showed
80%+ of planned schema already in 014/017/020/026/028/029/030/031 —
this PR adds only the missing pieces:

- 040 houses_extend: complex_id + avito_validated_at + cian_validated_at
- 041 house_sources_noop: COMMENT aliases for ext_source/ext_id/matched_method
- 042 listing_sources_price_divergence_idx: partial index for /search
- 043 house_reviews_extend: likes column
- 044 external_valuations_link: house_id/listing_id FKs + low/high price band
- 045 house_placement_history_extend: source_confidence + notes + 3-col UNIQUE
- 046 views: v_price_divergence + v_cross_source_health + v_data_quality

All BEGIN/COMMIT, IF NOT EXISTS, FK ON DELETE CASCADE.
Idempotent — safe re-apply on prod.

Refs Master Plan sec 1, 4-8 + Cross_Source_Matching sec 2-4.

lekss361 merged commit 5e90797a9f into main

2026-05-23 13:30:23 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 13:30:24 +00:00

feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464)

lekss361 commented

2026-05-23 13:30:51 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE (squash 5e90797a).

Verification done

All 5 reviewer asks pass:
1. No CREATE TABLE in PR diff (grep confirmed: only ALTER + CREATE INDEX + CREATE OR REPLACE VIEW)
2. All 10 ADD COLUMN have IF NOT EXISTS (040 ×3, 043 ×1, 044 ×4, 045 ×2)
3. v_data_quality column refs valid: listings.lat / kadastr_num / description / is_active / is_outlier / source / scraped_at / house_id_fk all exist (002+011); houses.year_built (009); house_sources.house_id / ext_source (028); listings WHERE source IN ('avito','cian','yandex') matches existing source values
4. No conflict with PR #456 — pure SQL in data/sql/, no overlap with Python scrapers/
5. 044 dual-UNIQUE strategy safe:
  - Existing UNIQUE (source, cache_key) from 026 — every row hits this
  - New UNIQUE (source, house_id, listing_id) WHERE house_id IS NOT NULL AND listing_id IS NOT NULL — partial, IMV cache rows with NULL canonical refs bypass it cleanly
  - Cross-source dedup of canonical (matched) rows guaranteed without locking out pre-matching cache writes

Idempotency — every file wrapped BEGIN; ... COMMIT;, every DDL guarded (IF NOT EXISTS / CREATE OR REPLACE). Re-apply safe.

Cross-file impact

Numbering 040-046 — no collision (next free slot was 040 after 032; lex-sort preserves order)
FK ON DELETE CASCADE on external_valuations.house_id / listing_id — acceptable (cache rows become orphan anyway if parent deleted)
045 ADD COLUMN source_confidence real DEFAULT 1.0 — PG ≥11 metadata-only, no full table rewrite

Minor (non-blocking)

v_data_quality is a regular view — every call re-executes 15+ subqueries. If /api/v1/admin/data-quality polls frequently, consider MATERIALIZED VIEW + REFRESH MATERIALIZED VIEW CONCURRENTLY (deferred — current scope OK)
3 grouping queries in v_data_quality (houses 2plus_sources, 3plus_sources, listings_dedup_2sources) repeat full GROUP BY; could share a CTE — defer

Phase 1.1 unblocks Phase 1.2 (matching module Python), Phase 3.1 (search optimization), Phase 3.3 (estimate refactor).

Merged via deep-code-reviewer — verdict APPROVE (squash 5e90797a). **Verification done** - All 5 reviewer asks pass: 1. **No CREATE TABLE** in PR diff (grep confirmed: only ALTER + CREATE INDEX + CREATE OR REPLACE VIEW) 2. **All 10 ADD COLUMN have IF NOT EXISTS** (040 ×3, 043 ×1, 044 ×4, 045 ×2) 3. **v_data_quality column refs valid**: `listings.lat / kadastr_num / description / is_active / is_outlier / source / scraped_at / house_id_fk` all exist (002+011); `houses.year_built` (009); `house_sources.house_id / ext_source` (028); `listings WHERE source IN ('avito','cian','yandex')` matches existing source values 4. **No conflict with PR #456** — pure SQL in `data/sql/`, no overlap with Python `scrapers/` 5. **044 dual-UNIQUE strategy safe**: - Existing `UNIQUE (source, cache_key)` from 026 — every row hits this - New `UNIQUE (source, house_id, listing_id) WHERE house_id IS NOT NULL AND listing_id IS NOT NULL` — partial, IMV cache rows with NULL canonical refs bypass it cleanly - Cross-source dedup of canonical (matched) rows guaranteed without locking out pre-matching cache writes **Idempotency** — every file wrapped `BEGIN; ... COMMIT;`, every DDL guarded (`IF NOT EXISTS` / `CREATE OR REPLACE`). Re-apply safe. **Cross-file impact** - Numbering 040-046 — no collision (next free slot was 040 after 032; lex-sort preserves order) - FK `ON DELETE CASCADE` on `external_valuations.house_id / listing_id` — acceptable (cache rows become orphan anyway if parent deleted) - `045 ADD COLUMN source_confidence real DEFAULT 1.0` — PG ≥11 metadata-only, no full table rewrite **Minor (non-blocking)** - `v_data_quality` is a regular view — every call re-executes 15+ subqueries. If `/api/v1/admin/data-quality` polls frequently, consider `MATERIALIZED VIEW` + `REFRESH MATERIALIZED VIEW CONCURRENTLY` (deferred — current scope OK) - 3 grouping queries in `v_data_quality` (`houses 2plus_sources`, `3plus_sources`, `listings_dedup_2sources`) repeat full GROUP BY; could share a CTE — defer Phase 1.1 unblocks Phase 1.2 (matching module Python), Phase 3.1 (search optimization), Phase 3.3 (estimate refactor).

lekss361 referenced this pull request

2026-05-23 13:47:48 +00:00

feat(tradein): search matview + indexes (Phase 3.1) #469

lekss361 referenced this pull request

2026-05-23 14:21:59 +00:00

feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources #475