feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache #440

Merged

lekss361 merged 1 commit from feat/tradein-scraper-v2-migrations into main

2026-05-23 11:41:45 +00:00

lekss361 commented

2026-05-23 11:33:07 +00:00

Owner

Summary

Stage 1 of Avito Scraper v2 — foundation для 4-source pipeline (search / detail / Houses Catalog / IMV API). Закрывает C-5 silent corruption (anchor+jitter → точные lat/lon после Stage 2) и подготавливает schema для 60+ новых полей.

Refs:

decisions/AvitoScraper_v2_Implementation_Plan.md — Stage 1
decisions/Schema_Avito_Houses_Listings_History.md — DDL specs (sec 4.2/4.3/4.4/6/7/13/25/29-31)

Files (10 миграций)

File	Content
`009_houses.sql`	CREATE TABLE `houses` (multi-source: avito/cian, geom + GIST)
`010_houses_alter.sql`	+ 15 cols (material_walls/floors, hot_water, parking_type, rating, jsonb fields)
`011_listings_alter.sql`	+ ~14 cols (`house_ext_id`, `kitchen_area_m2`, `owners_count`+`owners_at_least`, `metro_stations`, `detail_enriched_at`, `description`, `views_total`, `house_id_fk`, `listing_segment`)
`012_sellers.sql`	CREATE TABLE `sellers`
`013_listings_alter_seller.sql`	ALTER listings ADD `seller_id_fk`
`014_house_reviews.sql`	CREATE TABLE `house_reviews`
`015_scrape_runs.sql`	CREATE TABLE `scrape_runs` (enum running/done/failed/zombie/skipped/banned)
`016_listings_snapshots.sql`	append-only snapshots (pattern из `objective_lots_history`)
`017_house_placement_history.sql`	+ `removed_date` (заполняется только из IMV API)
`018_avito_imv_evaluations.sql`	IMV cache + `cache_key` index (для on-demand 24h TTL в Stage 3)

Smoke test (local docker)

✅ 7 новых таблиц созданы
✅ 34 индекса (включая imv_cache_key_idx, GIST на houses.geom, partial indexes)
✅ 14 новых колонок в listings
✅ Идемпотентность: повторный apply — только NOTICE, без ошибок

Notes

owners_2plus boolean из sec 13 заменён на owners_count + owners_at_least (per sec 29 update — domoteka exact fields)
map_pins jsonb объявлен в CREATE TABLE (sec 4.2); 010 ALTER оставлен с IF NOT EXISTS — идемпотентно
Все ALTER → nullable + IF NOT EXISTS → rollback тривиален (DROP COLUMN)

Test plan

Apply все 10 файлов в local postgres — pass
Verify 7 таблиц + 14 колонок listings — pass
Idempotency check — pass
Deploy auto-apply через GHA → verify production schema
Stage 2a unblocked после merge

После merge → spawn backend-engineer для Stage 2a (avito.py refactor).

## Summary Stage 1 of **Avito Scraper v2** — foundation для 4-source pipeline (search / detail / Houses Catalog / IMV API). Закрывает C-5 silent corruption (anchor+jitter → точные lat/lon после Stage 2) и подготавливает schema для 60+ новых полей. Refs: - `decisions/AvitoScraper_v2_Implementation_Plan.md` — Stage 1 - `decisions/Schema_Avito_Houses_Listings_History.md` — DDL specs (sec 4.2/4.3/4.4/6/7/13/25/29-31) ## Files (10 миграций) | File | Content | |---|---| | `009_houses.sql` | CREATE TABLE `houses` (multi-source: avito/cian, geom + GIST) | | `010_houses_alter.sql` | + 15 cols (material_walls/floors, hot_water, parking_type, rating, jsonb fields) | | `011_listings_alter.sql` | + ~14 cols (`house_ext_id`, `kitchen_area_m2`, `owners_count`+`owners_at_least`, `metro_stations`, `detail_enriched_at`, `description`, `views_total`, `house_id_fk`, `listing_segment`) | | `012_sellers.sql` | CREATE TABLE `sellers` | | `013_listings_alter_seller.sql` | ALTER listings ADD `seller_id_fk` | | `014_house_reviews.sql` | CREATE TABLE `house_reviews` | | `015_scrape_runs.sql` | CREATE TABLE `scrape_runs` (enum running/done/failed/zombie/skipped/banned) | | `016_listings_snapshots.sql` | append-only snapshots (pattern из `objective_lots_history`) | | `017_house_placement_history.sql` | + `removed_date` (заполняется только из IMV API) | | `018_avito_imv_evaluations.sql` | IMV cache + **`cache_key` index** (для on-demand 24h TTL в Stage 3) | ## Smoke test (local docker) - ✅ 7 новых таблиц созданы - ✅ 34 индекса (включая `imv_cache_key_idx`, GIST на `houses.geom`, partial indexes) - ✅ 14 новых колонок в `listings` - ✅ Идемпотентность: повторный apply — только NOTICE, без ошибок ## Notes - `owners_2plus boolean` из sec 13 заменён на `owners_count + owners_at_least` (per sec 29 update — domoteka exact fields) - `map_pins jsonb` объявлен в CREATE TABLE (sec 4.2); 010 ALTER оставлен с `IF NOT EXISTS` — идемпотентно - Все ALTER → nullable + `IF NOT EXISTS` → rollback тривиален (DROP COLUMN) ## Test plan - [x] Apply все 10 файлов в local postgres — pass - [x] Verify 7 таблиц + 14 колонок listings — pass - [x] Idempotency check — pass - [ ] Deploy auto-apply через GHA → verify production schema - [ ] Stage 2a unblocked после merge ## Next После merge → spawn backend-engineer для Stage 2a (avito.py refactor).

lekss361 added 1 commit 2026-05-23 11:33:07 +00:00

feat(tradein): SQL migrations 009-018 — houses + listings extension + history + IMV cache 28ddf6ef17

Closes Stage 1 of AvitoScraper_v2 plan.

- 009_houses.sql: CREATE TABLE houses (multi-source: avito/cian, geom trigger, 24 cols)
- 010_houses_alter.sql: + 15 cols (material_walls, parking_type, rating_distribution, raw_characteristics, ...)
- 011_listings_alter.sql: + 24 cols (house linking, kitchen_area_m2, owners_count, owners_at_least,
  metro_stations, detail_enriched_at, domoteka/Rosreestr badges, 8 partial indexes)
- 012_sellers.sql: CREATE TABLE sellers (Avito seller cache, no logos)
- 013_listings_alter_seller.sql: listings.seller_id_fk FK
- 014_house_reviews.sql: CREATE TABLE house_reviews (textSections main/pros/cons, no images)
- 015_scrape_runs.sql: CREATE TABLE scrape_runs (status CHECK enum, heartbeat, zombie partial idx)
- 016_listings_snapshots.sql: CREATE TABLE listings_snapshots (append-only, PK listing_id+date)
- 017_house_placement_history.sql: CREATE TABLE house_placement_history (removed_date from IMV only)
- 018_avito_imv_evaluations.sql: CREATE TABLE + cache_key col + imv_cache_key_idx (TTL 24h lookup)

All migrations: BEGIN/COMMIT, IF NOT EXISTS idempotent, PostGIS geom triggers reuse listings_set_geom().
Smoke tested locally: 7 new tables, 34 indexes, 14 listings columns verified.

Refs: decisions/Schema_Avito_Houses_Listings_History.md (sec 4.2, 4.3, 4.4, 6, 7, 13, 25, 29)
Refs: decisions/AvitoScraper_v2_Implementation_Plan.md Stage 1

lekss361 merged commit a2147872ed into main

2026-05-23 11:41:45 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 11:41:45 +00:00

feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440)

lekss361 commented

2026-05-23 11:42:11 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Deep review summary (10 SQL migrations, +615/-0, all P0):

BEGIN/COMMIT wrapper: all 10 files
Idempotency: CREATE TABLE/INDEX IF NOT EXISTS + ADD COLUMN IF NOT EXISTS everywhere
Status enum via text + CHECK (правильный выбор vs CREATE TYPE — нет duplicate_object risk)
FK references resolve in apply order (009→010→011→012→013→014→015→016→017→018)
ON DELETE semantics: CASCADE для child rows (snapshots, reviews, IMV evals), default NO ACTION иначе
GIST на houses.geom ✅
7 partial indexes с sensible WHERE clauses
JSONB (не JSON) для всех jsonb полей
timestamptz consistency ✅
Path trigger tradein-mvp/** в deploy-tradein.yml — корректно ловит tradein-mvp/backend/data/sql/**
Apply order: ls -1 ... | sort (lexicographic 001..018)
Target DB: psql -d tradein (отдельная DB, не main gendesign — zero conflict)

Post-merge verify steps:

deploy-tradein.yml triggers on push to main → check GHA runs
Apply log: → Migration: 009_houses.sql ... 018 sequential. NOTICE: relation already exists, skipping если re-apply, или CREATE TABLE для fresh.
SSH в tradein postgres: \dt → 7 новых tables (houses, sellers, house_reviews, scrape_runs, listings_snapshots, house_placement_history, avito_imv_evaluations) + 14 новых колонок в listings (\d listings).

Minor nits (можно отдельным PR, не блокирующее):

avito_imv_evaluations.cache_key не UNIQUE — append-only история acceptable (audit drift Avito IMV), imv_cache_key_idx (cache_key, fetched_at DESC) поддерживает latest-by-fetched_at lookup
houses.rating numeric(2,1) (009) дублирует rating_score numeric(4,2) (010) — post-Stage 2 consider DROP rating или generated column
listings_snapshots не partitioned — acceptable для pilot (~365k rows/year), revisit при > 10M

Stage 2a unblocked: spawn backend-engineer для avito.py refactor — separate session.

Merged via deep-code-reviewer — verdict APPROVE. **Deep review summary** (10 SQL migrations, +615/-0, all P0): - BEGIN/COMMIT wrapper: all 10 files - Idempotency: `CREATE TABLE/INDEX IF NOT EXISTS` + `ADD COLUMN IF NOT EXISTS` everywhere - Status enum via `text + CHECK` (правильный выбор vs `CREATE TYPE` — нет duplicate_object risk) - FK references resolve in apply order (009→010→011→012→013→014→015→016→017→018) - ON DELETE semantics: CASCADE для child rows (snapshots, reviews, IMV evals), default NO ACTION иначе - GIST на houses.geom ✅ - 7 partial indexes с sensible WHERE clauses - JSONB (не JSON) для всех jsonb полей - timestamptz consistency ✅ - Path trigger `tradein-mvp/**` в deploy-tradein.yml — корректно ловит `tradein-mvp/backend/data/sql/**` - Apply order: `ls -1 ... | sort` (lexicographic 001..018) - Target DB: `psql -d tradein` (отдельная DB, не main gendesign — zero conflict) **Post-merge verify steps**: 1. deploy-tradein.yml triggers on push to main → check GHA runs 2. Apply log: `→ Migration: 009_houses.sql` ... 018 sequential. `NOTICE: relation already exists, skipping` если re-apply, или `CREATE TABLE` для fresh. 3. SSH в tradein postgres: `\dt` → 7 новых tables (houses, sellers, house_reviews, scrape_runs, listings_snapshots, house_placement_history, avito_imv_evaluations) + 14 новых колонок в listings (`\d listings`). **Minor nits** (можно отдельным PR, не блокирующее): - `avito_imv_evaluations.cache_key` не UNIQUE — append-only история acceptable (audit drift Avito IMV), `imv_cache_key_idx (cache_key, fetched_at DESC)` поддерживает latest-by-fetched_at lookup - `houses.rating numeric(2,1)` (009) дублирует `rating_score numeric(4,2)` (010) — post-Stage 2 consider DROP `rating` или generated column - `listings_snapshots` не partitioned — acceptable для pilot (~365k rows/year), revisit при > 10M **Stage 2a unblocked**: spawn backend-engineer для avito.py refactor — separate session.

lekss361 referenced this pull request

2026-05-23 12:59:28 +00:00

fix(tradein): 030 — UNIQUE constraint on avito_imv_evaluations.cache_key (unblocks PR #452) #454