lekss361/gendesign

Fork 0

feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies #445

Merged

lekss361 merged 1 commit from feat/tradein-cian-migrations into main

2026-05-23 12:19:29 +00:00

lekss361 commented

2026-05-23 12:16:34 +00:00

Owner

Summary

Stage 1 / Wave 1 of CianScraper v1 — schema foundation для 4 Cian sources (SERP/detail/newbuilding/valuation) + cross-source matching.

Migrations

File	Purpose
`019_listings_alter_cian.sql`	ALTER listings — 22 cian-specific cols (windows/wcs/ceiling, repair, views, living_area, bedrooms, balconies/loggias, description_minhash, cadastres, phones, flags)
`020_houses_alter_cian.sql`	ALTER houses — BTI (series_name/entrances/flat_count/is_emergency/heat/gas/overlap) + cian (cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn jsonb + GIN)
`021_management_companies.sql`	NEW — ТСЖ/УК (name, phones[], email, opening_hours, chief_name); adds FK from houses.management_company_id
`022_agents_table.sql`	NEW — agent profiles (ext_agent_id, company_name, is_pro, skills jsonb, account_type)
`023_offer_price_history.sql`	NEW — per-listing price changes (change_time, price_rub, diff_percent)
`024_houses_price_dynamics.sql`	NEW — monthly time-series для newbuildings (realtyValuation chart)
`025_house_reliability_checks.sql`	NEW — наш.дом.рф checks
`026_external_valuations.sql`	NEW — unified Avito IMV + Cian Valuation cache (cache_key, 24h TTL)
`027_cian_session_cookies.sql`	NEW — encrypted cookies via pgcrypto (auth для Cian Valuation Calculator)
`028_matching_tables.sql`	NEW — house_sources, listing_sources, house_address_aliases (3-tier cross-source matching)

Constraints

All BEGIN; / COMMIT;
All CREATE TABLE IF NOT EXISTS, ADD COLUMN IF NOT EXISTS, CREATE INDEX IF NOT EXISTS
pgcrypto extension created in 027 для cookie encryption
020 FK to management_companies deferred to 021 (avoids forward-reference; both independently re-runnable)

Deviations from spec

020 adds management_company_id as plain bigint (no FK); 021 adds FK via ALTER TABLE houses ADD CONSTRAINT. Keeps each file idempotent without cross-file dependencies.
024_houses_price_dynamics uses simpler shape (price_per_sqm + UNIQUE(house_id, month_date, source)) vs vault doc's multi-dimension form. Sufficient for Stage 1 scope; extensible later.
021 management_companies.ext_id typed as bigint per task spec; UNIQUE(ext_source, ext_id, name) для completeness.

Refs

decisions/CianScraper_v1_Implementation_Plan.md Stage 1
decisions/Schema_Cian_SERP_Inventory.md (sec 16, 20.3, 25, 26)
decisions/Cross_Source_Matching_Strategy.md (sec 2)

Next waves (blocked by this merge)

Wave 2: Stage 2 — cian_state_parser.py + ScrapedLot extension
Wave 3: Stage 3 — cian.py SERP refactor
Wave 4 (parallel): cookies/detail/newbuilding workers

Test plan

CI green (deploy.yml auto-applies SQL via psql)
After deploy, verify 10 tables/alters exist in prod DB
No regression in existing 009-018 migrations

## Summary Stage 1 / Wave 1 of CianScraper v1 — schema foundation для 4 Cian sources (SERP/detail/newbuilding/valuation) + cross-source matching. ## Migrations | File | Purpose | |---|---| | `019_listings_alter_cian.sql` | ALTER listings — 22 cian-specific cols (windows/wcs/ceiling, repair, views, living_area, bedrooms, balconies/loggias, description_minhash, cadastres, phones, flags) | | `020_houses_alter_cian.sql` | ALTER houses — BTI (series_name/entrances/flat_count/is_emergency/heat/gas/overlap) + cian (cian_internal_house_id, management_company_id, transport_accessibility_rate, advantages/banks/builders/houses_by_turn jsonb + GIN) | | `021_management_companies.sql` | NEW — ТСЖ/УК (name, phones[], email, opening_hours, chief_name); adds FK from houses.management_company_id | | `022_agents_table.sql` | NEW — agent profiles (ext_agent_id, company_name, is_pro, skills jsonb, account_type) | | `023_offer_price_history.sql` | NEW — per-listing price changes (change_time, price_rub, diff_percent) | | `024_houses_price_dynamics.sql` | NEW — monthly time-series для newbuildings (realtyValuation chart) | | `025_house_reliability_checks.sql` | NEW — наш.дом.рф checks | | `026_external_valuations.sql` | NEW — unified Avito IMV + Cian Valuation cache (cache_key, 24h TTL) | | `027_cian_session_cookies.sql` | NEW — encrypted cookies via pgcrypto (auth для Cian Valuation Calculator) | | `028_matching_tables.sql` | NEW — house_sources, listing_sources, house_address_aliases (3-tier cross-source matching) | ## Constraints - All `BEGIN;` / `COMMIT;` - All `CREATE TABLE IF NOT EXISTS`, `ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` - pgcrypto extension created in 027 для cookie encryption - 020 FK to management_companies deferred to 021 (avoids forward-reference; both independently re-runnable) ## Deviations from spec 1. `020` adds `management_company_id` as plain `bigint` (no FK); `021` adds FK via `ALTER TABLE houses ADD CONSTRAINT`. Keeps each file idempotent without cross-file dependencies. 2. `024_houses_price_dynamics` uses simpler shape (price_per_sqm + UNIQUE(house_id, month_date, source)) vs vault doc's multi-dimension form. Sufficient for Stage 1 scope; extensible later. 3. `021 management_companies.ext_id` typed as `bigint` per task spec; UNIQUE(ext_source, ext_id, name) для completeness. ## Refs - `decisions/CianScraper_v1_Implementation_Plan.md` Stage 1 - `decisions/Schema_Cian_SERP_Inventory.md` (sec 16, 20.3, 25, 26) - `decisions/Cross_Source_Matching_Strategy.md` (sec 2) ## Next waves (blocked by this merge) - Wave 2: Stage 2 — `cian_state_parser.py` + ScrapedLot extension - Wave 3: Stage 3 — `cian.py` SERP refactor - Wave 4 (parallel): cookies/detail/newbuilding workers ## Test plan - [ ] CI green (deploy.yml auto-applies SQL via psql) - [ ] After deploy, verify 10 tables/alters exist in prod DB - [ ] No regression in existing 009-018 migrations

lekss361 added 1 commit 2026-05-23 12:16:34 +00:00

feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 1ad2565af1

- 019: ALTER listings (windows_view_type, wcs counts, ceiling_height,
       repair_type, views, living_area, bedrooms, balconies, loggias,
       description_minhash, cadastral_number, phones, bargain_allowed,
       sale_type, house_source/ext_id, metro_stations)
- 020: ALTER houses (BTI: series_name, entrances, flat_count, is_emergency,
       heat/gas/overlap types; Cian: cian_internal_house_id,
       management_company_id, transport_accessibility_rate,
       advantages/banks/builders/houses_by_turn jsonb + GIN indexes)
- 021: CREATE management_companies + FK to houses.management_company_id
- 022: CREATE agents table + agent_id_fk on listings
- 023: CREATE offer_price_history (per-listing price change timeline)
- 024: CREATE houses_price_dynamics (monthly price/sqm series)
- 025: CREATE house_reliability_checks (наш.дом.рф checks)
- 026: CREATE external_valuations (unified Avito IMV + Cian Valuation cache)
- 027: CREATE EXTENSION pgcrypto + cian_session_cookies (encrypted)
- 028: CREATE house_sources, listing_sources, house_address_aliases
       + address_fingerprint/canonical/merged_into alters

Refs: decisions/CianScraper_v1_Implementation_Plan.md Stage 1
      decisions/Schema_Cian_SERP_Inventory.md sec 16, 20.3, 26
      decisions/Cross_Source_Matching_Strategy.md sec 2

lekss361 merged commit 23e6cbc773 into main

2026-05-23 12:19:29 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 12:19:29 +00:00

feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies (#445)

lekss361 commented

2026-05-23 12:20:14 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Deep review summary

Status: APPROVE
Files reviewed: 10 (all P0 — SQL DDL in tradein-mvp/backend/data/sql/)
Lines: +501 / -0
Merge commit: 23e6cbc

Findings — none blocking

Idempotency: All 10 files wrapped in BEGIN/COMMIT, use CREATE TABLE IF NOT EXISTS / ADD COLUMN IF NOT EXISTS / CREATE INDEX IF NOT EXISTS / CREATE EXTENSION IF NOT EXISTS. Re-running any single file is safe. Re-running ALL files in order (alphabetical sort via ls -1 ... | sort in deploy.yml:234) is safe.

Ordering: Auto-apply via deploy.yml uses sort → 019→020→021→…→028 matches dependency chain. 020 adds management_company_id as plain bigint; 021 creates management_companies then ALTER TABLE houses ADD CONSTRAINT … FK. Drop-create pattern (DROP CONSTRAINT IF EXISTS before ADD CONSTRAINT) makes 021 re-runnable.

Security: pgcrypto extension created in 027. cian_session_cookies.cookies_encrypted is bytea (encrypted blob via pgp_sym_encrypt). Encryption key in .env.runtime (per file header). No hardcoded secrets in SQL.

Conventions: All files follow NN_topic.sql naming, all use timestamptz (not bare timestamp), numeric precision appropriate (numeric(7,2) for area, numeric(3,2) for ceiling, bigint for prices, real for confidence 0-1).

GIN indexes: Correctly placed on jsonb columns where querying expected (advantages, banks, builders, metro_stations). BTree on scalar lookups. Partial indexes (WHERE col IS NOT NULL) reduce index size on sparse cols.

Deviation verification (3 items from PR body)

020 plain bigint + 021 FK ALTER: Verified independently re-runnable. 020 adds management_company_id bigint without FK; 021 creates management_companies, then drops+adds FK constraint. No chicken-and-egg.
024 simpler shape: Confirmed — (house_id, month_date, source) UNIQUE is sufficient for Stage 1. Spec form (room_count, prices_type, period, month_date) preserves Cian realtyValuation multi-dimensional chart. Follow-up needed before Stage 6 newbuilding scraper — add migration 029 extending dimensions.
021 ext_id as bigint: Consistent with cian_internal_house_id bigint from Schema_Cian_SERP_Inventory sec 16.5. Cian internally uses numeric IDs.

Forward-compat concerns (non-blocking, follow-up tracked)

Issue	Impact	Severity
`house_sources`/`listing_sources` missing per-source `price_rub`/`area_m2`/`raw_payload`/`last_seen_at` (spec sec 2.2)	Stage 8 matching service needs these for conflict resolution + price cross-validation + stale-data detection	medium — needs migration 029
`external_valuations` simpler form (no sale/rent split, no chart_change_pct/dir)	Stage 7 Cian Valuation scraper can't save full sale+rent estimates separately	medium — needs migration 029
`management_companies` UNIQUE (`ext_source, name, ext_id`) allows duplicates when ext_id IS NULL	dup ТСЖ rows possible without Cian numeric ID	low
`houses.cadastral_number` referenced by Tier-0 matching algorithm but not added by any migration	Stage 8 Tier-0 house matching falls back to Tier 1 until backfilled from `cad_buildings.kadastr_num`	low
`cadastral_number` in `listings` (019) semantically overlaps with existing `kadastr_num` (002)	two columns for same data; scrapers must write to both or migration to merge	low — decide in Stage 8
`house_reliability_checks` no UNIQUE (`house_id, source`) vs spec	history rows accumulate per house; OK if intentional	low

Anti-regression cross-check

011 already has views_total/views_today/metro_stations/house_source/house_ext_id/sale_type on listings — 019 re-adds via ADD COLUMN IF NOT EXISTS (no error, idempotent). Types match (jsonb, int, text). Cosmetic duplication only.
houses.UNIQUE (source, ext_house_id) from 009 still present — Cross_Source_Matching_Strategy sec 2.1 says it should be dropped to enable multi-source via house_sources. New design works without dropping it since house_sources is the link layer; existing constraint just enforces 1 row per (source, ext_id) in master table. Cleanup deferred to Stage 8.

Follow-up issues recommended

Migration 029 to extend house_sources/listing_sources with price_rub/area_m2/raw_payload/last_seen_at/ext_url per spec sec 2.2 — blocks Stage 8 matching service
Migration 029 to extend external_valuations with sale/rent split + chart_change_pct/dir + filters_hash + external_house_id — blocks Stage 7 Cian Valuation scraper
Migration 029 to extend houses_price_dynamics with room_count/prices_type/period dimensions — blocks Stage 6 newbuilding scraper full chart save
Decision: merge listings.cadastral_number (019) with listings.kadastr_num (002) — pick one canonical column name

All follow-ups are independent of this PR and trackable separately.

Merged via deep-code-reviewer — verdict APPROVE. ## Deep review summary **Status**: APPROVE **Files reviewed**: 10 (all P0 — SQL DDL in `tradein-mvp/backend/data/sql/`) **Lines**: +501 / -0 **Merge commit**: 23e6cbc ### Findings — none blocking **Idempotency**: All 10 files wrapped in BEGIN/COMMIT, use CREATE TABLE IF NOT EXISTS / ADD COLUMN IF NOT EXISTS / CREATE INDEX IF NOT EXISTS / CREATE EXTENSION IF NOT EXISTS. Re-running any single file is safe. Re-running ALL files in order (alphabetical sort via `ls -1 ... | sort` in `deploy.yml:234`) is safe. **Ordering**: Auto-apply via deploy.yml uses sort → 019→020→021→…→028 matches dependency chain. 020 adds `management_company_id` as plain bigint; 021 creates `management_companies` then `ALTER TABLE houses ADD CONSTRAINT … FK`. Drop-create pattern (`DROP CONSTRAINT IF EXISTS` before `ADD CONSTRAINT`) makes 021 re-runnable. **Security**: pgcrypto extension created in 027. `cian_session_cookies.cookies_encrypted` is `bytea` (encrypted blob via `pgp_sym_encrypt`). Encryption key in `.env.runtime` (per file header). No hardcoded secrets in SQL. **Conventions**: All files follow `NN_topic.sql` naming, all use `timestamptz` (not bare timestamp), numeric precision appropriate (`numeric(7,2)` for area, `numeric(3,2)` for ceiling, `bigint` for prices, `real` for confidence 0-1). **GIN indexes**: Correctly placed on jsonb columns where querying expected (`advantages`, `banks`, `builders`, `metro_stations`). BTree on scalar lookups. Partial indexes (`WHERE col IS NOT NULL`) reduce index size on sparse cols. ### Deviation verification (3 items from PR body) 1. **020 plain bigint + 021 FK ALTER**: Verified independently re-runnable. 020 adds `management_company_id bigint` without FK; 021 creates `management_companies`, then drops+adds FK constraint. No chicken-and-egg. 2. **024 simpler shape**: Confirmed — `(house_id, month_date, source)` UNIQUE is sufficient for Stage 1. Spec form (`room_count, prices_type, period, month_date`) preserves Cian realtyValuation multi-dimensional chart. **Follow-up needed before Stage 6 newbuilding scraper** — add migration 029 extending dimensions. 3. **021 ext_id as bigint**: Consistent with `cian_internal_house_id bigint` from Schema_Cian_SERP_Inventory sec 16.5. Cian internally uses numeric IDs. ### Forward-compat concerns (non-blocking, follow-up tracked) | Issue | Impact | Severity | |---|---|---| | `house_sources`/`listing_sources` missing per-source `price_rub`/`area_m2`/`raw_payload`/`last_seen_at` (spec sec 2.2) | Stage 8 matching service needs these for conflict resolution + price cross-validation + stale-data detection | medium — needs migration 029 | | `external_valuations` simpler form (no sale/rent split, no chart_change_pct/dir) | Stage 7 Cian Valuation scraper can't save full sale+rent estimates separately | medium — needs migration 029 | | `management_companies` UNIQUE (`ext_source, name, ext_id`) allows duplicates when ext_id IS NULL | dup ТСЖ rows possible without Cian numeric ID | low | | `houses.cadastral_number` referenced by Tier-0 matching algorithm but not added by any migration | Stage 8 Tier-0 house matching falls back to Tier 1 until backfilled from `cad_buildings.kadastr_num` | low | | `cadastral_number` in `listings` (019) semantically overlaps with existing `kadastr_num` (002) | two columns for same data; scrapers must write to both or migration to merge | low — decide in Stage 8 | | `house_reliability_checks` no UNIQUE (`house_id, source`) vs spec | history rows accumulate per house; OK if intentional | low | ### Anti-regression cross-check - 011 already has `views_total/views_today/metro_stations/house_source/house_ext_id/sale_type` on listings — 019 re-adds via `ADD COLUMN IF NOT EXISTS` (no error, idempotent). Types match (jsonb, int, text). Cosmetic duplication only. - `houses.UNIQUE (source, ext_house_id)` from 009 still present — Cross_Source_Matching_Strategy sec 2.1 says it should be dropped to enable multi-source via `house_sources`. New design works without dropping it since `house_sources` is the link layer; existing constraint just enforces 1 row per (source, ext_id) in master table. Cleanup deferred to Stage 8. ### Follow-up issues recommended 1. Migration 029 to extend `house_sources`/`listing_sources` with `price_rub`/`area_m2`/`raw_payload`/`last_seen_at`/`ext_url` per spec sec 2.2 — **blocks Stage 8 matching service** 2. Migration 029 to extend `external_valuations` with sale/rent split + chart_change_pct/dir + filters_hash + external_house_id — **blocks Stage 7 Cian Valuation scraper** 3. Migration 029 to extend `houses_price_dynamics` with room_count/prices_type/period dimensions — **blocks Stage 6 newbuilding scraper full chart save** 4. Decision: merge `listings.cadastral_number` (019) with `listings.kadastr_num` (002) — pick one canonical column name All follow-ups are independent of this PR and trackable separately.

lekss361 referenced this pull request from a commit

2026-05-23 12:26:48 +00:00

feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up)

lekss361 referenced this pull request

2026-05-23 12:29:13 +00:00

feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields #447

lekss361 referenced this pull request

2026-05-23 12:29:38 +00:00

feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) #448

lekss361 referenced this pull request from a commit

2026-05-23 12:35:05 +00:00

feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) (#448)