feat(tradein): recurring house-dedup merge (schedule dormant) (#1772) #1933

Merged
bot-backend merged 2 commits from feat/1772-recurring-house-merge into main 2026-06-26 21:39:31 +00:00

2 commits

Author SHA1 Message Date
70ca65b62b test(tradein): fix real-DB merge test schema (#1772, #1933 review)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
The optional real-DB end-to-end test self-skipped without a DB but would
have errored WITH one (the safety fixture never actually ran). Fixed two
schema bugs + one impossible-collision setup, validated against a live
PostGIS 16 container (test now passes end-to-end, not just collected):

- houses_price_dynamics insert used a non-existent `avg_price` column →
  `price_per_sqm` (the real NOT NULL value column, migration 024). The two
  rows share all 5 dim fields with differing house_id → a genuine LIVE
  6-col UNIQUE collision the dedup step must resolve. Also assert the
  surviving row is the keeper's own (price_per_sqm=100000, not 999999).
- listings insert omitted NOT NULL no-default columns → added source_url,
  dedup_hash (also UNIQUE → distinct per row), price_rub (migration 002).
- house_sources UNIQUE(ext_source,ext_id) is GLOBAL, so the original
  same-key-on-both-houses setup violated at INSERT time (before the merge).
  Replaced with distinct loser/keeper source rows + assert the loser's row
  re-points onto the keeper and none dangle on the deleted loser.

Still self-skips in CI (no Postgres) — correct; now valid against a
tunnel/local DB. Full suite 2444 passed / 0 failed; ruff clean.
2026-06-27 02:38:15 +05:00
fa5e5d83f1 feat(tradein): recurring house-dedup merge (schedule dormant) (#1772)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Reuses migration 108's proven collision-safe pattern
(cluster → canonical keeper → re-point children with UNIQUE-collision
handling → delete losers → backfill sources/aliases) as a RECURRING,
idempotent scheduled job — dups recur because the matching pipeline keeps
creating them, so a one-time migration cannot fix the steady-state inflow.

- Service app/services/house_dedup_merge.py: merge_duplicate_houses(db, *,
  dry_run=False). Clusters houses by EXACT address (cadastral 100% NULL →
  address is the real key). Keeper: geom NOT NULL → most listings →
  most-populated → min(id). Handles ALL 11 FK children incl. live 6-col
  houses_price_dynamics UNIQUE (migration 029 replaced 108's stale 3-col).
  One transaction; dry_run computes counts then ROLLBACK; idempotent
  (clean table → empty mapping → 0-row no-op). Audit-logs every merge.
- Scheduler: trigger_house_dedup_merge_run (run_in_executor, sync DB-only) +
  dispatch branch + run_house_dedup_merge lifecycle wrapper.
- Migration 135: seeds scrape_schedules house_dedup_merge enabled=FALSE
  (DORMANT — deploy neutral; enable deliberately after dry-run + manual run),
  weekly window, ON CONFLICT DO NOTHING.
- Tests: static SQL-shape (every FK child + collision handling, migration-133
  guard), fake-db dry_run/idempotent, scheduler wiring, optional real-DB
  end-to-end merge test (self-skips without a DB).
2026-06-27 02:23:22 +05:00