feat(listings): card_hash column + snapshot-dedup on unchanged cards #1767

Merged

lekss361 merged 1 commit from feat/listings-card-hash into main

2026-06-18 19:03:28 +00:00

lekss361 commented

2026-06-18 19:02:53 +00:00

Owner

What

Adds listings.card_hash (migration 128) — a SHA-256 of the volatile SERP-card fields (price_rub, area_m2, floor, total_floors, rooms, address, photo-count), computed and stored on every scrape. Uses it to skip the redundant per-day listings_snapshots write when a listing's card is unchanged and a snapshot for today already exists.

Why / scope honesty

This does not skip the listings UPDATE itself — last_seen_at/is_active must be bumped on every sighting (10-day delisting TTL), and any row touch rewrites the tuple under MVCC, so there's no perf win there. The genuine win is dropping the redundant daily snapshot write. The larger purpose is laying the card_hash foundation for C2 (gating periodic detail-refresh on card change).

last_seen_at/is_active and the inserted/updated counters are unchanged. Snapshot is still written on: fresh insert, changed card, or unchanged-card-but-no-snapshot-today.

Implementation

ScrapedLot.compute_card_hash() — deterministic, fixed field order, normalized address, no dict-order dependence.
save_listings — card_hash added to INSERT + ON CONFLICT DO UPDATE. A lightweight SELECT card_hash WHERE dedup_hash=:dedup pre-read captures the prior hash; snapshot skipped only when prior == new AND a today-row exists.

Tests

tests/test_base_save_listings.py (+~311): hash determinism + sensitivity (price/area/floor/total_floors/rooms/address/photo-count), address normalization, stability when only timestamps/detail fields change; save_listings stores card_hash on insert+update, pre-read occurs, snapshot skipped when unchanged+today-exists, written on change/fresh/no-today-snapshot. 43 passed. Ruff clean. (Pre-existing B011/psycopg2 issues in unrelated modules noted, not from this change.)

## What Adds `listings.card_hash` (migration `128`) — a SHA-256 of the volatile SERP-card fields (`price_rub, area_m2, floor, total_floors, rooms, address, photo-count`), computed and stored on every scrape. Uses it to **skip the redundant per-day `listings_snapshots` write** when a listing's card is unchanged and a snapshot for today already exists. ## Why / scope honesty This does **not** skip the `listings` UPDATE itself — `last_seen_at`/`is_active` must be bumped on every sighting (10-day delisting TTL), and any row touch rewrites the tuple under MVCC, so there's no perf win there. The genuine win is dropping the redundant daily snapshot write. The larger purpose is laying the `card_hash` foundation for **C2** (gating periodic detail-refresh on card change). `last_seen_at`/`is_active` and the `inserted`/`updated` counters are unchanged. Snapshot is still written on: fresh insert, changed card, or unchanged-card-but-no-snapshot-today. ## Implementation - `ScrapedLot.compute_card_hash()` — deterministic, fixed field order, normalized address, no dict-order dependence. - `save_listings` — `card_hash` added to INSERT + `ON CONFLICT DO UPDATE`. A lightweight `SELECT card_hash WHERE dedup_hash=:dedup` pre-read captures the prior hash; snapshot skipped only when `prior == new` AND a today-row exists. ## Tests `tests/test_base_save_listings.py` (+~311): hash determinism + sensitivity (price/area/floor/total_floors/rooms/address/photo-count), address normalization, stability when only timestamps/detail fields change; save_listings stores card_hash on insert+update, pre-read occurs, snapshot skipped when unchanged+today-exists, written on change/fresh/no-today-snapshot. **43 passed.** Ruff clean. (Pre-existing `B011`/`psycopg2` issues in unrelated modules noted, not from this change.)

lekss361 added 1 commit 2026-06-18 19:02:54 +00:00

feat(listings): card_hash column + snapshot-dedup on unchanged cards

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

8436532bf4

Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and
stored on every scrape. Skips the redundant per-day listings_snapshots write
when a listing's card is unchanged. listings.last_seen_at/is_active still
bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh
gating (C2).