feat(tradein): per-source listing price history — schema + daily snapshot (#570) #646

Merged

Light1YT merged 1 commit from feat/570-per-source-history into main

2026-05-29 12:18:05 +00:00

Light1YT commented

2026-05-29 12:17:49 +00:00

Owner

Summary

Closes #570. Per-source listing price history — listing_sources only kept the latest snapshot, no history; scrapers are dormant so organic accumulation cant wait.

Migration 079: listing_source_snapshots (daily per-source, plain table — partitioning deferred, ~540k rows/mo is fine unpartitioned) + listing_source_events (change-log) + offer_price_history.listing_source_id backlink + v_listing_source_price_on_date view. Idempotent.

snapshot_listing_sources() (app/tasks): daily snapshot of the existing 18,355 listing_sources + set-based CTE diff of todays price vs prior snapshot → price_change events. Wired into the in-app scheduler (listing_source_snapshot, enabled=true — pure internal DB, no external calls/anti-bot, window 01:00-02:00 UTC). History accumulates from tonight, independent of dormant scrapers.

code-reviewer ✅ no blockers (FK cascades sound; event-diff guards correct; same-day double-fire impossible via _claim_run+has_running_run → no dup events; psycopg-v3 clean).

Test plan

23 tests (18 new + 5 scheduler), ruff, py_compile clean
Prod dry-run (ROLLBACK): migration applies clean (2 tables + view + column + schedule enabled=true); snapshot INSERT → 18,355 rows
Post-merge: 079 applied under _schema_migrations gate; first 01:00 UTC run populates snapshots; scheduler healthy

Closes #570

## Summary Closes #570. Per-source listing price **history** — `listing_sources` only kept the latest snapshot, no history; scrapers are dormant so organic accumulation cant wait. **Migration 079**: `listing_source_snapshots` (daily per-source, plain table — partitioning deferred, ~540k rows/mo is fine unpartitioned) + `listing_source_events` (change-log) + `offer_price_history.listing_source_id` backlink + `v_listing_source_price_on_date` view. Idempotent. **`snapshot_listing_sources()`** (app/tasks): daily snapshot of the existing **18,355** listing_sources + set-based CTE diff of todays price vs prior snapshot → `price_change` events. Wired into the in-app scheduler (`listing_source_snapshot`, **enabled=true** — pure internal DB, no external calls/anti-bot, window 01:00-02:00 UTC). **History accumulates from tonight, independent of dormant scrapers.** code-reviewer ✅ no blockers (FK cascades sound; event-diff guards correct; same-day double-fire impossible via `_claim_run`+`has_running_run` → no dup events; psycopg-v3 clean). ## Test plan - [x] 23 tests (18 new + 5 scheduler), ruff, py_compile clean - [x] **Prod dry-run (ROLLBACK)**: migration applies clean (2 tables + view + column + schedule enabled=true); snapshot INSERT → **18,355 rows** - [ ] Post-merge: 079 applied under _schema_migrations gate; first 01:00 UTC run populates snapshots; scheduler healthy Closes #570

Light1YT added 1 commit 2026-05-29 12:17:50 +00:00

feat(tradein): per-source listing price history — schema + daily snapshot (#570 ) 1e44ef09f4

Migration 079 adds listing_source_snapshots (daily per-source snapshot, plain
table — partitioning deferred), listing_source_events (change-log), an
offer_price_history.listing_source_id backlink, and v_listing_source_price_on_date.

snapshot_listing_sources() (app/tasks) writes a daily snapshot of the existing
18,355 listing_sources rows and diffs today's price vs the prior snapshot into
price_change events — both set-based SQL (CTE), no row loop. Wired into the
in-app scheduler (source='listing_source_snapshot', enabled=true: pure internal
DB, no external calls, window 01:00-02:00 UTC before the backfill jobs). History
accumulates immediately, independent of the dormant scrapers.

Prod dry-run (rollback): migration applies clean; snapshot inserts 18,355 rows.
Same-day double-fire impossible (_claim_run advances next_run_at + has_running_run
guard) so no duplicate price_change events.

Closes #570