-- 161_backfill_scraped_at_active_recent.sql
-- Issue #2206 — retro-backfill scraped_at for active listings whose scraped_at
-- lagged behind last_seen_at.
--
-- Root cause (fixed in scrapers/base.py + scraper-kit/base.py this same PR):
--   The listings upsert bumped last_seen_at=NOW() on every live re-confirmation
--   (ON CONFLICT DO UPDATE + dedup-drift reconcile UPDATE) but NEVER touched
--   scraped_at. The estimator filters LISTINGS_FRESH_DAYS via `scraped_at >
--   NOW() - INTERVAL '14 days'`, so re-confirmed-but-not-re-inserted listings
--   fell out of the estimator's visible inventory even while genuinely live.
--
-- Impact (verified on prod 2026-07-02, estimator-visible / total is_active):
--   avito:  6353 / 8474
--   yandex: 4561 / 8878
--   cian:    934 / 15759   (only 5.9% visible)
--   Overall only ~45.6% of the active inventory reached valuations.
--
-- Fix going forward: the upsert now sets scraped_at=NOW() alongside
--   last_seen_at=NOW(). This migration retro-corrects the already-stale rows.
--
-- Why last_seen_at is the honest freshness signal HERE (avito/cian/yandex):
--   For these three sources last_seen_at only advances when a real scrape run
--   re-sees the listing (per-lot upsert), so it faithfully marks the last time
--   the ad was genuinely observed live. Copying it into scraped_at restores the
--   true "last scraped" instant for rows that were re-confirmed before the code
--   fix landed.
--
-- Why domklik is DELIBERATELY EXCLUDED:
--   domklik's last_seen_at is advanced every day by an UNtracked bulk-touch
--   UPDATE (all 6296 rows share one identical timestamp; verified prod
--   2026-07-02, see migration 160). Backfilling scraped_at from that poisoned
--   last_seen_at would falsely "refresh" rows that were never actually
--   re-scraped, re-inflating the estimator with dead inventory. domklik
--   freshness is tracked via scraped_at directly (deactivate_stale_domklik,
--   #2204) — nothing to backfill. n1 is a dead source and irrelevant here.
--
-- Idempotent by construction: after the first run every matched row has
--   scraped_at = last_seen_at, so `last_seen_at > scraped_at` matches 0 rows on
--   any re-apply. Non-destructive: scraped_at only moves forward (last_seen_at
--   is >= scraped_at for every matched row), is_active rows only.
--
-- Expected volume ~6-7k rows (fast single UPDATE, indexable via is_active).
--
-- Dependencies: listings table with scraped_at + last_seen_at + is_active + source.

BEGIN;

UPDATE listings
   SET scraped_at = last_seen_at
 WHERE is_active
   AND source IN ('avito', 'cian', 'yandex')
   AND last_seen_at > scraped_at;

COMMIT;
