History

lekss361 1d9a8e4ac4 fix(tradein-sweep): paginate until total_objects reached, dedup pages Previous break condition (items < 13) stopped after page 1 even when Yandex's header declared total_objects=36 or 259. Live observation: Авиационная 63к2 declared 36 objects, sweep only saved 14 from page=1 (missed 22). New stop conditions: - Cumulative >= total_objects from house meta header (when present) - Page yields 0 new items (pure duplicates after dedup → end of feed) - max_pages reached Also dedup items across pages by (publish_date, area_m2, floor, start_price) to handle Yandex's occasional repeat at page boundaries.		2026-05-24 18:02:38 +03:00
..
backfill-house-imv.py	feat(tradein): Phase C — Avito IMV per-house backfill (SQL + script) (#534 )	2026-05-24 14:50:31 +00:00
backfill_001_address_fingerprints.py	feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources (#475 )	2026-05-23 14:29:39 +00:00
backfill_002_house_sources.sql	feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources (#475 )	2026-05-23 14:29:39 +00:00
backfill_003_listing_sources.sql	feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources (#475 )	2026-05-23 14:29:39 +00:00
probe-confidence-distribution.py	feat(tradein-estimator): unique-address-based confidence threshold (#513 )	2026-05-24 11:52:00 +00:00
probe-imv-by-address.py	feat(tradein-imv): retry with cleaned address + categorize errors (#505 )	2026-05-24 11:04:24 +00:00
README.md	feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources (#475 )	2026-05-23 14:29:39 +00:00
sweep-yandex-valuation-ekb.py	fix(tradein-sweep): paginate until total_objects reached, dedup pages	2026-05-24 18:02:38 +03:00
upload-cian-cookies.sh	fix(tradein): убрать require_admin полностью — Caddy basic_auth достаточен (#474 )	2026-05-23 14:21:01 +00:00

README.md

tradein-mvp scripts -- backfill

One-shot manual operator scripts. NOT auto-applied by deploy.yml (which only runs tradein-mvp/backend/data/sql/NN_*.sql).

Order

Run in numbered sequence after PR #470 (matching module) merged:

cd tradein-mvp/backend
export DATABASE_URL="postgresql+psycopg://user:pass@host:5432/tradein"

# 001 -- backfill houses.address_fingerprint (Python, idempotent)
uv run python ../scripts/backfill_001_address_fingerprints.py

# 002 -- backfill house_sources link rows (SQL, idempotent)
psql "${DATABASE_URL/postgresql+psycopg/postgresql}" -f ../scripts/backfill_002_house_sources.sql

# 003 -- backfill listing_sources link rows (SQL, idempotent)
psql "${DATABASE_URL/postgresql+psycopg/postgresql}" -f ../scripts/backfill_003_listing_sources.sql

All scripts idempotent -- re-running is safe (ON CONFLICT DO NOTHING on the SQL; row-level UPDATE WHERE NULL on the Python).

Other scripts

upload-cian-cookies.sh -- one-shot Cian session cookie upload (see inline docs)