Light1YT e83079f07a feat(tradein): backfill 18k existing listings → listing_sources (PR J) ...

PR I (#595) hookнул matching service в save_listings для new scrapes.
PR J — repeatable script для existing 18,428 listings (avito 9302,
cian 5158, yandex 3704, n1 264) — все они unlinked к houses.

Script:
- backfill_listing_sources.py: stream batches of 500 rows, per-row
  SAVEPOINT, idempotent via NOT EXISTS pre-check + ON CONFLICT в
  upsert_listing_source.
- Use lot.house_source/house_ext_id if scraper extract'нул (direct
  link path), otherwise match_or_create_house() (Tier 0-3 via PostGIS
  geo-proximity + address fingerprint).
- upsert_listing_source с method='backfill', confidence=0.9 (lower
  чем real-time source_link=1.0 — для forensic distinction).
- Args: --batch-size N (default 500), --limit N (canary), --dry-run.
- No network calls (in-DB matching через PostGIS + address aliases) —
  работает даже когда Yandex Geocoder заблокирован.

Tests: 19 pass в test_backfill_listing_sources.py:
- happy path, idempotency (re-run skip), house_source direct linkage,
  match failure → still upserts с NULL house_id, dry-run, --limit cap,
  cursor advance, CLI entry point.

README.md обновлён с canonical docker exec instruction.

Run post-merge через:
  ssh gendesign 'docker exec tradein-backend python -m \\
    scripts.backfill_listing_sources --batch-size 500'

Expected ~5-15 min wall-time для 18k rows (per-row Tier 0-3 matching
через advisory lock + 3 PostGIS queries). Final coverage ~80-90% (~10%
remaining без resolvable address/coords).

Closes #582 (matching gap).

2026-05-27 15:29:15 +05:00

0 lines

Python

Raw Blame History