PR I (#595) hookнул matching service в save_listings для new scrapes.
PR J — repeatable script для existing 18,428 listings (avito 9302,
cian 5158, yandex 3704, n1 264) — все они unlinked к houses.
Script:
- backfill_listing_sources.py: stream batches of 500 rows, per-row
SAVEPOINT, idempotent via NOT EXISTS pre-check + ON CONFLICT в
upsert_listing_source.
- Use lot.house_source/house_ext_id if scraper extract'нул (direct
link path), otherwise match_or_create_house() (Tier 0-3 via PostGIS
geo-proximity + address fingerprint).
- upsert_listing_source с method='backfill', confidence=0.9 (lower
чем real-time source_link=1.0 — для forensic distinction).
- Args: --batch-size N (default 500), --limit N (canary), --dry-run.
- No network calls (in-DB matching через PostGIS + address aliases) —
работает даже когда Yandex Geocoder заблокирован.
Tests: 19 pass в test_backfill_listing_sources.py:
- happy path, idempotency (re-run skip), house_source direct linkage,
match failure → still upserts с NULL house_id, dry-run, --limit cap,
cursor advance, CLI entry point.
README.md обновлён с canonical docker exec instruction.
Run post-merge через:
ssh gendesign 'docker exec tradein-backend python -m \\
scripts.backfill_listing_sources --batch-size 500'
Expected ~5-15 min wall-time для 18k rows (per-row Tier 0-3 matching
через advisory lock + 3 PostGIS queries). Final coverage ~80-90% (~10%
remaining без resolvable address/coords).
Closes#582 (matching gap).
Light1YT
e83079f07a