feat(tradein): backfill 18k existing listings (PR J) #596

Merged

Light1YT merged 1 commit from feat/tradein-backfill-listing-sources into main

2026-05-27 10:29:41 +00:00

Light1YT commented

2026-05-27 10:29:38 +00:00

Owner

Summary

PR I (#595) hookнул matching в save_listings() для NEW scrapes. PR J — repeatable script для EXISTING 18,428 listings (avito 9302, cian 5158, yandex 3704, n1 264) которые сейчас unlinked к houses (junction listing_sources пустая).

Implementation

scripts/backfill_listing_sources.py:

Stream batches 500 rows, per-row SAVEPOINT
Idempotent: NOT EXISTS pre-check + ON CONFLICT в upsert_listing_source
Direct path: use lot.house_source/house_ext_id если есть (Avito Houses Catalog, Cian newbuilding)
Fallback: match_or_create_house() Tier 0-3 (advisory_xact_lock + PostGIS + address_aliases)
upsert_listing_source method='backfill', confidence=0.9 (lower чем real-time 1.0 — forensic distinction)
Args: --batch-size N, --limit N (canary), --dry-run
No network calls — работает даже когда Yandex Geocoder заблокирован

Files (+1168 LOC)

scripts/backfill_listing_sources.py (new, 615 LOC)
tests/scripts/test_backfill_listing_sources.py (new, 476 LOC, 19 tests)
tests/scripts/__init__.py (new)
scripts/README.md (+77 LOC, canonical docker exec instructions)

Tests

19/19 pass:

Happy path, idempotency, house_source direct, match failure path, dry-run, --limit cap, cursor advance, CLI entrypoint
ruff clean

Run post-merge

ssh gendesign 'docker exec tradein-backend python -m \
  scripts.backfill_listing_sources --batch-size 500'

Expected ~5-15 min wall-time. Expected coverage ~80-90% (10% — listings без resolvable address/coords).

Verification

SELECT COUNT(*) FROM listing_sources;  -- ожидаем ~14000+
SELECT method, COUNT(*) FROM listing_sources GROUP BY method;
  -- backfill: ~14000  (PR J)
  -- source_link: ~0  (PR I — пока scrapers не запустились)

Closes #582 (matching gap full closure).

## Summary PR I (#595) hookнул matching в `save_listings()` для **NEW** scrapes. PR J — repeatable script для **EXISTING** 18,428 listings (avito 9302, cian 5158, yandex 3704, n1 264) которые сейчас unlinked к houses (junction `listing_sources` пустая). ## Implementation `scripts/backfill_listing_sources.py`: - Stream batches 500 rows, per-row SAVEPOINT - Idempotent: NOT EXISTS pre-check + ON CONFLICT в upsert_listing_source - Direct path: use `lot.house_source/house_ext_id` если есть (Avito Houses Catalog, Cian newbuilding) - Fallback: `match_or_create_house()` Tier 0-3 (advisory_xact_lock + PostGIS + address_aliases) - `upsert_listing_source` method='backfill', confidence=0.9 (lower чем real-time 1.0 — forensic distinction) - Args: `--batch-size N`, `--limit N` (canary), `--dry-run` - **No network calls** — работает даже когда Yandex Geocoder заблокирован ## Files (+1168 LOC) - `scripts/backfill_listing_sources.py` (new, 615 LOC) - `tests/scripts/test_backfill_listing_sources.py` (new, 476 LOC, 19 tests) - `tests/scripts/__init__.py` (new) - `scripts/README.md` (+77 LOC, canonical docker exec instructions) ## Tests 19/19 pass: - Happy path, idempotency, house_source direct, match failure path, dry-run, --limit cap, cursor advance, CLI entrypoint - ruff clean ## Run post-merge ```bash ssh gendesign 'docker exec tradein-backend python -m \ scripts.backfill_listing_sources --batch-size 500' ``` Expected ~5-15 min wall-time. Expected coverage ~80-90% (10% — listings без resolvable address/coords). ## Verification ```sql SELECT COUNT(*) FROM listing_sources; -- ожидаем ~14000+ SELECT method, COUNT(*) FROM listing_sources GROUP BY method; -- backfill: ~14000 (PR J) -- source_link: ~0 (PR I — пока scrapers не запустились) ``` Closes #582 (matching gap full closure).

Light1YT added 1 commit 2026-05-27 10:29:39 +00:00

feat(tradein): backfill 18k existing listings → listing_sources (PR J) e83079f07a

PR I (#595) hookнул matching service в save_listings для new scrapes.
PR J — repeatable script для existing 18,428 listings (avito 9302,
cian 5158, yandex 3704, n1 264) — все они unlinked к houses.

Script:
- backfill_listing_sources.py: stream batches of 500 rows, per-row
  SAVEPOINT, idempotent via NOT EXISTS pre-check + ON CONFLICT в
  upsert_listing_source.
- Use lot.house_source/house_ext_id if scraper extract'нул (direct
  link path), otherwise match_or_create_house() (Tier 0-3 via PostGIS
  geo-proximity + address fingerprint).
- upsert_listing_source с method='backfill', confidence=0.9 (lower
  чем real-time source_link=1.0 — для forensic distinction).
- Args: --batch-size N (default 500), --limit N (canary), --dry-run.
- No network calls (in-DB matching через PostGIS + address aliases) —
  работает даже когда Yandex Geocoder заблокирован.

Tests: 19 pass в test_backfill_listing_sources.py:
- happy path, idempotency (re-run skip), house_source direct linkage,
  match failure → still upserts с NULL house_id, dry-run, --limit cap,
  cursor advance, CLI entry point.

README.md обновлён с canonical docker exec instruction.

Run post-merge через:
  ssh gendesign 'docker exec tradein-backend python -m \\
    scripts.backfill_listing_sources --batch-size 500'

Expected ~5-15 min wall-time для 18k rows (per-row Tier 0-3 matching
через advisory lock + 3 PostGIS queries). Final coverage ~80-90% (~10%
remaining без resolvable address/coords).

Closes #582 (matching gap).