fix(scrapers): write resolved house_id to listings.house_id_fk in realtime link hook #1727

Merged

lekss361 merged 1 commit from fix/house-id-fk-realtime-link into main

2026-06-17 20:16:28 +00:00

lekss361 commented

2026-06-17 20:09:56 +00:00

Owner

Summary

Active listings barely link to master houses (avito 0/444, cian 7%) because the realtime hook resolves a house but never writes the FK to the listing.

Root cause

base.py::_link_listing_to_house calls match_or_create_house() (full Tier 0-3) for every saved listing, but writes the resolved house_id ONLY into listing_sources.source_data JSON — there is no UPDATE listings SET house_id_fk, and the listings INSERT/UPSERT never sets it either. So house_id_fk is populated only by unscheduled batch jobs (063_*.sql, scripts/backfill_listing_sources.py). Every new/re-scraped active row stays NULL. (Yandex looked ~88% linked only because past manual backfills caught its ext_id rows.)

Fix (base.py only)

After match_or_create_house() returns a non-None house_id, inside the existing per-listing SAVEPOINT:

UPDATE listings SET house_id_fk = CAST(:hid AS bigint)
 WHERE id = CAST(:lid AS bigint) AND house_id_fk IS DISTINCT FROM CAST(:hid AS bigint)

listing_id already passed into the function; mirrors backfill_listing_sources.py. IS DISTINCT FROM re-links stale rows + skips no-op writes. Matcher/JSON/Tier logic untouched. psycopg v3 CAST(:x AS bigint).

Now every scrape (avito/cian/yandex) links via the matcher's fingerprint+geo tiers (avito 93% / cian ~100% geo coverage), not just ext_id.

Test plan

pytest -k "save_listings or base or matching or link or house" → 317 passed; ruff clean
FK UPDATE runs with resolved id; no UPDATE when no house; no-op guard
Post-deploy: active avito/cian house_id_fk climbs after a sweep; + one-time python -m scripts.backfill_listing_sources (drains existing NULL backlog)

Refs #1188

## Summary Active listings barely link to master `houses` (avito 0/444, cian 7%) because the realtime hook resolves a house but never writes the FK to the listing. ## Root cause `base.py::_link_listing_to_house` calls `match_or_create_house()` (full Tier 0-3) for every saved listing, but writes the resolved `house_id` ONLY into `listing_sources.source_data` JSON — there is **no `UPDATE listings SET house_id_fk`**, and the listings INSERT/UPSERT never sets it either. So `house_id_fk` is populated only by unscheduled batch jobs (`063_*.sql`, `scripts/backfill_listing_sources.py`). Every new/re-scraped active row stays NULL. (Yandex looked ~88% linked only because past manual backfills caught its ext_id rows.) ## Fix (base.py only) After `match_or_create_house()` returns a non-None `house_id`, inside the existing per-listing SAVEPOINT: ```sql UPDATE listings SET house_id_fk = CAST(:hid AS bigint) WHERE id = CAST(:lid AS bigint) AND house_id_fk IS DISTINCT FROM CAST(:hid AS bigint) ``` `listing_id` already passed into the function; mirrors `backfill_listing_sources.py`. `IS DISTINCT FROM` re-links stale rows + skips no-op writes. Matcher/JSON/Tier logic untouched. psycopg v3 `CAST(:x AS bigint)`. Now every scrape (avito/cian/yandex) links via the matcher's fingerprint+geo tiers (avito 93% / cian ~100% geo coverage), not just ext_id. ## Test plan - [x] `pytest -k "save_listings or base or matching or link or house"` → 317 passed; ruff clean - [x] FK UPDATE runs with resolved id; no UPDATE when no house; no-op guard - [ ] Post-deploy: active avito/cian `house_id_fk` climbs after a sweep; + one-time `python -m scripts.backfill_listing_sources` (drains existing NULL backlog) Refs #1188

lekss361 added 1 commit 2026-06-17 20:09:57 +00:00