feat(cadastral): geo-nearest building matcher via local cad mirror #1750

Merged

lekss361 merged 3 commits from feat/cadastral-geo-match into main

2026-06-18 08:44:25 +00:00

lekss361 commented

2026-06-18 07:28:47 +00:00

Owner

What & why

listings.building_cadastral_number / cadastral_number are 0% filled (columns exist, empty), while listing geo is ~93–100% present. The cadastral building registry (gendesign_cad_buildings, ~36.7k EKB rows) is reachable via postgres_fdw but has lat/lon only, no geom, and a per-listing nearest query over the FDW is ~1.16s/row (~13h for 43k — unusable). This adds a geo-nearest matcher that fills building_cadastral_number for the ~93% of listings that have geo. (estimator.py Tier-0 already consumes building_cadastral_number as a house-match hint, so this is pure net gain — no caller change.)

Approach

Local mirror (124_cad_buildings_local.sql): empty cad_buildings_local(cad_num PK, …, geom geometry(Point,4326)) + GIST. Migration does NOT read the FDW (deploy never depends on the gendesign DB being up).
Refresh job: single bulk FDW scan TRUNCATE + INSERT … FROM gendesign_cad_buildings building ST_SetSRID(ST_MakePoint(lon,lat),4326) — 36732 rows in 1.47s.
Matcher: set-based LATERAL KNN — a geometry-space ST_DWithin(cb.geom, l.geom, :deg_gate) (GIST-indexable degrees, no geography cast) + geom <-> l.geom KNN order + ST_DistanceSphere(...) <= :threshold_m metric recheck on the single nearest row. :deg_gate is a degree radius (1.5× safety on the cos(lat)-shrunk longitude axis) that always encloses the metric threshold, so the GIST pre-gate never drops true positives. A geography-cast ST_DWithin in the WHERE defeats the index (ran 58s+ unfinished); this is ~175× faster.
Scheduler: trigger_cadastral_geo_match_run + dispatch source='cadastral_geo_match'; seed (125_…sql) with next_run_at=tomorrow 09:00 UTC (no deploy-time fire).

Correctness fix included (caught in review)

The original matcher chunked with OFFSET over the very predicate it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET (reset or advanced) skips un-processed rows: once ≥batch_size unmatched listings (beyond threshold) pile up at the low-id front, the chunk matches 0 and the loop breaks early, leaving higher-id matchable listings unfilled. Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 & 4, then breaks — missing 7 & 10. The unit tests used single-chunk fixtures so never caught it. Fix: the full UPDATE is ~5s for ~41k listings, so it's now one set-based LATERAL KNN UPDATE over all candidates — correct and fast (batch_size kept in the signature for schedule-param compat, unused).

Prod evidence (read-only; scratch table, rolled back, `listings` untouched)

Refresh: 36732 rows / 1.47s. Full match UPDATE: 5.43s, 16027 listings matched across 2229 distinct buildings.
EXPLAIN: Index Scan using cad_buildings_local_geom_idx, geom <-> l.geom KNN, GIST used per listing.
Coverage (5000 sample, geo-present + bcn NULL): 30m → 22.2%, 50m → 35.2%, 100m → 68.2% (full-pop @ 50m = 38.3%).

Tests

test_cadastral_geo_match.py: 18 + 1 optional real-PostGIS (self-skips). Includes test_match_is_single_statement_no_offset (guards the OFFSET regression) + the geometry-gate/DistanceSphere shape + deg-gate-encloses-threshold + psycopg-v3 CAST discipline.
Full gate: pytest -q (2 deselects) → 1931 passed, 1 skipped, 2 deselected. ruff clean.

Note

Geo-nearest is approximate (street-level-geocoded listings match the nearest building, not necessarily the exact one) — documented in module/match docstrings + migration comments + logged threshold. Exact cadastral + parcel-containment are deferred (cad_parcels FDW not exposed). Post-merge: migrations auto-apply; nudge UPDATE scrape_schedules SET next_run_at=now() WHERE source='cadastral_geo_match' to backfill immediately.

## What & why `listings.building_cadastral_number` / `cadastral_number` are **0% filled** (columns exist, empty), while listing geo is ~93–100% present. The cadastral building registry (`gendesign_cad_buildings`, ~36.7k EKB rows) is reachable via postgres_fdw but has **lat/lon only, no geom**, and a per-listing nearest query over the FDW is **~1.16s/row** (~13h for 43k — unusable). This adds a geo-nearest matcher that fills `building_cadastral_number` for the ~93% of listings that have geo. (`estimator.py` Tier-0 already consumes `building_cadastral_number` as a house-match hint, so this is pure net gain — no caller change.) ## Approach - **Local mirror** (`124_cad_buildings_local.sql`): empty `cad_buildings_local(cad_num PK, …, geom geometry(Point,4326))` + GIST. Migration does NOT read the FDW (deploy never depends on the gendesign DB being up). - **Refresh job**: single bulk FDW scan `TRUNCATE + INSERT … FROM gendesign_cad_buildings` building `ST_SetSRID(ST_MakePoint(lon,lat),4326)` — **36732 rows in 1.47s**. - **Matcher**: set-based LATERAL KNN — a **geometry-space** `ST_DWithin(cb.geom, l.geom, :deg_gate)` (GIST-indexable degrees, no geography cast) + `geom <-> l.geom` KNN order + `ST_DistanceSphere(...) <= :threshold_m` metric recheck on the single nearest row. `:deg_gate` is a degree radius (1.5× safety on the cos(lat)-shrunk longitude axis) that always encloses the metric threshold, so the GIST pre-gate never drops true positives. A geography-cast `ST_DWithin` in the WHERE defeats the index (ran 58s+ unfinished); this is **~175× faster**. - **Scheduler**: `trigger_cadastral_geo_match_run` + dispatch `source='cadastral_geo_match'`; seed (`125_…sql`) with `next_run_at`=tomorrow 09:00 UTC (no deploy-time fire). ## Correctness fix included (caught in review) The original matcher chunked with `OFFSET` over the very predicate it mutates (`building_cadastral_number IS NULL`). Matched rows drop out, so OFFSET (reset **or** advanced) **skips un-processed rows**: once ≥`batch_size` unmatched listings (beyond threshold) pile up at the low-id front, the chunk matches 0 and the loop breaks early, leaving higher-id matchable listings unfilled. Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 & 4, then breaks — missing 7 & 10. The unit tests used single-chunk fixtures so never caught it. **Fix:** the full UPDATE is ~5s for ~41k listings, so it's now **one set-based LATERAL KNN UPDATE** over all candidates — correct and fast (`batch_size` kept in the signature for schedule-param compat, unused). ## Prod evidence (read-only; scratch table, rolled back, `listings` untouched) - Refresh: 36732 rows / 1.47s. Full match UPDATE: **5.43s**, **16027 listings matched** across 2229 distinct buildings. - EXPLAIN: `Index Scan using cad_buildings_local_geom_idx`, `geom <-> l.geom` KNN, GIST used per listing. - Coverage (5000 sample, geo-present + bcn NULL): 30m → 22.2%, 50m → 35.2%, 100m → 68.2% (full-pop @ 50m = 38.3%). ## Tests - `test_cadastral_geo_match.py`: 18 + 1 optional real-PostGIS (self-skips). Includes `test_match_is_single_statement_no_offset` (guards the OFFSET regression) + the geometry-gate/DistanceSphere shape + deg-gate-encloses-threshold + psycopg-v3 CAST discipline. - Full gate: `pytest -q` (2 deselects) → **1931 passed, 1 skipped, 2 deselected**. ruff clean. ## Note Geo-nearest is **approximate** (street-level-geocoded listings match the nearest building, not necessarily the exact one) — documented in module/match docstrings + migration comments + logged threshold. Exact cadastral + parcel-containment are deferred (cad_parcels FDW not exposed). Post-merge: migrations auto-apply; nudge `UPDATE scrape_schedules SET next_run_at=now() WHERE source='cadastral_geo_match'` to backfill immediately.

lekss361 added 2 commits 2026-06-18 07:28:47 +00:00

feat(cadastral): geo-nearest building matcher via local cad mirror edd187b47f

Backfill listings.building_cadastral_number (was 0%) from the nearest
cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table
with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k
listings). Instead materialize the FDW once into a LOCAL cad_buildings_local
table (Point geom + GIST), then run a fast local KNN nearest-neighbour join.

Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST
index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric
recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE
defeated the index (58s+, full update never finished); the geometry-gate design
does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end),
matching 16027 listings at 50m across 2229 distinct buildings.

The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches
the nearest building within threshold_m (default 50m), not necessarily its exact
cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels
FDW not exposed). Threshold is logged.

- 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW
  (deploy-independent); populated by the refresh job.
- 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at
  tomorrow 09:00 UTC (after geocode_missing).
- tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan),
  match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match
  run-lifecycle wrapper.
- scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).

fix(cadastral): single-statement match UPDATE — OFFSET chunk skipped rows

CI / changes (pull_request) Successful in 7s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

f9769571a3

The chunked matcher paged the candidate set with OFFSET over the very predicate
it mutates (building_cadastral_number IS NULL). Matched rows drop out, so OFFSET
(reset to 0 OR advanced) skips un-processed rows: once >= batch_size unmatched
listings (beyond threshold) accumulate at the low-id front, the chunk matches 0
and the loop breaks early, leaving higher-id matchable listings unfilled.
Counterexample (batch=3, matchable ids 1,4,7,10): matches 1 and 4 then breaks,
missing 7 and 10. Unit tests used single-chunk fixtures so never caught it.

The full UPDATE is ~5s for ~41k listings, so replace the loop with ONE set-based
LATERAL KNN UPDATE over all candidates — correct and fast. batch_size kept in the
signature for schedule-param compat (now unused).