bot-backend
edd187b47f
Backfill listings.building_cadastral_number (was 0%) from the nearest cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k listings). Instead materialize the FDW once into a LOCAL cad_buildings_local table (Point geom + GIST), then run a fast local KNN nearest-neighbour join. Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE defeated the index (58s+, full update never finished); the geometry-gate design does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end), matching 16027 listings at 50m across 2229 distinct buildings. The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches the nearest building within threshold_m (default 50m), not necessarily its exact cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels FDW not exposed). Threshold is logged. - 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW (deploy-independent); populated by the refresh job. - 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at tomorrow 09:00 UTC (after geocode_missing). - tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan), match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match run-lifecycle wrapper. - scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).
47 lines
2.2 KiB
PL/PgSQL
47 lines
2.2 KiB
PL/PgSQL
-- 124_cad_buildings_local.sql
|
|
-- Local materialized mirror of gendesign_cad_buildings (postgres_fdw foreign table)
|
|
-- for fast geo-nearest building matching of listings.
|
|
--
|
|
-- WHY (perf fact, measured live):
|
|
-- gendesign_cad_buildings is a postgres_fdw foreign table (~36.7k EKB rows) with NO
|
|
-- PostGIS geom — only scalar lat/lon. A per-listing nearest-building query over the FDW
|
|
-- takes ~1.16s/row (lat/lon bbox prefilter). With ~43k listings that is ~13h — UNUSABLE.
|
|
-- Solution: materialize the FDW once into a LOCAL table with a real Point geom + GIST
|
|
-- index, then run a fast local KNN nearest-neighbour join (geom <-> point, GIST-backed).
|
|
--
|
|
-- This migration creates the EMPTY table + indexes only. It does NOT read the FDW —
|
|
-- so deploy never depends on the gendesign DB / FDW server being reachable.
|
|
-- The table is populated by the refresh job (app/tasks/refresh_cad_buildings_local.py),
|
|
-- driven by the in-app scheduler (source='cadastral_geo_match', combined refresh+match).
|
|
--
|
|
-- Schedule seed (cadastral_geo_match) lives in 125_scrape_schedules_seed_cadastral_geo_match.sql.
|
|
--
|
|
-- DEPENDENCIES: PostGIS (postgis extension, present since geom on listings/houses).
|
|
-- Idempotent: CREATE TABLE/INDEX IF NOT EXISTS. Safe to re-run.
|
|
|
|
BEGIN;
|
|
|
|
CREATE TABLE IF NOT EXISTS cad_buildings_local (
|
|
cad_num text PRIMARY KEY,
|
|
readable_address text,
|
|
year_built int,
|
|
floors int,
|
|
area_m2 numeric,
|
|
purpose text,
|
|
lat double precision,
|
|
lon double precision,
|
|
geom geometry(Point, 4326)
|
|
);
|
|
|
|
COMMENT ON TABLE cad_buildings_local IS
|
|
'Local mirror of gendesign_cad_buildings FDW (Rosreestr building registry, EKB ~36.7k). '
|
|
'Populated by app/tasks/refresh_cad_buildings_local.py (single bulk FDW scan, TRUNCATE+INSERT). '
|
|
'Point geom + GIST enables fast local KNN nearest-building matching '
|
|
'(run_cadastral_geo_match) for listings.building_cadastral_number. '
|
|
'Migration 124. Refresh+match driven by scheduler source=cadastral_geo_match.';
|
|
|
|
-- GIST on geom — required for the KNN <-> operator and ST_DWithin predicate.
|
|
CREATE INDEX IF NOT EXISTS cad_buildings_local_geom_idx
|
|
ON cad_buildings_local USING GIST (geom);
|
|
|
|
COMMIT;
|