fix(cadastre): filter Point/LineString geometry in upsert_parcel/zouit (worker crash root cause) #184

Merged

lekss361 merged 1 commit from fix/cadastre-parcel-geometry-type-filter into main

2026-05-15 15:37:29 +00:00

lekss361 commented

2026-05-15 15:33:19 +00:00

(Migrated from github.com)

Summary

Worker crash root cause from pilot v8 hang at 25/50. Worker logs revealed:
```
psycopg.errors.InvalidParameterValue:
Geometry type (Point) does not match column type (Polygon)
```

Quarter `66:41:0104002` parcel `66:41:0104002:18` — NSPD returned Point geometry for ЗУ без detailed boundary. `cad_parcels.geom` schema is strict POLYGON → INSERT fail → autoretry exhausted → worker died → job hung (heartbeat stale at 15:27:43, never resumed).

Fix

Filter geometry.type at Python level in:

`upsert_parcel` — cad_parcels schema POLYGON strict
`upsert_zouit` — cad_zouit schema MULTIPOLYGON strict (defensive — LineString could also appear)

Same pattern as existing `upsert_quarter_geom_from_feature`. Non-Polygon geometry → `geom=NULL`, raw_props preserved (координаты не теряются, доступны для analyze).

Tables with permissive GEOMETRY schema unaffected:

cad_buildings, cad_constructions, cad_enk, cad_oncs — accept any type
cad_quarters_geom (MultiPolygon) — already filtered

Pilot v8 validation (before crash)

Despite crash, 25 quarters processed validated all 4 prior fixes:

✅ Race fix #183 — parcels +446, buildings +393 (vs prior 0)
✅ Auto-heal #181 — 20 broken quarters healed
✅ Metrics #182 — requests_count 1825 (vs prior 5)
⚠️ Per-cat #180 — pilot batch без ZOUIT/ENK meta, нужен другой quarter set

Test plan

36 → 39 tests passing locally
After merge + deploy → fire pilot v9 → expect 50/50 done без crash
Затем ekb_full → expect 2408/2408 done

Closes part of #168 (Phase 1 geometry crash protection).

## Summary Worker crash root cause from pilot v8 hang at 25/50. Worker logs revealed: \`\`\` psycopg.errors.InvalidParameterValue: Geometry type (Point) does not match column type (Polygon) \`\`\` Quarter \`66:41:0104002\` parcel \`66:41:0104002:18\` — NSPD returned Point geometry for ЗУ без detailed boundary. \`cad_parcels.geom\` schema is **strict POLYGON** → INSERT fail → autoretry exhausted → worker died → job hung (heartbeat stale at 15:27:43, never resumed). ## Fix Filter geometry.type at Python level in: - \`upsert_parcel\` — cad_parcels schema POLYGON strict - \`upsert_zouit\` — cad_zouit schema MULTIPOLYGON strict (defensive — LineString could also appear) Same pattern as existing \`upsert_quarter_geom_from_feature\`. Non-Polygon geometry → \`geom=NULL\`, raw_props preserved (координаты не теряются, доступны для analyze). Tables with permissive GEOMETRY schema unaffected: - cad_buildings, cad_constructions, cad_enk, cad_oncs — accept any type - cad_quarters_geom (MultiPolygon) — already filtered ## Pilot v8 validation (before crash) Despite crash, 25 quarters processed validated all 4 prior fixes: - ✅ Race fix #183 — parcels +446, buildings +393 (vs prior 0) - ✅ Auto-heal #181 — 20 broken quarters healed - ✅ Metrics #182 — requests_count 1825 (vs prior 5) - ⚠️ Per-cat #180 — pilot batch без ZOUIT/ENK meta, нужен другой quarter set ## Test plan - [x] 36 → 39 tests passing locally - [ ] After merge + deploy → fire pilot v9 → expect 50/50 done без crash - [ ] Затем ekb_full → expect 2408/2408 done Closes part of #168 (Phase 1 geometry crash protection).

lekss361 commented

2026-05-15 15:37:15 +00:00

(Migrated from github.com)

Review

CI зелёный (backend 59s + frontend 1m38s) + cross-check (agent):

✅ Schema claims подтверждены

По data/sql/92_cad_bulk_layers.sql + 58_schema_cad_quarters.sql:

cad_parcels.geom = GEOMETRY(Polygon, 4326) strict — filter применён ✓
cad_zouit.geom = GEOMETRY(MultiPolygon, 4326) strict — filter применён ✓
cad_buildings_new/cad_constructions/cad_oncs/cad_enk.geom = GEOMETRY(Geometry, 4326) permissive — filter не нужен ✓
cad_quarters_geom.geom = geometry(MultiPolygon, 4326) strict — уже filter'нут в upsert_quarter_geom_from_feature (PR #181) ✓
cad_parcels_geom — back-compat VIEW (migration 92 step M), не writeable

Все NSPD-write точки strict-schema таблиц покрыты. Других strict таблиц, которые пишет NSPD bulk harvest, нет.

✅ Pattern consistency

Whitelist ("Polygon", "MultiPolygon") идентичен upsert_quarter_geom_from_feature:1015. Та же defensive idiom: geom = feature.geometry or {}, geom.get("type") if isinstance(geom, dict), добавлен INFO log на non-Polygon для ops visibility.

✅ raw_props preserved

raw_props строится из raw_opts независимо от geom_json (bulk_harvest.py:502, 977) — координаты не теряются, доступны для analyze пайплайна.

✅ Edge cases

MultiPoint/GeometryCollection/MultiLineString/empty dict/None — все падают в geom_json=None через whitelist. Sane default.

✅ Test coverage

3 новых теста:

test_upsert_parcel_skips_geom_for_point_geometry — Point geom → NULL, INSERT proceeds, raw_props сохранён
test_upsert_parcel_keeps_polygon_geometry — Polygon → kept (happy path)
test_upsert_zouit_skips_geom_for_linestring — LineString → NULL

Assertions корректны: db.execute.assert_called_once() + call_params["geom"] is None.

✅ Drift отсутствует

Server-side only filter, no TS callers.

Контекст

Это пятый root-cause fix в серии #168 после #175 SSL, #176 floors, #177 parse, #178 cast, #179 broken geom skip, #180 per-cat, #181 auto-heal, #182 metrics, #183 race. После merge + deploy pilot v9 должен наконец-то завершиться 50/50 без crash → ekb_full 2408/2408.

approve merge

## Review CI зелёный (backend 59s + frontend 1m38s) + cross-check (agent): ### ✅ Schema claims подтверждены По `data/sql/92_cad_bulk_layers.sql` + `58_schema_cad_quarters.sql`: - `cad_parcels.geom` = `GEOMETRY(Polygon, 4326)` strict — filter применён ✓ - `cad_zouit.geom` = `GEOMETRY(MultiPolygon, 4326)` strict — filter применён ✓ - `cad_buildings_new/cad_constructions/cad_oncs/cad_enk.geom` = `GEOMETRY(Geometry, 4326)` permissive — filter не нужен ✓ - `cad_quarters_geom.geom` = `geometry(MultiPolygon, 4326)` strict — уже filter'нут в `upsert_quarter_geom_from_feature` (PR #181) ✓ - `cad_parcels_geom` — back-compat VIEW (migration 92 step M), не writeable Все NSPD-write точки strict-schema таблиц покрыты. Других strict таблиц, которые пишет NSPD bulk harvest, нет. ### ✅ Pattern consistency Whitelist `("Polygon", "MultiPolygon")` идентичен `upsert_quarter_geom_from_feature:1015`. Та же defensive idiom: `geom = feature.geometry or {}`, `geom.get("type") if isinstance(geom, dict)`, добавлен INFO log на non-Polygon для ops visibility. ### ✅ raw_props preserved `raw_props` строится из `raw_opts` независимо от `geom_json` (`bulk_harvest.py:502, 977`) — координаты не теряются, доступны для analyze пайплайна. ### ✅ Edge cases `MultiPoint`/`GeometryCollection`/`MultiLineString`/empty dict/None — все падают в `geom_json=None` через whitelist. Sane default. ### ✅ Test coverage 3 новых теста: - `test_upsert_parcel_skips_geom_for_point_geometry` — Point geom → NULL, INSERT proceeds, raw_props сохранён - `test_upsert_parcel_keeps_polygon_geometry` — Polygon → kept (happy path) - `test_upsert_zouit_skips_geom_for_linestring` — LineString → NULL Assertions корректны: `db.execute.assert_called_once()` + `call_params["geom"] is None`. ### ✅ Drift отсутствует Server-side only filter, no TS callers. ### Контекст Это пятый root-cause fix в серии #168 после #175 SSL, #176 floors, #177 parse, #178 cast, #179 broken geom skip, #180 per-cat, #181 auto-heal, #182 metrics, #183 race. После merge + deploy pilot v9 должен наконец-то завершиться 50/50 без crash → ekb_full 2408/2408. approve merge