feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1091, #1062) #1118

Merged

bot-reviewer merged 1 commit from feat/page-reservation-parser into main

2026-06-07 10:15:03 +00:00

lekss361 commented

2026-06-07 09:59:58 +00:00

Owner

Что

MVP-скаффолд для ИРД-5 (#1062 изъятие/резервирование ЗУ) эпика #1067. Геометрии зон изъятия нет ни в одном spatial-источнике — перечень ЗУ существует только текстом в постановлениях. Парсим PDF → кад-номера → join к cad_parcels за геометрией → per-parcel флаг «в действующем постановлении №X».

Состав

data/sql/136_land_reservation.sql — таблица land_reservation(cad_num, reservation_kind, basis_act, act_number, act_date, purpose, doc_url, is_active, raw_excerpt, …), UNIQUE(cad_num, act_number), idempotent (auto-apply deploy).
page_reservation_parser.py — extract_reservations(text) / extract_from_pdf(bytes) (pdfplumber): regex кад-номеров \d{2}:\d{2}:\d{6,7}:\d+, kind (изъятие/резервирование), act_number/date, purpose best-effort, raw_excerpt.
reservation_ingest.py — Celery task ingest_reservations(docs), seed = 509-ПП (Татищева). Ручной/on-demand (beat НЕ добавлен — постановления редки). httpx-fetch graceful per-doc.
reservation_lookup.py — parcel_reservations(db, cad_num) graceful read-side.
pdfplumber добавлен в deps (+uv.lock).
21 unit-тест (parser regex / lookup graceful / upsert), сеть/PDF замоканы.

НЕ входит (намеренно)

Врезка parcel_reservations в build_ird_analyze_block — follow-up после merge #1115 (избегаем конфликта по ird_analyze.py).

VERIFY перед прод-прогоном

_SEED_DOCS[0]["url"] = placeholder pravo.gov66.ru/.../509-pp — уточнить точный URL.
TLS: pravo.gov66.ru на росс-CA → в проде добавить CA-bundle как в ekb_geoportal_client.
purpose-эвристика и точность извлечения — сверить на живом PDF 509-ПП.

Часть #1062 / #1091 / #1067.

## Что MVP-скаффолд для ИРД-5 (#1062 изъятие/резервирование ЗУ) эпика #1067. Геометрии зон изъятия нет ни в одном spatial-источнике — перечень ЗУ существует только текстом в постановлениях. Парсим PDF → кад-номера → join к `cad_parcels` за геометрией → per-parcel флаг «в действующем постановлении №X». ## Состав - `data/sql/136_land_reservation.sql` — таблица `land_reservation(cad_num, reservation_kind, basis_act, act_number, act_date, purpose, doc_url, is_active, raw_excerpt, …)`, UNIQUE(cad_num, act_number), idempotent (auto-apply deploy). - `page_reservation_parser.py` — `extract_reservations(text)` / `extract_from_pdf(bytes)` (pdfplumber): regex кад-номеров `\d{2}:\d{2}:\d{6,7}:\d+`, kind (изъятие/резервирование), act_number/date, purpose best-effort, raw_excerpt. - `reservation_ingest.py` — Celery task `ingest_reservations(docs)`, seed = 509-ПП (Татищева). Ручной/on-demand (beat НЕ добавлен — постановления редки). httpx-fetch graceful per-doc. - `reservation_lookup.py` — `parcel_reservations(db, cad_num)` graceful read-side. - pdfplumber добавлен в deps (+uv.lock). - 21 unit-тест (parser regex / lookup graceful / upsert), сеть/PDF замоканы. ## НЕ входит (намеренно) Врезка `parcel_reservations` в `build_ird_analyze_block` — follow-up после merge #1115 (избегаем конфликта по `ird_analyze.py`). ## VERIFY перед прод-прогоном - `_SEED_DOCS[0]["url"]` = placeholder `pravo.gov66.ru/.../509-pp` — уточнить точный URL. - TLS: pravo.gov66.ru на росс-CA → в проде добавить CA-bundle как в `ekb_geoportal_client`. - purpose-эвристика и точность извлечения — сверить на живом PDF 509-ПП. Часть #1062 / #1091 / #1067.

lekss361 added 1 commit 2026-06-07 09:59:59 +00:00

feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1091 , #1062 )

CI / changes (push) Successful in 7s

Details

CI / changes (pull_request) Successful in 7s

Details

CI / backend-tests (push) Successful in 5m56s

Details

CI / backend-tests (pull_request) Successful in 5m57s

Details

b58f399d2d

lekss361 added the

labels 2026-06-07 10:00:03 +00:00

bot-reviewer approved these changes 2026-06-07 10:14:57 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE — staff+ review (9 files / ~926 lines, MVP scaffold).

Verified against live prod (deploy-safety gate):

Migration 136_land_reservation.sql: land_reservation absent, 136 not in _schema_migrations (prod max=135) → next-free, applies once. cad_parcels present (read-side join target).
Idempotent: BEGIN/COMMIT, CREATE TABLE/INDEX IF NOT EXISTS, UNIQUE via guarded pg_constraint DO-block (mirrors м.132). No data inserts.
Task inert on prod: no beat schedule, seed URL is acknowledged placeholder → ingest_reservations is a no-op until manually triggered. Merge risk = migration only (verified safe).

Conventions: SAVEPOINT-per-row + SessionLocal try/finally + CAST(:x AS type) (psycopg v3) mirror ird_harvest.py; parcel_reservations graceful-degrade (OperationalError/ProgrammingError→[]) mirrors ird_overlay_lookup.py. verify=False for pravo.gov66.ru matches existing repo precedent (ekburg_permits.py/pzz_loader.py — Russian Минцифры CA). uv.lock consistent with pyproject (pdfplumber 0.11.9 + pypdfium2 5.9.0 manylinux x86_64 wheels → Docker build OK). Vault research consistent (no spatial source for изъятие/резервирование zones).

🟡 MINOR findings → consolidated hardening follow-up #1120 (fix BEFORE first real ingest run):

ReDoS _RE_ACT_NUMBER (page_reservation_parser.py:31) — nested quantifier (?:[chars]+[sep]?)* + suffix alternation → exponential backtracking (~2.2s @ 25 chars) on № <digits> not followed by a valid suffix (common in PDF preamble). Mitigated short-term: .search returns on first header match (fast for well-formed docs). Fix: mandatory separator between tokens / cap search to header.
NULL idempotency — UNIQUE(cad_num, act_number) doesn't dedupe NULL act_number (NULL≠NULL) → dup rows on re-ingest when act parse fails. Fix: partial unique idx (cad_num) WHERE act_number IS NULL.
CA bundle — verify=False acceptable for MVP (public read-only PDF, no creds); track repo-wide Russian-CA-bundle env var.
_MONTH_MAP insertion-order dep — "март" must precede "ма" (startswith); correct now but fragile to dict re-sort. Fix: explicit stems.

Also: vault entry for the PDF-parse approach + land_reservation grain (CLAUDE.md rule 6) — folded into follow-up.

None block: migration safe + task inert. Merging (squash); findings tracked in #1120 (p2, must-fix-before-go-live).

✅ **APPROVE** — staff+ review (9 files / ~926 lines, MVP scaffold). **Verified against live prod (deploy-safety gate):** - Migration `136_land_reservation.sql`: `land_reservation` absent, `136` not in `_schema_migrations` (prod max=135) → next-free, applies once. `cad_parcels` present (read-side join target). - Idempotent: BEGIN/COMMIT, CREATE TABLE/INDEX IF NOT EXISTS, UNIQUE via guarded `pg_constraint` DO-block (mirrors м.132). No data inserts. - **Task inert on prod**: no beat schedule, seed URL is acknowledged placeholder → `ingest_reservations` is a no-op until manually triggered. **Merge risk = migration only (verified safe).** **Conventions:** SAVEPOINT-per-row + SessionLocal try/finally + `CAST(:x AS type)` (psycopg v3) mirror `ird_harvest.py`; `parcel_reservations` graceful-degrade (OperationalError/ProgrammingError→[]) mirrors `ird_overlay_lookup.py`. `verify=False` for pravo.gov66.ru matches existing repo precedent (`ekburg_permits.py`/`pzz_loader.py` — Russian Минцифры CA). uv.lock consistent with pyproject (pdfplumber 0.11.9 + pypdfium2 5.9.0 manylinux x86_64 wheels → Docker build OK). Vault research consistent (no spatial source for изъятие/резервирование zones). **🟡 MINOR findings → consolidated hardening follow-up #1120 (fix BEFORE first real ingest run):** 1. **ReDoS** `_RE_ACT_NUMBER` (`page_reservation_parser.py:31`) — nested quantifier `(?:[chars]+[sep]?)*` + suffix alternation → exponential backtracking (~2.2s @ 25 chars) on `№ <digits>` not followed by a valid suffix (common in PDF preamble). Mitigated short-term: `.search` returns on first header match (fast for well-formed docs). Fix: mandatory separator between tokens / cap search to header. 2. **NULL idempotency** — `UNIQUE(cad_num, act_number)` doesn't dedupe NULL act_number (NULL≠NULL) → dup rows on re-ingest when act parse fails. Fix: partial unique idx `(cad_num) WHERE act_number IS NULL`. 3. **CA bundle** — `verify=False` acceptable for MVP (public read-only PDF, no creds); track repo-wide Russian-CA-bundle env var. 4. **`_MONTH_MAP` insertion-order dep** — `"март"` must precede `"ма"` (startswith); correct now but fragile to dict re-sort. Fix: explicit stems. Also: vault entry for the PDF-parse approach + `land_reservation` grain (CLAUDE.md rule 6) — folded into follow-up. None block: migration safe + task inert. Merging (squash); findings tracked in #1120 (p2, must-fix-before-go-live).