lekss361/gendesign

Fork 0

feat(22i): domrf_kn_documents new table + parser #305

Merged

lekss361 merged 1 commit from fix/22i-documents-v2 into main

2026-05-17 15:21:32 +00:00

lekss361 commented

2026-05-17 15:14:16 +00:00

Owner

Summary

Issue #297 sub-task 22i (1d). NEW table domrf_kn_documents + parser для PDF документов (декларации, разрешения, проектная, отчётность, прочее).

Payload audit (obj_id=65136)

DOM.РФ kn/object bulk list НЕ содержит documents field. Documents живут на per-object endpoint /сервисы/api/object/{obj_id}/documents (как /infrastructure, /photos) — ещё не scraped.

В этом PR — только schema + parser. Wiring endpoint в scrape loop — follow-up PR.

Changes

data/sql/112_22i_domrf_documents.sql (NN renamed 100→112 — collision с merged 100_fix_mv_layout_velocity + 111_22f_obj_checks):

CREATE TABLE domrf_kn_documents (
  id            bigserial PRIMARY KEY,
  obj_id        bigint NOT NULL,
  doc_type      text NOT NULL,  -- декларация/разрешение/проектная/отчётность/прочее
  doc_num       text,
  posted_at     date,
  file_url      text NOT NULL,
  size_bytes    bigint,
  local_path    text,
  downloaded_at timestamptz,
  scraped_at    timestamptz DEFAULT NOW(),
  UNIQUE (obj_id, doc_type, doc_num, file_url)
);
CREATE INDEX domrf_kn_documents_obj_type_idx ON (obj_id, doc_type);

backend/app/services/scrapers/documents.py (NEW, +193 lines):
- extract_documents(raw_payload) -> list[dict] — parse from payload (when endpoint scraped)
- upsert_documents(db, obj_id, docs) — INSERT с ON CONFLICT DO UPDATE для local_path/downloaded_at, SAVEPOINT per-row
- download_document_stub() — NotImplementedError placeholder (follow-up PR)
Vault: fixes/Fix_BUG22i_Documents_NewTable_May17.md

domrf_kn.py НЕ trog'нут — wiring отдельным PR.

Test plan

Migration applies cleanly
Re-apply idempotent
Follow-up: scrape /сервисы/api/object/{obj_id}/documents + persist через upsert_documents()

Related: issue #297 sub-task 22i

## Summary Issue #297 sub-task **22i** (1d). NEW table `domrf_kn_documents` + parser для PDF документов (декларации, разрешения, проектная, отчётность, прочее). ## Payload audit (obj_id=65136) DOM.РФ kn/object **bulk list** НЕ содержит documents field. Documents живут на **per-object endpoint** `/сервисы/api/object/{obj_id}/documents` (как `/infrastructure`, `/photos`) — ещё **не scraped**. В этом PR — только schema + parser. Wiring endpoint в scrape loop — **follow-up PR**. ## Changes - `data/sql/112_22i_domrf_documents.sql` (NN renamed 100→112 — collision с merged 100_fix_mv_layout_velocity + 111_22f_obj_checks): ```sql CREATE TABLE domrf_kn_documents ( id bigserial PRIMARY KEY, obj_id bigint NOT NULL, doc_type text NOT NULL, -- декларация/разрешение/проектная/отчётность/прочее doc_num text, posted_at date, file_url text NOT NULL, size_bytes bigint, local_path text, downloaded_at timestamptz, scraped_at timestamptz DEFAULT NOW(), UNIQUE (obj_id, doc_type, doc_num, file_url) ); CREATE INDEX domrf_kn_documents_obj_type_idx ON (obj_id, doc_type); ``` - `backend/app/services/scrapers/documents.py` (NEW, +193 lines): - `extract_documents(raw_payload) -> list[dict]` — parse from payload (when endpoint scraped) - `upsert_documents(db, obj_id, docs)` — INSERT с ON CONFLICT DO UPDATE для local_path/downloaded_at, SAVEPOINT per-row - `download_document_stub()` — `NotImplementedError` placeholder (follow-up PR) - Vault: `fixes/Fix_BUG22i_Documents_NewTable_May17.md` **`domrf_kn.py` НЕ trog'нут** — wiring отдельным PR. ## Test plan - [ ] Migration applies cleanly - [ ] Re-apply idempotent - [ ] Follow-up: scrape `/сервисы/api/object/{obj_id}/documents` + persist через `upsert_documents()` Related: issue #297 sub-task 22i

lekss361 added 1 commit 2026-05-17 15:14:17 +00:00

feat(22i): domrf_kn_documents new table + parser 5bcaa67a6d

- data/sql/112_22i_domrf_documents.sql: CREATE TABLE domrf_kn_documents
  with UNIQUE(obj_id, doc_type, doc_num, file_url) + index (obj_id, doc_type).
  Renamed from original 100_* to 112 to avoid NN collision (100 = mv_layout
  weighted_avg PR #290 merged; 111 = 22f obj_checks PR #303).
- backend/app/services/scrapers/documents.py: extract_documents() +
  upsert_documents() with SAVEPOINT per-row + canonical doc_type normalisation
  + download stub.

Payload audit (obj_id=65136): kn/object bulk list does NOT contain documents
field. Documents live on per-object endpoint /object/{obj_id}/documents
(not yet scraped). Follow-up PR needed to wire endpoint into domrf_kn.py
scrape loop.

Issue #297 22i.

lekss361 referenced this pull request from a commit

2026-05-17 15:21:09 +00:00

fix(22begh): rename migration 112 → 113 (collision with PR #305 sf-22i)

lekss361 referenced this pull request

2026-05-17 15:21:31 +00:00

feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 #306

lekss361 merged commit 171a029997 into main

2026-05-17 15:21:32 +00:00

lekss361 referenced this pull request from a commit

2026-05-17 15:21:32 +00:00

feat(22i): domrf_kn_documents new table + parser (#305)

lekss361 commented

2026-05-17 15:22:01 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Что проверено:

Migration 112_ sequential после 111_22f (PR #303). BEGIN/COMMIT + IF NOT EXISTS — idempotent.
upsert_documents(): ON CONFLICT DO UPDATE с COALESCE(local_path, EXCLUDED.local_path) — preserve already-downloaded paths. SAVEPOINT per-row через begin_nested() — error isolation OK.
extract_documents(): defensive (isinstance(item, dict), file_url.startswith("http"), dual key fallback fileUrl/file_url).
download_document_stub() — NotImplementedError, no production callers (grep clean).
domrf_kn.py не trog'нут — wiring отдельным PR, safe deploy.
New table, no consumers → нулевой anti-regression risk.

Nits (non-blocking, можно в follow-up):

🟡 UNIQUE (obj_id, doc_type, doc_num, file_url) — doc_num nullable, default NULLS DISTINCT. PR #304 использовал NULLS NOT DISTINCT — рекомендую consistency, но в реальном payload file_url уникален per документ, дублей не будет.
🟢 size_raw = item.get("fileSize") or item.get("file_size") — для 0-byte файлов or skip'нет в fallback. Edge case, реально PDF никогда не 0 bytes.
🟢 Header comment в SQL говорит "Migration 100" но файл 112_ — стало stale при rename.

Smoke после deploy:

\d domrf_kn_documents
-- проверить columns + unique constraint + index

Follow-up PR:

Wiring /сервисы/api/object/{obj_id}/documents в scrape loop (domrf_kn.py)
Celery-задача download_domrf_documents (replace stub)

Merged via deep-code-reviewer — verdict APPROVE. **Что проверено:** - Migration `112_` sequential после 111_22f (PR #303). `BEGIN/COMMIT` + `IF NOT EXISTS` — idempotent. - `upsert_documents()`: `ON CONFLICT DO UPDATE` с `COALESCE(local_path, EXCLUDED.local_path)` — preserve already-downloaded paths. SAVEPOINT per-row через `begin_nested()` — error isolation OK. - `extract_documents()`: defensive (`isinstance(item, dict)`, `file_url.startswith("http")`, dual key fallback `fileUrl`/`file_url`). - `download_document_stub()` — `NotImplementedError`, no production callers (grep clean). - `domrf_kn.py` не trog'нут — wiring отдельным PR, safe deploy. - New table, no consumers → нулевой anti-regression risk. **Nits (non-blocking, можно в follow-up):** - 🟡 `UNIQUE (obj_id, doc_type, doc_num, file_url)` — `doc_num` nullable, default `NULLS DISTINCT`. PR #304 использовал `NULLS NOT DISTINCT` — рекомендую consistency, но в реальном payload `file_url` уникален per документ, дублей не будет. - 🟢 `size_raw = item.get("fileSize") or item.get("file_size")` — для 0-byte файлов `or` skip'нет в fallback. Edge case, реально PDF никогда не 0 bytes. - 🟢 Header comment в SQL говорит "Migration 100" но файл `112_` — стало stale при rename. **Smoke после deploy:** ```sql \d domrf_kn_documents -- проверить columns + unique constraint + index ``` **Follow-up PR:** - Wiring `/сервисы/api/object/{obj_id}/documents` в scrape loop (`domrf_kn.py`) - Celery-задача `download_domrf_documents` (replace stub)

lekss361 referenced this pull request

2026-05-17 15:27:26 +00:00

feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 #306

lekss361 referenced this pull request

2026-05-17 15:57:06 +00:00

feat(22k): object page — full exposure 9 blocks (sales/quartirography/metro/photos/docs/specs/checks) #310

lekss361 referenced this pull request from a commit

2026-05-17 19:35:08 +00:00

feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive

lekss361 referenced this pull request

2026-05-17 19:36:37 +00:00

feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive #322