feat(22c): domrf_kn_flat_plans new table + parser stub #308

Merged

lekss361 merged 2 commits from fix/22c-flat-plans-v2 into main

2026-05-17 15:39:34 +00:00

lekss361 commented

2026-05-17 15:36:16 +00:00

Owner

Summary

Issue #297 sub-task 22c (2d). NEW table domrf_kn_flat_plans + parser stub для plan images per-flat.

Payload audit (obj_id=65136)

Plan image URL ОТСУТСТВУЕТ в kn-API payload. Проверено:

kn_object_place_66 (2.3 MB): ILIKE %planImage%, %plan_image%, %planUrl%, %layoutImage%, %imageUrl% → все FALSE
portal/table flat payload не stored в domrf_raw_endpoints; поля planImageUrl/layoutUrl отсутствуют

Источник: только SSR-HTML страницы /сервисы/каталог-квартир/квартира/{hash} → нужен 22d catalog scraper (PR #307) для извлечения plan_image_url.

Changes

data/sql/114_22c_flat_plans.sql (renamed from 100 — collision с merged 100_fix_mv_layout_velocity):

CREATE TABLE domrf_kn_flat_plans (
  ods_id        text PRIMARY KEY,
  obj_id        bigint NOT NULL,
  plan_image_url text NOT NULL,
  local_path    text,
  width_px      int,
  height_px     int,
  size_bytes    int,
  downloaded_at timestamptz,
  snapshot_date date NOT NULL,
  scraped_at    timestamptz DEFAULT NOW()
);
CREATE INDEX domrf_kn_flat_plans_obj_idx ON (obj_id);

backend/app/services/scrapers/flat_plans.py (NEW, +241 lines):
- extract_flat_plans(raw_payload) -> list[dict] — parse plan image URLs (from catalog HTML when 22d wired)
- upsert_flat_plans(db, obj_id, plans) — UPSERT с COALESCE для local_path/downloaded_at
- download_plan_image_stub() — NotImplementedError placeholder (Celery task — follow-up PR)
Vault: fixes/Fix_BUG22c_FlatPlans_NewTable_May17.md

domrf_kn.py НЕ trog'нут.

Wiring TODO (separate PR)

Требует merge 22d catalog scraper (PR #307) для извлечения plan_image_url из SSR HTML. Затем wiring + Celery task для download images.

Test plan

Migration applies cleanly
Re-apply idempotent
After 22d merge: scrape catalog → extract plan_image_url → upsert + download

Related: issue #297 sub-task 22c · depends on 22d for source

## Summary Issue #297 sub-task **22c** (2d). NEW table `domrf_kn_flat_plans` + parser stub для plan images per-flat. ## Payload audit (obj_id=65136) Plan image URL **ОТСУТСТВУЕТ** в kn-API payload. Проверено: - `kn_object_place_66` (2.3 MB): ILIKE `%planImage%`, `%plan_image%`, `%planUrl%`, `%layoutImage%`, `%imageUrl%` → все `FALSE` - portal/table flat payload не stored в `domrf_raw_endpoints`; поля `planImageUrl/layoutUrl` отсутствуют **Источник**: только SSR-HTML страницы `/сервисы/каталог-квартир/квартира/{hash}` → нужен **22d catalog scraper** (PR #307) для извлечения plan_image_url. ## Changes - `data/sql/114_22c_flat_plans.sql` (renamed from 100 — collision с merged 100_fix_mv_layout_velocity): ```sql CREATE TABLE domrf_kn_flat_plans ( ods_id text PRIMARY KEY, obj_id bigint NOT NULL, plan_image_url text NOT NULL, local_path text, width_px int, height_px int, size_bytes int, downloaded_at timestamptz, snapshot_date date NOT NULL, scraped_at timestamptz DEFAULT NOW() ); CREATE INDEX domrf_kn_flat_plans_obj_idx ON (obj_id); ``` - `backend/app/services/scrapers/flat_plans.py` (NEW, +241 lines): - `extract_flat_plans(raw_payload) -> list[dict]` — parse plan image URLs (from catalog HTML when 22d wired) - `upsert_flat_plans(db, obj_id, plans)` — UPSERT с COALESCE для local_path/downloaded_at - `download_plan_image_stub()` — `NotImplementedError` placeholder (Celery task — follow-up PR) - Vault: `fixes/Fix_BUG22c_FlatPlans_NewTable_May17.md` `domrf_kn.py` НЕ trog'нут. ## Wiring TODO (separate PR) Требует merge 22d catalog scraper (PR #307) для извлечения `plan_image_url` из SSR HTML. Затем wiring + Celery task для download images. ## Test plan - [ ] Migration applies cleanly - [ ] Re-apply idempotent - [ ] After 22d merge: scrape catalog → extract plan_image_url → upsert + download Related: issue #297 sub-task 22c · depends on 22d for source

lekss361 added 2 commits 2026-05-17 15:36:16 +00:00

feat(22c): domrf_kn_flat_plans new table + parser stub 215441e4d8

- data/sql/100_22c_flat_plans.sql: CREATE TABLE domrf_kn_flat_plans
  (ods_id PK, obj_id, plan_image_url, local_path, dimensions, snapshot_date)
  + index on obj_id.

- backend/app/services/scrapers/flat_plans.py (NEW):
  * extract_flat_plans(raw_payload) — reserved stub; kn-API audit confirms
    plan images are NOT in portal/table payload, only in catalog SSR HTML.
    Function probes planImageUrl/layoutImageUrl/planUrl/layoutUrl/imageUrl
    for forward-compat if API ever adds the field.
  * upsert_flat_plans(db, obj_id, plans, snapshot_date) — UPSERT with
    COALESCE guard for local_path/downloaded_at/dimensions.
  * download_plan_image_stub() — NotImplementedError placeholder;
    real downloader is a future Celery task (#299 / 22d-track).

Audit (obj_id=65136): plan_image absent from kn_object_place_66 payload
(2.3 MB, checked planImage/planUrl/layoutImage/imageUrl — all FALSE).
Source is catalog flat page SSR only. Wiring domrf_kn.py — separate PR.

Issue #297 sub-task 22c.

fix(22c): rename migration 100 → 114 (collision avoid with merged 100_fix_mv_*) 7e73505b50

lekss361 merged commit b37e011aaa into main

2026-05-17 15:39:34 +00:00

lekss361 referenced this pull request from a commit

2026-05-17 15:39:34 +00:00

feat(22c): domrf_kn_flat_plans new table + parser stub (#308)

lekss361 commented

2026-05-17 15:39:56 +00:00

Author

Owner

Deep review APPROVE — merged squash (b37e011).

Что проверено:

data/sql/114_22c_flat_plans.sql — sequential после 113_22begh_*, BEGIN/COMMIT + IF NOT EXISTS идемпотентно, COMMENT'ы корректные. Nit: header still says "Migration 100" после rename — не блокер.
extract_flat_plans() — None-safe (or []), tries multiple возможных field names, debug log при пустом результате.
upsert_flat_plans() — SAVEPOINT per row через db.begin_nested(), named binds (:param, psycopg v3 OK), COALESCE на local_path/downloaded_at/width_px/height_px/size_bytes — скачанные артефакты не перезаписываются при re-scrape.
download_plan_image_stub() — NotImplementedError с descriptive message, без silent fallback.

Blast radius: zero — domrf_kn.py не trog'нут, no callers, новая таблица без FK.

Smoke после deploy:

\d domrf_kn_flat_plans
SELECT count(*) FROM domrf_kn_flat_plans;

Follow-up (после PR #309 catalog scraper merge):

wiring extract_flat_plans ← SSR-HTML parser
Celery task для download_plan_image_stub (issue #299)

Deep review APPROVE — merged squash (b37e011). Что проверено: - `data/sql/114_22c_flat_plans.sql` — sequential после `113_22begh_*`, `BEGIN/COMMIT` + `IF NOT EXISTS` идемпотентно, COMMENT'ы корректные. Nit: header still says "Migration 100" после rename — не блокер. - `extract_flat_plans()` — None-safe (`or []`), tries multiple возможных field names, debug log при пустом результате. - `upsert_flat_plans()` — SAVEPOINT per row через `db.begin_nested()`, named binds (`:param`, psycopg v3 OK), COALESCE на `local_path`/`downloaded_at`/`width_px`/`height_px`/`size_bytes` — скачанные артефакты не перезаписываются при re-scrape. - `download_plan_image_stub()` — `NotImplementedError` с descriptive message, без silent fallback. Blast radius: zero — `domrf_kn.py` не trog'нут, no callers, новая таблица без FK. Smoke после deploy: ``` \d domrf_kn_flat_plans SELECT count(*) FROM domrf_kn_flat_plans; ``` Follow-up (после PR #309 catalog scraper merge): - wiring `extract_flat_plans` ← SSR-HTML parser - Celery task для `download_plan_image_stub` (issue #299)