feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ #335

Merged

lekss361 merged 1 commit from feat/22d-catalog-object-scraper into main

2026-05-17 21:26:23 +00:00

lekss361 commented

2026-05-17 21:16:34 +00:00

Owner

Summary

kn-API не возвращает wall_type, energy_eff, ceiling_height_m, parking_*, playground_*, finishing_variants_count, has_*, scores — все эти поля доступны только на SSR catalog page /сервисы/каталог-новостроек/объект/{id} через __NEXT_DATA__ JSON блок (Next.js).

Новый scraper:

Fetch HTML через BrowserSession (WAF bypass через Playwright, тот же pattern что domrf_catalog.py)
Extract __NEXT_DATA__ JSON (regex + json.loads)
Parse 25+ полей из pageProps / pageProps.additionalInfo / pageProps.quartography / pageProps.indexes
UPDATE domrf_kn_objects SET field = COALESCE(:field, field) — НЕ перетирает kn-API данные
Celery task с atomic SELECT (obj_id + snapshot_date одним запросом — без race с kn-scraper)
Beat schedule: Tuesday 04:00 UTC, batch 300/run

Mapping (sample obj 65136)

DB column	`__NEXT_DATA__` path	Sample
`wall_type`	`pageProps.wallMaterial`	"Монолит-кирпич"
`energy_eff`	`pageProps.objEnergyEfficiency`	"B"
`ceiling_height_m`	`additionalInfo.ceilingHeight` "2,7"	2.7
`parking_total_slots`	`pageProps.parkingCount`	246
`parking_provision_pct`	`additionalInfo.parkingAvailabilityPerc`	60.5
`elevators_cargo_count`	`cargoElevatorsCount + cargoPassengerElevatorCount`	4
`playground_kids_count`	`additionalInfo.playgroundsCount`	6
... 18 more fields

Files

backend/app/services/scrapers/domrf_catalog_object.py (NEW, ~290 lines)
backend/app/workers/tasks/scrape_kn_catalog_objects.py (NEW, ~110 lines)
backend/app/workers/beat_schedule.py (MODIFIED, +10 lines)
data/sql/118_22d_catalog_object_scraped_at.sql (NEW) — catalog_scraped_at TIMESTAMP + index
backend/tests/services/scrapers/test_domrf_catalog_object.py (NEW, 41 tests pass)

Not in this PR

obj_checks 6 чекбоксов "Проверено на наш.дом.рф" — на странице только общий isChecked flag, 6 detailed checks требуют отдельной investigation.

Test plan

41/41 unit tests pass (pytest tests/services/scrapers/test_domrf_catalog_object.py)
Ruff clean
Pre-push code review (🟡 MEDIUM → fixups applied → re-verified clean)
Migration 118 auto-applies via deploy.yml
Post-deploy: celery -A app.workers call scrape_kn_catalog_objects --kwargs='{"max_objects": 3}' smoke test
Verify SELECT obj_id, catalog_scraped_at, wall_type, energy_eff, ceiling_height_m FROM domrf_kn_objects WHERE catalog_scraped_at IS NOT NULL LIMIT 5 returns data
Beat schedule fires next Tuesday 04:00 UTC

Reviewer fixups applied (pre-push, 🟡 MEDIUM)

M1: parking_provision_pct _safe_int → _to_numeric_comma (column is numeric(5,1), 60.5 was truncated)
M3: Atomic SELECT возвращает (obj_id, snapshot_date) одним запросом — устраняет race с kn-scraper между двумя SELECT'ами
M2: Concurrency rationale в docstring (no Redis lock, consistent с sibling tasks)
N2: avg_flat_area_m2 defensive через _to_numeric_comma

## Summary kn-API не возвращает `wall_type`, `energy_eff`, `ceiling_height_m`, `parking_*`, `playground_*`, `finishing_variants_count`, `has_*`, scores — все эти поля доступны только на SSR catalog page `/сервисы/каталог-новостроек/объект/{id}` через `__NEXT_DATA__` JSON блок (Next.js). Новый scraper: - Fetch HTML через `BrowserSession` (WAF bypass через Playwright, тот же pattern что `domrf_catalog.py`) - Extract `__NEXT_DATA__` JSON (regex + json.loads) - Parse 25+ полей из `pageProps` / `pageProps.additionalInfo` / `pageProps.quartography` / `pageProps.indexes` - `UPDATE domrf_kn_objects SET field = COALESCE(:field, field)` — НЕ перетирает kn-API данные - Celery task с atomic SELECT (`obj_id + snapshot_date` одним запросом — без race с kn-scraper) - Beat schedule: Tuesday 04:00 UTC, batch 300/run ### Mapping (sample obj 65136) | DB column | `__NEXT_DATA__` path | Sample | |---|---|---| | `wall_type` | `pageProps.wallMaterial` | "Монолит-кирпич" | | `energy_eff` | `pageProps.objEnergyEfficiency` | "B" | | `ceiling_height_m` | `additionalInfo.ceilingHeight` "2,7" | 2.7 | | `parking_total_slots` | `pageProps.parkingCount` | 246 | | `parking_provision_pct` | `additionalInfo.parkingAvailabilityPerc` | 60.5 | | `elevators_cargo_count` | `cargoElevatorsCount + cargoPassengerElevatorCount` | 4 | | `playground_kids_count` | `additionalInfo.playgroundsCount` | 6 | | ... 18 more fields | | | ### Files - `backend/app/services/scrapers/domrf_catalog_object.py` (NEW, ~290 lines) - `backend/app/workers/tasks/scrape_kn_catalog_objects.py` (NEW, ~110 lines) - `backend/app/workers/beat_schedule.py` (MODIFIED, +10 lines) - `data/sql/118_22d_catalog_object_scraped_at.sql` (NEW) — `catalog_scraped_at TIMESTAMP` + index - `backend/tests/services/scrapers/test_domrf_catalog_object.py` (NEW, 41 tests pass) ### Not in this PR - `obj_checks` 6 чекбоксов "Проверено на наш.дом.рф" — на странице только общий `isChecked` flag, 6 detailed checks требуют отдельной investigation. ## Test plan - [x] 41/41 unit tests pass (`pytest tests/services/scrapers/test_domrf_catalog_object.py`) - [x] Ruff clean - [x] Pre-push code review (🟡 MEDIUM → fixups applied → re-verified clean) - [ ] Migration 118 auto-applies via deploy.yml - [ ] Post-deploy: `celery -A app.workers call scrape_kn_catalog_objects --kwargs='{"max_objects": 3}'` smoke test - [ ] Verify `SELECT obj_id, catalog_scraped_at, wall_type, energy_eff, ceiling_height_m FROM domrf_kn_objects WHERE catalog_scraped_at IS NOT NULL LIMIT 5` returns data - [ ] Beat schedule fires next Tuesday 04:00 UTC ## Reviewer fixups applied (pre-push, 🟡 MEDIUM) - M1: `parking_provision_pct` `_safe_int` → `_to_numeric_comma` (column is `numeric(5,1)`, 60.5 was truncated) - M3: Atomic SELECT возвращает `(obj_id, snapshot_date)` одним запросом — устраняет race с kn-scraper между двумя SELECT'ами - M2: Concurrency rationale в docstring (no Redis lock, consistent с sibling tasks) - N2: `avg_flat_area_m2` defensive через `_to_numeric_comma`

lekss361 added 1 commit 2026-05-17 21:16:35 +00:00

feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ ef3e0404a6

kn-API не возвращает wall_type, energy_eff, ceiling_height_m, parking_*,
playground_*, finishing_variants_count, has_*, scores и др. Эти поля доступны
только на SSR catalog странице /сервисы/каталог-новостроек/объект/{id} в
__NEXT_DATA__ JSON блоке (Next.js).

- backend/app/services/scrapers/domrf_catalog_object.py — fetch HTML через
  BrowserSession (WAF bypass), extract __NEXT_DATA__, parse 25+ fields, COALESCE
  UPDATE (не перетирает kn-API данные)
- backend/app/workers/tasks/scrape_kn_catalog_objects.py — Celery task, atomic
  SELECT (obj_id + snapshot_date одним запросом, без race с kn-scraper)
- backend/app/workers/beat_schedule.py — Tuesday 04:00 UTC, batch 300/run
- data/sql/118_22d_catalog_object_scraped_at.sql — catalog_scraped_at TIMESTAMP
  + index NULLS FIRST для "missing/stale first" selection
- backend/tests/services/scrapers/test_domrf_catalog_object.py — 41 tests pass

obj_checks 6 чекбоксов "Проверено на наш.дом.рф" — отдельная investigation
(на странице только общий isChecked flag).

lekss361 commented

2026-05-17 21:26:09 +00:00

Author

Owner

Deep Code Review — verdict APPROVE

Summary

Status: APPROVE (no blocking, no high, no medium issues)
Files reviewed: 6 (P0: 1 migration, P1: 3 scraper/task/beat, P3: 2 tests)
Lines: +931 / -0
Head SHA: ef3e0404 · mergeable=true · no existing reviews

Migration 118 collision — SAFE

118_22d_catalog_object_scraped_at.sql → ALTER TABLE domrf_kn_objects ADD COLUMN IF NOT EXISTS catalog_scraped_at TIMESTAMP + CREATE INDEX IF NOT EXISTS
118_pilot_requests.sql (already merged on main) → CREATE TABLE IF NOT EXISTS pilot_requests (lead-gen, issue #307)
Different target tables, both IF [NOT] EXISTS idempotent. _schema_migrations PK = full filename. Lex order: 118_22d… runs first, then 118_pilot…. Replays no-op.
ALTER on existing column-list of domrf_kn_objects — ADD COLUMN без NOT NULL DEFAULT → metadata-only operation в Postgres 11+, не блокирует таблицу даже на больших объёмах. CREATE INDEX без CONCURRENTLY: таблица 1.5k objects per region (per beat schedule batch 300/run → 1532 объекта за 5 нед) — index build instant, locking risk нулевой.

Scraper robustness — verified

Regex __NEXT_DATA__: anchored literal prefix + lazy (.+?) + literal closing tag, re.DOTALL. Tested locally на 1MB no-match body — 0.65 ms; 50k script-tag haystack — 2.56 ms. No catastrophic backtracking (нет nested quantifier / alternation overlap).
Graceful skip: __NEXT_DATA__ missing → ValueError → caught в scrape_catalog_object → logger.warning + return False → batch продолжается. WAF block / 404 / fetch err — тот же pattern, никаких uncaught crashes на partial pages.
COALESCE(:field, field) — preserves kn-API данные при NULL parse result. Идемпотентно при re-scrape.
CAST trap clean: SQL не использует :x::type ни разу, все binds — обычные :name, типизация на стороне Python (_safe_int, _to_numeric_comma, _to_date_ddmmyyyy).
SAVEPOINT per row (with db.begin_nested():) — соответствует .claude/rules/backend.md, изолирует per-row failures.
Atomic SELECT (M3 fix) — (obj_id, snapshot_date) одним query с snapshot_date = (SELECT MAX … WHERE region_cd = :region_code) — устраняет race с kn-scraper между двумя запросами. Грамотно ограничено тем же region_cd.
BrowserSession private attrs (_page, _sem, _request_count) — consistent с domrf_catalog.py:99 (existing prod pattern).

Celery task

time_limit=3600 — adequate (batch 300 × ~1.2s jitter = ~6 min realistic).
Concurrency rationale в docstring (no Redis lock, OK для Tuesday-only beat без overlap).
asyncio.run() в sync Celery task — correct sync↔async bridge per project convention.
Все exceptions либо logger.error + raise, либо logger.warning + skip — нет silent swallow.

Beat schedule

0 4 * * 2 (Tue 04:00 UTC) — slot не конфликтует с другими weekly tasks (poi/noise — Monday, ekburg — monthly 1st, pzz — monthly 1st, nspd — Monday 01:00).

Tests — substantive

15 def test_* + 30 parametrize cases = 41 tests (math confirms author's claim).
Coverage: regex extraction (single/double quotes, missing tag, broken JSON), full sample (obj 65136 mapping), partial response (только buildingClass), empty response (no crash), edge cases для всех 4 value helpers, parking_provision_pct=60.5 float preservation (M1 regression test), freePlan='Да', first_floor=жилой/нежилой, elevators_cargo_count = cargoElevatorsCount + cargoPassengerElevatorCount sum logic.
Никаких trivial assert True — реальные ассерты с expected values.

Convention compliance

✅ Никаких import psycopg2, import requests, print(), hardcoded credentials, f-string SQL.
✅ psycopg v3 patterns: bind params + text(...) + executemany style.
✅ Cyrillic URL paths (сервисы/каталог-новостроек/объект/{obj_id}) — Python str + Playwright fetch handles UTF-8 natively, тот же pattern что domrf_catalog.py.
✅ Vault entries уже созданы: fixes/Fix_BUG22d_CatalogScraper_May17.md + domains/domrf/scraper/domrf-scraper-MOC.md (per project rule #6).

Pre-flight

Feature branch feat/22d-catalog-object-scraper, не main ✅
mergeable=true, no conflicts ✅
Head SHA unchanged с моего scan ✅
Pre-push 🟡 MEDIUM fixups (M1/M2/M3/N2) applied и видны в diff ✅

Risk / reversibility

Risk: Low. ALTER ADD COLUMN nullable + INDEX — non-blocking. UPDATE использует COALESCE — старые данные не перетираются. Beat задача disabled-by-default нет, но запустится только в Tue 04:00 UTC.
Reversibility: Easy. Drop column + revert code если потребуется.
Merge window: anytime.

Мержу через merge_pull_request (squash).

## Deep Code Review — verdict APPROVE ### Summary - **Status**: APPROVE (no blocking, no high, no medium issues) - **Files reviewed**: 6 (P0: 1 migration, P1: 3 scraper/task/beat, P3: 2 tests) - **Lines**: +931 / -0 - **Head SHA**: ef3e0404 · mergeable=true · no existing reviews ### Migration 118 collision — SAFE - `118_22d_catalog_object_scraped_at.sql` → `ALTER TABLE domrf_kn_objects ADD COLUMN IF NOT EXISTS catalog_scraped_at TIMESTAMP` + `CREATE INDEX IF NOT EXISTS` - `118_pilot_requests.sql` (already merged on main) → `CREATE TABLE IF NOT EXISTS pilot_requests` (lead-gen, issue #307) - Different target tables, both `IF [NOT] EXISTS` idempotent. `_schema_migrations` PK = full filename. Lex order: `118_22d…` runs first, then `118_pilot…`. Replays no-op. - ALTER on existing column-list of `domrf_kn_objects` — `ADD COLUMN` без `NOT NULL DEFAULT` → metadata-only operation в Postgres 11+, не блокирует таблицу даже на больших объёмах. CREATE INDEX без CONCURRENTLY: таблица 1.5k objects per region (per beat schedule `batch 300/run → 1532 объекта за 5 нед`) — index build instant, locking risk нулевой. ### Scraper robustness — verified - **Regex `__NEXT_DATA__`**: anchored literal prefix + lazy `(.+?)` + literal closing tag, `re.DOTALL`. Tested locally на 1MB no-match body — 0.65 ms; 50k script-tag haystack — 2.56 ms. No catastrophic backtracking (нет nested quantifier / alternation overlap). - **Graceful skip**: `__NEXT_DATA__` missing → `ValueError` → caught в `scrape_catalog_object` → `logger.warning` + `return False` → batch продолжается. WAF block / 404 / fetch err — тот же pattern, никаких uncaught crashes на partial pages. - **`COALESCE(:field, field)`** — preserves kn-API данные при NULL parse result. Идемпотентно при re-scrape. - **CAST trap clean**: SQL не использует `:x::type` ни разу, все binds — обычные `:name`, типизация на стороне Python (`_safe_int`, `_to_numeric_comma`, `_to_date_ddmmyyyy`). - **SAVEPOINT per row** (`with db.begin_nested():`) — соответствует `.claude/rules/backend.md`, изолирует per-row failures. - **Atomic SELECT (M3 fix)** — `(obj_id, snapshot_date)` одним query с `snapshot_date = (SELECT MAX … WHERE region_cd = :region_code)` — устраняет race с kn-scraper между двумя запросами. Грамотно ограничено тем же `region_cd`. - **BrowserSession private attrs** (`_page`, `_sem`, `_request_count`) — consistent с `domrf_catalog.py:99` (existing prod pattern). ### Celery task - `time_limit=3600` — adequate (batch 300 × ~1.2s jitter = ~6 min realistic). - Concurrency rationale в docstring (no Redis lock, OK для Tuesday-only beat без overlap). - `asyncio.run()` в sync Celery task — correct sync↔async bridge per project convention. - Все exceptions либо `logger.error` + `raise`, либо `logger.warning` + skip — нет silent swallow. ### Beat schedule - `0 4 * * 2` (Tue 04:00 UTC) — slot не конфликтует с другими weekly tasks (poi/noise — Monday, ekburg — monthly 1st, pzz — monthly 1st, nspd — Monday 01:00). ### Tests — substantive - 15 `def test_*` + 30 parametrize cases = 41 tests (math confirms author's claim). - Coverage: regex extraction (single/double quotes, missing tag, broken JSON), full sample (obj 65136 mapping), partial response (только `buildingClass`), empty response (no crash), edge cases для всех 4 value helpers, `parking_provision_pct=60.5` float preservation (M1 regression test), `freePlan='Да'`, `first_floor=жилой/нежилой`, `elevators_cargo_count = cargoElevatorsCount + cargoPassengerElevatorCount` sum logic. - Никаких trivial `assert True` — реальные ассерты с expected values. ### Convention compliance - ✅ Никаких `import psycopg2`, `import requests`, `print()`, hardcoded credentials, f-string SQL. - ✅ `psycopg v3` patterns: bind params + `text(...)` + `executemany` style. - ✅ Cyrillic URL paths (`сервисы/каталог-новостроек/объект/{obj_id}`) — Python str + Playwright fetch handles UTF-8 natively, тот же pattern что `domrf_catalog.py`. - ✅ Vault entries уже созданы: `fixes/Fix_BUG22d_CatalogScraper_May17.md` + `domains/domrf/scraper/domrf-scraper-MOC.md` (per project rule #6). ### Pre-flight - Feature branch `feat/22d-catalog-object-scraper`, не main ✅ - `mergeable=true`, no conflicts ✅ - Head SHA unchanged с моего scan ✅ - Pre-push 🟡 MEDIUM fixups (M1/M2/M3/N2) applied и видны в diff ✅ ### Risk / reversibility - **Risk**: Low. ALTER ADD COLUMN nullable + INDEX — non-blocking. UPDATE использует COALESCE — старые данные не перетираются. Beat задача disabled-by-default нет, но запустится только в Tue 04:00 UTC. - **Reversibility**: Easy. Drop column + revert code если потребуется. - **Merge window**: anytime. Мержу через `merge_pull_request` (squash).