Three distinct SQL syntax errors caught from worker logs:
1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326
in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'.
Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc).
Likely introduced by ruff-format auto-comma on closing paren.
2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz`
psycopg double-colon trap → 'syntax error at or near ":"'.
Crashed legacy on-demand harvest_quarter called from analyze.
3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb`
same double-colon trap in admin settings UPDATE.
All converted to CAST(:x AS type) pattern (psycopg v3 safe).
40 tests still passing.
Co-authored-by: lekss361 <claudestars@proton.me>
cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD
occasionally returns Point geometry for parcels without detailed
boundary, causing INSERT failure:
psycopg.errors.InvalidParameterValue:
Geometry type (Point) does not match column type (Polygon)
This killed pilot v8 at quarter 66:41:0104002 — worker autoretry
exhausted, job hung at 25/50 (heartbeat stale).
Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit
(same pattern as upsert_quarter_geom_from_feature). Non-Polygon
geometry → geom=NULL, raw_props preserved.
Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions,
cad_enk, cad_oncs) unaffected — they accept any geometry type.
Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit).
Co-authored-by: lekss361 <claudestars@proton.me>
cadastre_jobs.phase_state is shared between ALL parallel Celery workers
of the same job. The early-exit triggered as soon as the FIRST worker
wrote phase=done via progress_cb — all subsequent workers (reading same
shared row) bailed without work.
Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter).
Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers
early-exited without doing snapshot phase.
Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped
because most workers never reached Phase 1/1.5.
Solution: delete the early-exit. Idempotency is guaranteed by
ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks
write same rows without duplicates.
Closes part of #168 (root-cause fix for race condition revealed by #182 metrics).
Co-authored-by: lekss361 <claudestars@proton.me>
scrape_cadastre.py previously incremented requests_count only by
grid_walk_requests, so snapshot-only quarters (and the entire ekb_full
job after PR #179 skips grid-walk on broken geom) reported 0 NSPD
requests — misleading for observability.
Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5
per-cat probes) and a total_requests property. Update Celery task to
write total_requests into cadastre_jobs.requests_count.
Tests added for the new property + Phase 1 snapshot_requests=1 assertion.
Co-authored-by: lekss361 <claudestars@proton.me>
72% of EKB quarters (1735/2408) have broken micro-precision geom in
cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS
(36381) feature but Phase 4 was only consuming stats.
Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms
3857 polygon to 4326 and writes only when existing geom is NULL or broken
(bbox width outside 100-10000m), verified via SQL-side sanity check.
Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON
(self-intersecting ring) rollbacks only geom UPDATE, preserves
upsert_quarter_stats.
Closes part of #168 (follow-up data quality fix for #179).
Co-authored-by: lekss361 <claudestars@proton.me>
Switch base search_by_quarter from thematicSearchId=1 to thematic=1
(includes ZOUIT in features). Add optional category_id param that routes
ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5,
ENK (39663) -> thematicSearchId=15.
Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS
category present in meta_counts (meta_total>0), run dedicated
per-category search call. Skips if category in GRID_WALK_CATS and
bbox is valid (grid-walk gives better coverage).
Curl-probe confirmed NSPD API caps at 20/category with no pagination;
categoryId param is server-ignored; thematic=1 strictly better than
thematicSearchId=1 (includes ZOUIT).
Closes part of #168 (ZOUIT/ENK coverage gap).
Co-authored-by: lekss361 <claudestars@proton.me>
* fix(cadastre): NSPD response parse — properties.category + top-level meta (#168)
* fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168)
Bot review of #177 flagged that fixtures still used legacy properties.categoryId
+ data.meta — meaning CI exercised only fallback branches, not primary paths
where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes
(#175 verify, #176 headers, #177 parse) confirmed need for test coverage.
Changes:
- test_nspd_bulk_client.py: sample_quarter_response → properties.category +
meta moved to top-level (real NSPD shape verified live)
- test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace)
- test_nspd_bulk_feature_parse_basic: primary path now exercised
Plus schema hardening per bot review:
- NSPDBulkFeature.category_id: `is not None` check (not truthy `or`)
to avoid edge case category=0; int() wrapped in try/except so
non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop.
35/35 tests pass locally.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
* fix(parcels): _parse_floors handle int (post-migration #169 schema change)
After PR #169 cad_buildings schema migration, `floors` column is INT
(was TEXT in legacy schema). Existing `_parse_floors(r.get("floors"))`
call in analyze_parcel → _neighbors_summary crashes with:
AttributeError: 'int' object has no attribute 'strip'
Fix: type union str | int | None. If int → return directly (no strip).
Preserve TEXT range parsing ("5-7" → 7) for backwards-compat with
any legacy data still in cad_buildings_old_apr26.
* test(smoke): production smoke tests for post-deploy regressions (#168)
Add tests/smoke/test_prod_smoke.py covering known regression surfaces.
Marks: prod_smoke + slow. Env: PROD_SMOKE_BASE_URL, PROD_SMOKE_ADMIN_TOKEN.
Run manually: cd backend && uv run pytest tests/smoke/ -m prod_smoke -v
* fix(cadastre): exact-match headers to nspd_lite to bypass NSPD WAF (#168)
Pilot v2 (job_id=2) failed 50/50 with HTTP 403 WAF block. Comparing
nspd_bulk_client.DEFAULT_HEADERS vs legacy nspd_lite.HEADERS (which works
on VPS IP since April 2026):
PascalCase keys → lowercase keys
Chrome/148 UA → Chrome/144 UA
No cache-control / pragma → "no-cache" both
accept-language ru first → en first
No origin → "https://nspd.gov.ru"
referer "/map" → "/map?thematic=PKK"
NSPD WAF (BotShield-class) likely fingerprints на header order + values
combined with TLS fingerprint. Matching legacy exactly minimizes deltas.
Test plan: retry pilot job after deploy, expect 0 WAF blocks for first
5 quarters.
* fix(test): exclude prod_smoke tests by default (#168)
CI ran tests/smoke/test_prod_smoke.py and they hit production
https://gendsgn.ru/api/v1/parcels/.../analyze which currently returns 500
(parse_floors regression — exactly what this PR fixes). Catch-22: PR can't
merge because smoke tests fail against pre-merge prod.
Fix: add `addopts = ["-m", "not prod_smoke"]` so default pytest excludes
them. Run manually post-deploy with: pytest -m prod_smoke -v
---------
Co-authored-by: lekss361 <claudestars@proton.me>
Pilot run #1 failed on all 50 quarters with:
NspdBulkError: [SSL: CERTIFICATE_VERIFY_FAILED] self-signed certificate
in certificate chain
NSPD prod chain on Beget VPS contains an internal/self-signed CA →
httpx default verify=True trips. Reuse pattern from legacy
app.services.scrapers.nspd_lite (uses ssl._create_unverified_context()
for same reason since April 2026).
Risk profile: NSPD public gov data, no sensitive payload outbound,
admin token не уходит к NSPD endpoints. MITM risk minimal.
Co-authored-by: lekss361 <claudestars@proton.me>
PR #173 fixed cad_buildings_pkey but missed 5 other indexes. Same root cause:
PG does NOT auto-rename indexes on table rename → all 6 backing indexes
remain on cad_buildings_old_apr26 with original names.
Add 5 more `ALTER INDEX IF EXISTS ... RENAME TO ..._old_apr26_*` statements
for: geom_gist, quarter_idx, objdoc_idx, complex_idx, purpose_idx.
Migration 92 still NOT in _schema_migrations (4× rollback now). Next deploy
will re-apply cleanly.
Co-authored-by: lekss361 <claudestars@proton.me>
Migration 92_cad_bulk_layers.sql fails on deploy with:
ERROR: relation "cad_buildings_pkey" already exists
CONTEXT: ALTER INDEX cad_buildings_new_pkey RENAME TO cad_buildings_pkey
Root cause: K.1 renamed cad_buildings → cad_buildings_old_apr26, but PG
does NOT auto-rename the backing PK index on table rename. The legacy
cad_buildings_pkey index is still attached to cad_buildings_old_apr26.
K.3 then tries to rename cad_buildings_new_pkey to the same name → collision.
Fix: add an explicit `ALTER INDEX IF EXISTS cad_buildings_pkey RENAME TO
cad_buildings_old_apr26_pkey` BEFORE the new rename. Idempotent (IF EXISTS).
Migration 92 was rolled back in last 3 deploy attempts (not in
_schema_migrations), so this edit will re-apply cleanly on next deploy.
Co-authored-by: lekss361 <claudestars@proton.me>
Add admin page for triggering and monitoring cadastre_jobs:
- Trigger form: pilot (50) / ekb_full (2408) / manual_list scopes
- Live jobs table 5s auto-refresh (TanStack Query)
- Per-row expand: phase_state JSON, req/s rate, WAF count, error
- Cancel button for queued/running, Resume for failed/cancelled/paused
- Heartbeat staleness: red "STALE" if running and gap > 10min
- WAF count highlighted red+⚠ when > 5
- Progress bar from targets_done/targets_total
- New tab in admin layout
- Token in localStorage same as geo page
Co-authored-by: lekss361 <claudestars@proton.me>
Bars rendered as ~1-15px because outer flex container had
align-items:flex-end which made each column shrink to label
height (~15.8px), so bar percentage heights resolved against
that instead of the intended 80px container.
Switched to align-items:stretch + fixed-pixel bar heights
anchored to a 60px area (reserves ~20px for label row). Each
column now uses justify-content:flex-end so bars + labels sit
at the bottom baseline.
Co-authored-by: lekss361 <claudestars@proton.me>
SQLAlchemy text() parser confuses ':window_interval' (named param) followed
by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb.
Result: psycopg видит literal ':window_interval' в SQL → syntax error →
exception caught by velocity catch → return None → UI shows null.
## Fix
':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест:
sales_rows query + _get_ekb_median percentile).
Pre-push code-reviewer должен был catch — добавим в feedback что для каждого
text() SQL grep ':[a-z]*::' before push.
Refs: PR #158 deploy verify, e2e velocity = null root cause
Co-authored-by: lekss361 <claudestars@proton.me>
Production POST /parcels/{cad}/analyze падает с 500:
sqlalchemy.exc.InternalError: (psycopg.errors.InFailedSqlTransaction)
current transaction is aborted, commands ignored until end of transaction block
## Root cause
`compute_velocity()` SQL fails (probably на cad's без conkurrentов / sparse
sale_graph data) → exception caught в try/except → НО db.rollback() отсутствует
→ transaction остаётся в aborted state.
Следующая query (_geotech_risk на line 828) пытается выполниться → крашится
с InFailedSqlTransaction.
## Fix
Wrap velocity в `with db.begin_nested()` — SAVEPOINT pattern (consistent
with PR #124 pzz_loader fix). Failure внутри savepoint:
- Rollbacks ТОЛЬКО savepoint
- Outer transaction остаётся clean
- Subsequent queries (_geotech_risk и пр.) работают
Pattern matches feedback_subagent_delegation audit recommendation для
in-loop / per-section exception handling.
## Impact
POST /parcels/{cad}/analyze больше не 500 при velocity failure. Возвращает
`velocity: null` + остальные fields normal.
Refs: user report 2026-05-15 InFailedSqlTransaction
Co-authored-by: lekss361 <claudestars@proton.me>
PR #151 deploy упал с 'POSTGRES_USER: unbound variable'.
source backend/.env — backend/.env содержит app vars (SCRAPE_ADMIN_TOKEN,
OPENROUTESERVICE_API_KEY и пр.), НО НЕ POSTGRES_USER/PASSWORD/DB.
Postgres credentials живут в /opt/gendesign/.env (root) — оттуда их читает
compose через env_file для postgres service. Это canonical location для
DB creds на этом стеке.
Fix: source .env (root) — corresponds к compose env_file resolution.
После merge → next deploy auto-apply применит pending #89/#91 (90 уже
applied via MCP).
Refs: #150, incident 2026-05-15 deploy #25901378924
Co-authored-by: lekss361 <claudestars@proton.me>
* fix(frontend): apiFetchWithStatus body stream double-read crash
Per user report 2026-05-14: 'Failed to execute text on Response: body stream
already read' при analyze с non-JSON error response.
apiFetchWithStatus делал .json() (consumes stream), потом .text() в catch
(FAILS — stream already consumed). Fetch API: body — ReadableStream,
consumable один раз.
Fix: read text() once → JSON.parse on string. На fail (HTML / non-JSON)
оборачиваем в {detail: rawText}. Никаких double-reads.
Site-finder cad search больше не крашится на error pages.
* feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150)
Закрывает root cause production 500 на /api/v1/admin/site-finder/weight-profiles
(а также 2 unapplied migrations: #89/#91).
## Why это нужно
После audit обнаружили что:
- backend/alembic/versions/ пустой (Alembic configured но не используется)
- deploy.yml не имеет migration step
- 47+ raw SQL files накопилось без auto-apply
- 3 файла unapplied: 87 (engineering), 89 (drop brin), 90 (weight profiles)
User report: GET weight-profiles → 500 потому что user_weight_profiles
table не существует на prod.
## Changes
### deploy.yml +39 lines
- Path trigger расширен на data/sql/*.sql
- New step 'Apply DB migrations' между compose pull и compose up -d:
- Idempotent CREATE TABLE _schema_migrations
- Loop по data/sql/*.sql sorted → skip applied → psql with ON_ERROR_STOP=on
- Record applied filename в tracking table
- Exit 1 на failure → containers НЕ обновляются (no partial state)
### NN collision fix
- 87_engineering_networks_191fz.sql → 91_engineering_networks_191fz.sql
(collision с 87_on_demand_indexes.sql, мой renumber 85→87 был неудачным)
### CLAUDE.md +32 lines
- Subsection 'Auto-apply' под Migration pattern
- Bootstrap procedure docs
- Idempotency requirement
- Failure recovery
### scripts/bootstrap_schema_migrations.sh +100 lines
- One-time seed _schema_migrations с 48 already-applied files
- 3 файла intentionally NOT seeded: 89/90/91 (auto-apply на first deploy)
- Usage docs + verification query
## Vault
domains/infra/Runbook_Auto_Apply_Migrations.md NEW
infra-MOC.md updated
## Required manual steps ДО merge
1. SSH gendesign (tunnel)
2. POSTGRES creds export
3. bash scripts/bootstrap_schema_migrations.sh — seed 48 already-applied files
4. Verify: SELECT COUNT(*) FROM _schema_migrations → 48
5. После merge → first deploy auto-apply #89/#90/#91 → 500 closes
Closes#150
---------
Co-authored-by: lekss361 <claudestars@proton.me>
Per user report 2026-05-14: 'Failed to execute text on Response: body stream
already read' при analyze с non-JSON error response.
apiFetchWithStatus делал .json() (consumes stream), потом .text() в catch
(FAILS — stream already consumed). Fetch API: body — ReadableStream,
consumable один раз.
Fix: read text() once → JSON.parse on string. На fail (HTML / non-JSON)
оборачиваем в {detail: rawText}. Никаких double-reads.
Site-finder cad search больше не крашится на error pages.
Co-authored-by: lekss361 <claudestars@proton.me>
Per code review audit (May 14): файл без transaction wrapper — backfill CTE
INSERTs + view recreations executed как individual auto-commit statements.
Mid-file crash → orphan rows в complexes/complex_sources с no rollback path.
## Fix
`BEGIN;` после header comments (line 31, перед section 1).
`COMMIT;` в конце файла после последнего COMMENT ON VIEW.
Wraps все 6 секций atomically:
1-2: CREATE TABLE + CREATE INDEX для complexes + complex_sources
3-5: 3 backfill CTE INSERTs с ON CONFLICT DO NOTHING
6: DROP VIEW IF EXISTS + CREATE VIEW + COMMENT для v_complex_full
## No CONCURRENTLY split needed
Проверил — все 10 indexes используют plain `CREATE INDEX IF NOT EXISTS` без
CONCURRENTLY keyword. Безопасно обернуть в одну tx.
## Idempotency preserved
Все existing guards остаются:
- CREATE TABLE/INDEX IF NOT EXISTS
- ON CONFLICT (source, source_id) DO NOTHING
- DROP VIEW IF EXISTS
Re-running file двукратно — safe.
## Vault
`fixes/Migration_73_Complexes_Atomic_May14.md` — created (status: resolved).
Co-authored-by: lekss361 <claudestars@proton.me>
Per code review audit (May 14): real SQL injection vector в `enqueue_geo_job`.
Manual `chr(39)*2` quote escape НЕ injection-safe — не защищает от backslash
escape, Unicode quote tricks. Если cad_num когда-либо придёт из user input —
exploitable.
## Change
`backend/app/workers/tasks/nspd_geo.py:298-310` — заменил f-string VALUES
concatenation на SQLAlchemy `text(...)` + list of dicts (psycopg v3 handles
batch executemany под капотом).
Pattern consistent с остальными INSERT'ами в этом же файле (_log,
_start_job, _save_quarter, _save_parcel, _save_building).
## Preserved
- ON CONFLICT (job_id, cad_num, thematic_id) DO NOTHING — verbatim
- Return value (int(job_id))
- Surrounding tx/commit/close logic
- DB schema, function signature, caller interface — unchanged
## Tests
- No test_nspd_geo.py существует — skipped
- ruff check passed
## Vault
`fixes/Bug_Nspd_Geo_Sql_Injection_May14.md` — created (status: resolved).
No backward compat / migration concerns — pure in-place query rewrite.
Co-authored-by: lekss361 <claudestars@proton.me>
Per code review audit (May 14): BRIN index на partitioned parent table dead
weight. PG16 НЕ propagates такие индексы на child partitions; pg_stat
показывает idx_scan=0 → никогда не использовался.
## Why dead
rosreestr_deals — yearly partitioned по period_start_date. Queries делают
partition pruning через partition key. BRIN index на parent существует, но:
1. На child partitions его нет (PG16 не reflects)
2. Parent сам не имеет данных (partitioned out)
3. Queries никогда не trigger BRIN scan
Maintenance overhead (storage, autoanalyze) без value.
## Migration
`data/sql/89_drop_dead_brin_rosreestr_deals.sql` — атомарный DROP с idempotent
`IF EXISTS`. Apply order: после 88_*.
## Cleanup
`data/sql/01_schema_rosreestr_deals.sql:65-67` — закомментировал оригинальное
CREATE с note: fresh DB bootstrap НЕ создаст index заново.
## Rollback path
Если в будущем нужен range scan optimization — re-create per-partition:
```sql
CREATE INDEX rosreestr_deals_2024q1_period_brin
ON rosreestr_deals_2024q1 USING BRIN (period_start_date);
-- per-partition × N partitions
```
Но обычно partition pruning достаточно. Измерь EXPLAIN перед re-creation.
## Vault
`fixes/Migration_Drop_Dead_Brin_May14.md` создан (status: ready_to_apply).
Apply на prod: `psql ... -f data/sql/89_drop_dead_brin_rosreestr_deals.sql`
после merge.
Co-authored-by: lekss361 <claudestars@proton.me>
Forward-fix после incident May 14 (см. vault events/Incident_Prod_Down_May14_2026).
PR #117 был полностью reverted PR #119 (Caddy depends_on frontend: service_healthy
+ failing wget --spider / → Caddy не стартовал → vsё прод down ~25 мин).
Этот PR делает то же что планировал #117, но корректно:
### Frontend healthcheck — TCP probe через node (НЕ HTTP)
```yaml
test: ["CMD", "node", "-e",
"require('net').createConnection({port:3000,host:'127.0.0.1'})
.once('connect',function(){process.exit(0)})
.once('error',function(){process.exit(1)})"]
start_period: 60s
retries: 6
interval: 15s
```
- НЕ зависит от HTTP status (200/404/308 — все OK если порт слушает)
- `node` гарантированно в alpine image (запускает server.js)
- `127.0.0.1` — никакой DNS ambiguity (busybox alpine localhost IPv6 vs IPv4)
- 60s + 6×15s = 150s window — с запасом на Next.js standalone cold start
### Caddy depends_on — backend healthy, frontend started (НЕ healthy)
```yaml
caddy:
depends_on:
backend: { condition: service_healthy } # backend имеет /health endpoint
frontend: { condition: service_started } # safety net — НЕ блокирует Caddy
```
Lesson from incident: frontend health flaky (Next.js cold start variable).
Strict service_healthy для frontend → одна misconfig = full outage обоих доменов
(gendsgn.ru + obsidian.gendsgn.ru через тот же Caddy на shared network).
### backend/worker/beat redis: service_started → service_healthy
Redis уже имеет `redis-cli ping` healthcheck. Tightening безопасно — был
artifact pre-healthcheck Redis.
### Pin rosreestr2coord>=5.0.0 (lock уже 5.3.3)
v5 убрала delay параметр; код адаптирован (rate limit на worker уровне).
Защита от silent downgrade при `uv lock --upgrade`.
### Что НЕ повторено из #117
- Caddy depends_on frontend: service_healthy — заменено на service_started
- wget --spider / — заменено на node TCP probe
### Vault обновлён
- `events/Incident_Prod_Down_May14_2026.md` — постмортем + timeline + lessons
- Update Changelog в `domains/infra/infra-MOC.md` (если есть)
Closes incident. Re-applies #117 goals safely.
Co-authored-by: lekss361 <claudestars@proton.me>
