lekss361/gendesign

Fork 0

test(infra): integration phantom column gate (#197) #216

Merged

lekss361 merged 1 commit from feat/197-test-infra-gate into main

2026-05-16 13:07:14 +00:00

lekss361 commented

2026-05-16 12:59:41 +00:00

Owner

Summary

Pytest integration gate, prevents phantom column bugs (как geom_3857 PR #196 + units_count PR #213, оба пойманные вручную на code review)
12 EXPLAIN-тестов (READONLY) против real Postgres schema через SSH-tunneled localhost:15432
Fixture phantom_check_session: создаёт временную schema test_phantom_<random> через pg_dump --schema-only, гарантированный teardown DROP SCHEMA CASCADE (schema_created flag tracking — fix code-reviewer concern)
Fallback: если pg_dump недоступен — work in public (EXPLAIN safe)
Skip mechanism: pytest.mark.skipif(not TEST_DATABASE_URL) — без env var все 12 SKIPPED

Coverage (12 queries)

#	Test	Table	Detects historic phantom
1-3	competitors / velocity	`domrf_kn_objects`	`geom_3857` PR #196
4-6	recent_permits RNS/RVE	`ekburg_construction_permits`	`units_count` PR #213
7	velocity room bucket	`mv_layout_velocity`	(future)
8-9	supply / avg_price	`domrf_kn_flats`	(future)
10-11	objective mapping	`objective_*`	(future)
12	parcel centroid	`cad_*_geom`	(future)

Files

backend/tests/integration/__init__.py (NEW, empty)
backend/tests/integration/conftest.py (+155) — fixture + skipif marker
backend/tests/integration/test_phantom_columns.py (+379) — 12 tests
backend/pyproject.toml (+1) — pytest marker integration

Test plan

Locally with SSH tunnel:

ssh -N gendesign &   # localhost:15432
export TEST_DATABASE_URL="postgresql+psycopg://USER:PASS@localhost:15432/DB"
cd backend && uv run pytest tests/integration/ -v
# Expected: 12/12 PASS (если все queries clean — иначе fail с phantom column name)

Без TEST_DATABASE_URL: все 12 SKIPPED (pytest tests/integration/)
Demo fail: temporarily change total_area_sqm → units_count в test#4 → expect: column "units_count" does not exist

Pre-push checks

ruff + ruff-format: pass
code-reviewer subagent: APPROVE (after schema_created flag fix)
Schema verification против live DB: 100% column references real
EXPLAIN (не EXPLAIN ANALYZE) — никаких side effects
No hardcoded creds, no INSERT/UPDATE/DELETE/TRUNCATE/DROP TABLE

Follow-up

CI integration в Forgejo Actions (SSH tunnel из runner) — отдельный issue
Phase 2: больше queries (когда добавим новые critical SQL paths)

Closes #197

## Summary - Pytest integration gate, prevents phantom column bugs (как `geom_3857` PR #196 + `units_count` PR #213, оба пойманные вручную на code review) - 12 `EXPLAIN`-тестов (READONLY) против real Postgres schema через SSH-tunneled `localhost:15432` - Fixture `phantom_check_session`: создаёт временную schema `test_phantom_<random>` через `pg_dump --schema-only`, гарантированный teardown `DROP SCHEMA CASCADE` (schema_created flag tracking — fix code-reviewer concern) - Fallback: если `pg_dump` недоступен — work in `public` (EXPLAIN safe) - Skip mechanism: `pytest.mark.skipif(not TEST_DATABASE_URL)` — без env var все 12 SKIPPED ## Coverage (12 queries) | # | Test | Table | Detects historic phantom | |---|---|---|---| | 1-3 | competitors / velocity | `domrf_kn_objects` | **`geom_3857` PR #196** | | 4-6 | recent_permits RNS/RVE | `ekburg_construction_permits` | **`units_count` PR #213** | | 7 | velocity room bucket | `mv_layout_velocity` | (future) | | 8-9 | supply / avg_price | `domrf_kn_flats` | (future) | | 10-11 | objective mapping | `objective_*` | (future) | | 12 | parcel centroid | `cad_*_geom` | (future) | ## Files - `backend/tests/integration/__init__.py` (NEW, empty) - `backend/tests/integration/conftest.py` (+155) — fixture + skipif marker - `backend/tests/integration/test_phantom_columns.py` (+379) — 12 tests - `backend/pyproject.toml` (+1) — pytest marker `integration` ## Test plan - [ ] Locally with SSH tunnel: ```bash ssh -N gendesign & # localhost:15432 export TEST_DATABASE_URL="postgresql+psycopg://USER:PASS@localhost:15432/DB" cd backend && uv run pytest tests/integration/ -v # Expected: 12/12 PASS (если все queries clean — иначе fail с phantom column name) ``` - [ ] Без TEST_DATABASE_URL: все 12 SKIPPED (`pytest tests/integration/`) - [ ] Demo fail: temporarily change `total_area_sqm` → `units_count` в test#4 → expect: `column "units_count" does not exist` ## Pre-push checks - ruff + ruff-format: pass - code-reviewer subagent: APPROVE (after schema_created flag fix) - Schema verification против live DB: 100% column references real - EXPLAIN (не EXPLAIN ANALYZE) — никаких side effects - No hardcoded creds, no INSERT/UPDATE/DELETE/TRUNCATE/DROP TABLE ## Follow-up - CI integration в Forgejo Actions (SSH tunnel из runner) — отдельный issue - Phase 2: больше queries (когда добавим новые critical SQL paths) Closes #197

lekss361 added 1 commit 2026-05-16 12:59:41 +00:00

test(infra): integration phantom column gate (#197 ) 133379235c

Add pytest integration tests that run EXPLAIN against real Postgres schema
to detect phantom column/table references before PR merge. 12 EXPLAIN test
cases covering domrf_kn_objects, ekburg_construction_permits,
mv_layout_velocity, domrf_kn_flats, objective_* and cad geo tables.
Tests skip without TEST_DATABASE_URL; CI integration is a follow-up.

Pre-empts phantom column bugs caught manually on PR #196 (geom_3857)
and PR #213 (units_count).

Closes #197

lekss361 referenced this pull request

2026-05-16 12:59:43 +00:00

test-infra: add real-DB integration test gate for site_finder services #197

lekss361 commented

2026-05-16 13:07:02 +00:00

Author

Owner

Deep Code Review — verdict

Summary

Status: APPROVE
Head SHA: 1333792 · Mergeable: true · Auto-merge scope: ALLOWED (tests-only)
Files reviewed: 4 · Lines: +533 / -0

Schema verification

All 12 test queries verified against real DB. Zero phantom columns found.

domrf_kn_objects — all columns confirmed real:

obj_id (bigint), latitude/longitude (numeric), comm_name, dev_name, obj_class, flat_count, site_status, snapshot_date, district_name, region_cd — all exist
geom_3857 — correctly not referenced (was PR #196 phantom)

ekburg_construction_permits — all columns confirmed real:

permit_type, permit_number, issue_date, expiry_date, developer_name, developer_inn — exist
object_name, object_type, construction_address, cadastral_number — exist
total_area_sqm, living_area_sqm, living_area_fact_sqm, rve_number, rve_date — exist
units_count — correctly not referenced (was PR #213 phantom)

mv_layout_velocity (matview, confirmed via pg_attribute):

obj_id, room_bucket, total_deals_24mo, avg_area_m2, avg_price_thousand_rub_per_m2, window_start, window_end — all real

domrf_kn_flats — obj_id, total_area, rooms, is_studio, flat_type, price_per_m2, status, snapshot_date — all real

objective_complex_mapping — domrf_obj_id, objective_complex_name — real

objective_corpus_room_month — deals_total_vol_m2, deals_total_count, report_month, project_name — real

cad_parcels_geom (VIEW) — cad_num, geom — real

cad_quarters_geom — cad_number, geom — real

Note: m.obj_id in test_objective_corpus_room_month_columns resolves from derived subquery alias (cm.domrf_obj_id AS obj_id), not from objective_corpus_room_month directly. Correct SQL.

Fixture safety

Teardown on failure: YES. try/finally block (lines 141-153) guarantees db.close() and DROP SCHEMA CASCADE run even on test exceptions.
schema_created flag: YES (line 113). Set only after CREATE SCHEMA IF NOT EXISTS succeeds (line 119). Teardown gates on flag (line 147) — DROP never attempted if CREATE SCHEMA failed. No accidental DROP SCHEMA public CASCADE possible.
pg_dump fallback: YES. FileNotFoundError, nonzero returncode, timeout — all handled gracefully, falls back to operating directly in public schema. EXPLAIN-only tests are read-only so this is safe.
PostGIS handling: pg_dump --schema-only correctly dumps geometry column metadata and GIST index definitions. --no-owner --no-acl prevent permission errors on restore.
Schema name injection: _random_suffix() uses only ascii_lowercase + digits (alphanumeric). Schema name double-quoted in all executions (lines 117, 137, 150). No user-controlled input reaches the name.

DML / side-effect check

No INSERT, UPDATE, DELETE, TRUNCATE, DROP TABLE, ALTER TABLE, CREATE INDEX in any test query. Grep confirmed clean.
All test queries use EXPLAIN only (not EXPLAIN ANALYZE). No query execution, no side effects.
Fixture DDL is limited to CREATE SCHEMA IF NOT EXISTS + DROP SCHEMA IF EXISTS ... CASCADE on temp schema only.

Credentials check

No hardcoded passwords, tokens, or secrets. Credentials flow exclusively through os.environ.get("TEST_DATABASE_URL", "") (line 31).

pyproject.toml change

One line only: integration marker appended to [tool.pytest.ini_options].markers (line 86). No other changes.

Skip logic

requires_test_db = pytest.mark.skipif(not _TEST_DATABASE_URL, ...) at module level in conftest.py (line 36). Applied as pytestmark = [...] at module level in test file (lines 32-35). All 12 tests inherit it and skip cleanly when TEST_DATABASE_URL absent. Additional pytest.skip() guard inside fixture body (line 96) as belt-and-suspenders.

Cross-file impact

No modifications to existing test files. No backend/tests/conftest.py at root tests level (does not exist). New files only. No fixture name collisions.

Minor observations (non-blocking)

conftest.py line 63: db_url.replace("+psycopg", "") for pg_dump URL is fragile — would silently fail if dialect string changes. Failure path is handled gracefully (fallback), so non-blocking.
session scope: both fixtures are session-scoped. EXPLAIN-only tests make this safe today.
f.status = 'sold' in test_avg_price_sold_query: no CHECK constraint on domrf_kn_flats.status, so EXPLAIN passes regardless of value validity. Acceptable for phantom-column detection.

Recommended next steps

No blocking issues. Merge.

Optional follow-up (separate PR): replace db_url.replace with proper URL parsing for pg_dump invocation.

Auto-merge policy

ALLOWED — all changes under backend/tests/** and backend/pyproject.toml. Not in blocked-scope list. APPROVE + ALLOWED — merging.

## Deep Code Review — verdict ### Summary - Status: APPROVE - Head SHA: 1333792 · Mergeable: true · Auto-merge scope: ALLOWED (tests-only) - Files reviewed: 4 · Lines: +533 / -0 ### Schema verification All 12 test queries verified against real DB. Zero phantom columns found. **domrf_kn_objects** — all columns confirmed real: - obj_id (bigint), latitude/longitude (numeric), comm_name, dev_name, obj_class, flat_count, site_status, snapshot_date, district_name, region_cd — all exist - geom_3857 — correctly not referenced (was PR #196 phantom) **ekburg_construction_permits** — all columns confirmed real: - permit_type, permit_number, issue_date, expiry_date, developer_name, developer_inn — exist - object_name, object_type, construction_address, cadastral_number — exist - total_area_sqm, living_area_sqm, living_area_fact_sqm, rve_number, rve_date — exist - units_count — correctly not referenced (was PR #213 phantom) **mv_layout_velocity** (matview, confirmed via pg_attribute): - obj_id, room_bucket, total_deals_24mo, avg_area_m2, avg_price_thousand_rub_per_m2, window_start, window_end — all real **domrf_kn_flats** — obj_id, total_area, rooms, is_studio, flat_type, price_per_m2, status, snapshot_date — all real **objective_complex_mapping** — domrf_obj_id, objective_complex_name — real **objective_corpus_room_month** — deals_total_vol_m2, deals_total_count, report_month, project_name — real **cad_parcels_geom** (VIEW) — cad_num, geom — real **cad_quarters_geom** — cad_number, geom — real Note: m.obj_id in test_objective_corpus_room_month_columns resolves from derived subquery alias (cm.domrf_obj_id AS obj_id), not from objective_corpus_room_month directly. Correct SQL. ### Fixture safety - Teardown on failure: YES. try/finally block (lines 141-153) guarantees db.close() and DROP SCHEMA CASCADE run even on test exceptions. - schema_created flag: YES (line 113). Set only after CREATE SCHEMA IF NOT EXISTS succeeds (line 119). Teardown gates on flag (line 147) — DROP never attempted if CREATE SCHEMA failed. No accidental DROP SCHEMA public CASCADE possible. - pg_dump fallback: YES. FileNotFoundError, nonzero returncode, timeout — all handled gracefully, falls back to operating directly in public schema. EXPLAIN-only tests are read-only so this is safe. - PostGIS handling: pg_dump --schema-only correctly dumps geometry column metadata and GIST index definitions. --no-owner --no-acl prevent permission errors on restore. - Schema name injection: _random_suffix() uses only ascii_lowercase + digits (alphanumeric). Schema name double-quoted in all executions (lines 117, 137, 150). No user-controlled input reaches the name. ### DML / side-effect check - No INSERT, UPDATE, DELETE, TRUNCATE, DROP TABLE, ALTER TABLE, CREATE INDEX in any test query. Grep confirmed clean. - All test queries use EXPLAIN only (not EXPLAIN ANALYZE). No query execution, no side effects. - Fixture DDL is limited to CREATE SCHEMA IF NOT EXISTS + DROP SCHEMA IF EXISTS ... CASCADE on temp schema only. ### Credentials check No hardcoded passwords, tokens, or secrets. Credentials flow exclusively through os.environ.get("TEST_DATABASE_URL", "") (line 31). ### pyproject.toml change One line only: integration marker appended to [tool.pytest.ini_options].markers (line 86). No other changes. ### Skip logic requires_test_db = pytest.mark.skipif(not _TEST_DATABASE_URL, ...) at module level in conftest.py (line 36). Applied as pytestmark = [...] at module level in test file (lines 32-35). All 12 tests inherit it and skip cleanly when TEST_DATABASE_URL absent. Additional pytest.skip() guard inside fixture body (line 96) as belt-and-suspenders. ### Cross-file impact No modifications to existing test files. No backend/tests/conftest.py at root tests level (does not exist). New files only. No fixture name collisions. ### Minor observations (non-blocking) - conftest.py line 63: db_url.replace("+psycopg", "") for pg_dump URL is fragile — would silently fail if dialect string changes. Failure path is handled gracefully (fallback), so non-blocking. - session scope: both fixtures are session-scoped. EXPLAIN-only tests make this safe today. - f.status = 'sold' in test_avg_price_sold_query: no CHECK constraint on domrf_kn_flats.status, so EXPLAIN passes regardless of value validity. Acceptable for phantom-column detection. ### Recommended next steps No blocking issues. Merge. Optional follow-up (separate PR): replace db_url.replace with proper URL parsing for pg_dump invocation. ### Auto-merge policy ALLOWED — all changes under backend/tests/** and backend/pyproject.toml. Not in blocked-scope list. APPROVE + ALLOWED — merging.

lekss361 merged commit 76b1f8673f into main

2026-05-16 13:07:14 +00:00

lekss361 deleted branch feat/197-test-infra-gate

2026-05-16 13:07:14 +00:00

lekss361 referenced this pull request from a commit

2026-05-16 13:07:16 +00:00

test(infra): integration phantom column gate (#197) (#216)

lekss361 referenced this pull request

2026-05-16 13:12:36 +00:00

feat(nspd): TIER 3 risk zones (#94 PR1 of 4) #217

lekss361 referenced this pull request

2026-05-16 15:30:08 +00:00