Commit graph

22 commits

Author SHA1 Message Date
6a32acb3aa fix(competitors): size-weight avg velocity by flats_total (#949 audit, option B)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m32s
Deploy / build-worker (push) Successful in 3m10s
Deploy / deploy (push) Successful in 1m56s
weighted_avg_velocity was a naive mean despite the name — a 500-flat ЖК weighed
the same as a 20-flat one. Now count-weighted by flats_total (sql.md AVG
principle): Σ(velocity*flats_total)/Σ(flats_total). Competitors with unknown
flats_total are excluded from weights; if sizes are unknown for ALL, graceful
fallback to the simple mean (den>0 guard). Field name + API contract UNCHANGED
(zero consumer ripple — traced: only CompetitorsSummary, no frontend ref).

Tests: equal sizes → weighted==naive (existing 6.0 stays); NEW test with
500-flat@40 + 20-flat@2 → 38.54 (not naive 21.0), proving the weighting.
2026-06-04 14:05:20 +05:00
e9585bcd3b test(suite): re-align drifted tests with current code (CI-rehab 2/3)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 1m8s
Deploy / build-backend (push) Successful in 36s
Deploy / deploy (push) Successful in 1m13s
Suite was uncollectable for a while → code evolved, mocks didn't (42 failed /
5 errors → 1687 passed / 0 failed / 0 errors). TEST-ONLY, no app code changed,
no real bugs found (code-reviewed  — no test weakened to fake-green).

- best_layouts: sum_deals→deals_window rename; velocity-scaling direction for
  new SF-01 divisors (last_month=1.0/last_year=12.0)
- quarter_dump_lookup: 7→10-tuple mock rows (risks/opportunity/red_lines cols);
  fix early-exit call-counts for the cad_zouit fallback (#232)
- cadastre_bulk: add xmin/ymin/xmax/ymax to harvest-quarter db mock
- nspd_client: add required QuarterDump.opportunity
- TopLayoutRow: add required is_oversold
- analyze_{market_price,recent_permits,inline_weights}: SQL-signature dispatch
  instead of fragile positional db.execute indices (SF-B5 query reorder; also
  fixes a double-POST geom-starvation false-negative)
- custom_pois: move centroid >15km so center_bonus=0 isolates custom-POI delta
- layout/report PDF: runtime WeasyPrint probe (skip macOS dev, RUN on CI)
- mv_layout SQL: normalize +psycopg DSN + connectivity-probe skipif
- admin token tests: skip (X-Admin-Token gate removed #437/#426; protection is
  Caddy basic_auth + RBAC, covered by test_rbac)

Refs #944.
2026-06-03 19:05:43 +05:00
c8ab3e8be9 fix(analyze): district KeyError 500 in #994 persist + revive by-bbox tests
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m31s
Deploy / build-worker (push) Successful in 2m28s
Deploy / deploy (push) Successful in 1m44s
LIVE BUG (from #994, already merged): the analysis_runs persist call-site in
analyze_parcel extracted district via result_payload["district"]["district_name"]
— but a district dict without that key (real data: partial district lookup)
raised KeyError OUTSIDE the best-effort SAVEPOINT (extraction is at the call
site, not inside persist_analysis_run) → 500 on the LIVE /analyze endpoint.
Fix: .get("district_name") → None instead of raising. Caught by reviving the
analyze test suite (below).

Test-infra (the suite was un-runnable, which is why the bug shipped):
- Lazy-import WeasyPrint in layout_tz_pdf.py + trade_in_pdf.py (matches the
  existing report_pdf.py / snapshot_pdf.py pattern). The eager top-level imports
  made `app.main` (→ parcels/trade_in routers) un-importable on hosts without
  WeasyPrint native libs (e.g. macOS dev), breaking pytest COLLECTION of the
  whole api/v1 suite. WeasyPrint is still imported when a PDF is actually rendered.
- tests/conftest.py: autouse fixture clears app.dependency_overrides after each
  test (anti-leak — a leaked get_db override caused real-DB connection attempts).
- test_parcel_by_bbox.py: rewrite get_db mocking from patch("...get_db") (a no-op
  — FastAPI Depends holds the original ref) to app.dependency_overrides[get_db],
  add explicit X-Authenticated-User header (RBAC gate), patch latest_run_dates,
  + a new test asserting last_analysis_date from a latest run (#994). 5/5 green.

NOTE: reviving collectability exposes PRE-EXISTING rot in other api/v1 suites
(analyze/admin/best_layouts: RBAC-header + stale-assertion/mock drift) — those
are NOT regressions from this PR (they were uncollectable before) and are
tracked separately for a deliberate suite-rehab + CI test-gate effort.

Refs #994.
2026-06-03 18:04:37 +05:00
72eae3bc9b feat(parcels): persist analysis runs + real last_analysis_date (#994)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m35s
Deploy / build-worker (push) Successful in 2m31s
Deploy / deploy (push) Successful in 1m42s
Persist each successful POST /analyze into analysis_runs (migration 127):
best-effort SAVEPOINT-wrapped INSERT in a thin analysis_runs repository, then
explicit db.commit() (get_db has no commit-on-success; SAVEPOINT RELEASE alone
does not persist — without the commit the row rolls back on request teardown,
a silent no-op caught in code review). A persist failure never breaks the
response or poisons the session. Result serialized via jsonable_encoder→
json.dumps (handles embedded Pydantic models/dates); confidence/status
normalized to satisfy the 127 CHECK constraints (unknown→NULL/'complete').

Populate by-bbox last_analysis_date from v_analysis_runs_latest via a single
batch query (no N+1), replacing the #307 placeholder. The read is best-effort
wrapped too — a view-missing deploy window or future drift must not 500 the
map (falls back to last_analysis_date=None).

Additive only — analyze response shape unchanged. Tests: by-bbox suite patches
latest_run_dates (mock-db returns same rows for any query) + new test asserts
ISO last_analysis_date from a latest run. Analyze mock-suite unaffected
(graceful side_effect overflow + best-effort persist absorb the extra INSERT).
Incidental: ruff-format fixed one pre-existing f-string spacing (line ~2391)
the format hook flags on touch.

Closes #994. Refs #961.
2026-06-03 17:36:01 +05:00
c61c8c86af feat(sf-b1): GET /parcels/by-bbox — map entry endpoint + parcel_user_status table (#336)
Some checks failed
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Has been cancelled
Deploy / build-backend (push) Has been cancelled
2026-05-17 21:25:48 +00:00
lekss361
1799e1fb30 fix(sf-02): добавить site_status/ready_dt в /analyze competitors + sort строящиеся first
SQL в /analyze не выбирал site_status, ready_dt из domrf_kn_objects;
ORDER BY flat_count DESC ставил сданные ЖК в топ.

Closes (epic part) #271 item 2
2026-05-17 13:29:59 +03:00
lekss361
c701636515 Merge forgejo/main into fix/sf-01-velocity-time-window
Resolve conflict in best_layouts.py: keep SF-01 inline velocity
(deals_window numerator) + add SF-05 sold_pct clamp at 100% and
is_oversold flag from main.
2026-05-17 12:15:40 +03:00
lekss361
5a4408e0ab fix(sf-01): time_window honest velocity — inline SQL с реальным фильтром report_month
Раньше _VELOCITY_DIVISORS делил агрегаты mv_layout_velocity (24 мес)
на 4/12 для quarter/year, не меняя реальное окно данных. Теперь
inline SQL из objective_corpus_room_month с CAST(:window_interval AS interval).

velocity_per_month = deals_window / months_in_window (1.0/3.0/12.0).
Разные time_window → разные строки из БД → разный mix/velocity/jk_count.

Closes (epic part) #271 item 1
2026-05-17 12:12:19 +03:00
lekss361
247c252b04 fix(sf-05): clamp sold_pct_of_supply at 100% + add is_oversold flag
Symptom: sold_pct_of_supply = sum_deals_24mo / supply_count_snapshot * 100
yields >100% (e.g. 199%) for fast-selling small formats due to incompatible
time windows (24-month deals vs point-in-time supply snapshot).

Option A (hotfix): clamp to 100.0, expose is_oversold=True for UI badge.

- best_layouts.py: compute sold_pct_raw, clamp with min(..., 100.0),
  set is_oversold = raw > 100%
- parcel.py TopLayoutRow: add is_oversold: bool field
- best-layouts.ts TopLayoutRow: add is_oversold: boolean
- BestLayoutsBlock.tsx: show warn badge ">100%" when is_oversold=True
- tests: two new cases — raw 199% clamps + is_oversold=True;
  raw 50% passes through + is_oversold=False

Closes (epic part) #271 item 5
2026-05-17 11:57:29 +03:00
lekss361
c94545f0eb fix(#261): add db.commit() regression tests для custom_pois service
Проблема: POST /custom-pois возвращал 201 с id=N, но row не сохранялась
в БД — user_custom_pois оставалась пустой после request close.

Симптом возникает если service-функция не вызывает db.commit() явно:
get_db.finally → db.close() без commit → psycopg3 rollbacks pending tx.

Код (#257) содержит db.commit() во всех 3 мутациях, но тесты полностью
мокировали service layer через patch() — regression в commit не ловился.

Fix: добавлены 6 service-level тестов с паттерном "два сеанса":
- MagicMock(db) + db.commit.assert_called_once() для create/update/delete
- db.commit.assert_not_called() для not-found и empty-payload случаев

Closes #261
2026-05-17 10:16:36 +03:00
lekss361
485f489bd2 feat(#254): user_custom_pois — backend schema + CRUD + scoring integration
- SQL migration 101_user_custom_pois.sql: table with GEOGRAPHY(POINT) + GIST index
- Pydantic schemas: CustomPoiCreate / CustomPoiUpdate / CustomPoiOut
- Service custom_pois.py: CRUD + get_overlaps_for_scoring (1km radius, psycopg v3 CAST)
- API /api/v1/custom-pois: POST(201)/GET/PATCH/DELETE, X-Session-Id header auth
- Scoring in analyze_parcel: custom POI block after OSM POI loop,
  decay weight * max(0, 1 - dist/1000), custom_poi_score_items in response
- Tests: CRUD + validation (weight/lon/lat 422) + scoring integration (mock-based)
- Vault: Schema_User_Custom_Pois.md + Endpoint_Parcel_Analyze v3.9 section
2026-05-17 09:34:33 +03:00
7d0bc4bb6b Merge pull request 'feat(#29,#232): wire cad_parcels.permitted_use в analyze + cad_zouit fallback (G2+G3)' (#240) from feat/g2-g3-analyze-parcel-meta-zouit-fallback into main
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m26s
Deploy / deploy (push) Successful in 48s
2026-05-17 05:26:02 +00:00
lekss361
f70be684da feat(#29,#232): wire cad_parcels.permitted_use in analyze + cad_zouit fallback (G2+G3)
TASK A (#29 G2): add parcel_meta to analyze response
- New ParcelMeta Pydantic schema in app/schemas/parcel.py
- SELECT from cad_parcels WHERE cad_num=:c in analyze_parcel() (step 9f)
- Returns permitted_use_established_by_document, land_record_category_type,
  land_record_subtype, cost_value; None when row absent
- Tests: test_analyze_parcel_meta.py (found + not-found cases)

TASK B (#232 G3): cad_zouit fallback in _get_zouit_overlaps
- When nspd_quarter_dumps has zouit_count==0, fall back to ST_Intersects
  query on cad_zouit (3483 rows, GIST indexed)
- Overlaps tagged with source='cad_zouit'; format compatible with NSPD path
- gate_verdict.py: BLOCKER_TYPE_ZONE_KEYWORDS tuple for keyword-based
  classification (охранная зона / трубопровод / электр / газ -> blocker;
  СЗЗ -> warning); NSPD subcategory path preserved backward-compat
- Tests: 6 new test cases in test_gate_verdict.py covering cad_zouit path
  and backward-compat for NSPD subcategory path

Updated db.execute call sequence in test_analyze_*.py (index shift +1 at pos 10).
2026-05-17 08:17:22 +03:00
lekss361
0135f3c7c4 feat(#104): add POST /admin/scrape/ekburg-permits for manual trigger
Wraps refresh_all / refresh_year Celery tasks behind the existing
AdminTokenAuth gate so the table can be populated on demand after
first deploy instead of waiting for the monthly beat (1st of month).

- TriggerEkburgPermitsRequest: year int|None, ge=2022 le=2030
- year=None  -> refresh_all.apply_async()  (scope all_years_2022_2026)
- year=N     -> refresh_year.apply_async(args=[N])  (scope year_N)
- 4 smoke tests: all/year/invalid_year/no_token
2026-05-17 08:13:32 +03:00
lekss361
31581cedd2 fix(#112): remove broken status='sold' filter from competitors avg_price query
domrf_kn_flats.status is NULL in ~99.8% of rows, so WHERE status='sold'
always returned 0 rows and avg_price_per_m2 was always None. Drop the
filter; AVG over all rows with price_per_m2 IS NOT NULL is semantically
correct for a complex-level price estimate.

Adds regression test test_competitors_avg_price_populated (Issue #227).
2026-05-16 22:46:10 +03:00
46203ab365 feat(permits): Celery beat monthly + analyze recent_permits (#105 Phase 4+5) (#213)
Some checks failed
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Failing after 16s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 3m44s
Deploy / deploy (push) Has been skipped
2026-05-16 12:23:52 +00:00
27a0957bb5 feat(analyze): add market_price из mv_quarter_price_per_m2 (#33 PR B) (#210)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m27s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m43s
Deploy / deploy (push) Successful in 40s
2026-05-16 11:28:16 +00:00
092976f656 feat(site-finder): inline POI weights pass-through в /analyze (#201 Phase 1) (#206)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m35s
Deploy / build-worker (push) Successful in 2m44s
Deploy / build-frontend (push) Successful in 2m9s
Deploy / deploy (push) Successful in 37s
2026-05-16 11:00:41 +00:00
ad16fc0e42 feat(parcels): best-layouts endpoint + service (#113 PR C) (#196)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m37s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m44s
Deploy / deploy (push) Successful in 37s
2026-05-16 09:10:06 +00:00
4deb1b0284 feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)
Some checks failed
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Failing after 0s
Deploy / build-worker (push) Successful in 5m11s
Deploy / deploy (push) Failing after 0s
Deploy / build-backend (push) Successful in 3m44s
2026-05-16 05:58:27 +00:00
e561df1b55 feat(parcels): competitors endpoint (Forgejo #112 Phase 1 backend) (#191)
Some checks failed
CI / backend (push) Successful in 1m16s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Failing after 12s
Deploy / build-worker (push) Failing after 12s
Deploy / build-frontend (push) Failing after 0s
Deploy / deploy (push) Failing after 0s
CI / frontend (push) Successful in 2m17s
2026-05-15 22:21:05 +00:00
lekss361
5da66bbc93
feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171)
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)

Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
  + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
  using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
  + generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
  dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests

* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)

* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)

Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.

Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1:
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.

Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.

* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)

Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.

Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.

All 28 cadastre tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:31:32 +03:00