#1244 (security): внешние/скрейпинг-строки (comm_name из DOM.РФ, headline/usp_text)
с ведущим = + - @ \t \r писались как есть → openpyxl сохранял как формулы
(data_type='f'), исполнялись при открытии в Excel/LibreOffice. _sanitize_formula
префиксует такие строки апострофом (OWASP CSV-injection escape); числа/даты/bool
не трогаются. _write_kv labels тоже санитизируются. Подтверждено на openpyxl 3.1.5.
#1245 (concurrency): async ask() вызывал sync get_report_for_chat() (sync SQLAlchemy
тянет крупный JSONB §22-отчёт) напрямую — блокировал event loop, в отличие от
LLM-ветки (run_in_threadpool). Обёрнуто в run_in_threadpool.
Closes#1244Closes#1245
API отвергал ?horizon=24 (422), хотя ТЗ §12.1 называет 6/12/18/24, а движок
УЖЕ считает 24 на каждом ране: _DEFAULT_HORIZONS=(6,12,18,24) во всех 6 точках
стека (orchestrator/forecast-task/demand_supply_forecast/scenarios/
special_indices/report_assembler), PIPELINE_HORIZON_MONTHS=24.
_hidden_release_fraction клампит h/18→1.0 на 24 (без переполнения),
future_supply._horizon_weight расширяет окно чисто — скрытых ≤18 потолков нет.
Чистое расширение валидатора-enum, не новая математика.
Backend: _ALLOWED_FORECAST_HORIZONS → {6,12,18,24}, Query/docstring/error-msg.
Frontend: HorizonSelector HORIZONS=[6,12,18,24] (тип horizon=number, union не нужен;
прочие потребители data-driven через meta.horizons/forecasts_by_horizon).
Тесты: API принимает 24/отвергает 30; движок-тесты доказывают h=24 осмыслен
(поля посчитаны, demand(24)>demand(18), hidden созрел, индексы в диапазонах).
Closes#944 (Q1 горизонт 24)
UPDATE без проверки rowcount затрагивал 0 строк для несуществующего
job_type; get_one возвращал hardcoded _fallback. PUT отвечал 200,
админ считал настройку сохранённой — но она терялась (новый job_type
до seed-миграции, опечатка в path).
Patch: update() проверяет result.rowcount: при 0 делает INSERT с
переданными колонками (непереданные → дефолты таблицы из м.81).
INSERT использует ON CONFLICT (job_type) DO NOTHING + повторный
UPDATE как защита от гонки. Поведение для существующих строк не
изменилось. 8 новых тестов (6 service + 2 API).
Closes#1223
_ACTIVE_STATUSES = frozenset({"sales", "construction"}) — английский словарь
никогда не совпадал с domrf_kn_objects.site_status, который scraper берёт
СЫРЫМ из siteStatus дом.рф (domrf_kn.py:316). Реальные prod-значения
русские: «Строящиеся»/«Сданные».
Прод-аудит:
- data/sql/105_add_sales_started_flag.sql фильтрует по 'Строящиеся' (~1322 строки).
- partial index 66_indexes_recommend.sql использует те же.
- analytics_queries.py, MarketTab.tsx, CompetitorTable.tsx — все на русских.
Эффект: у ВСЕХ Competitor в POST /parcels/{cad}/competitors is_active=False
и CompetitorsSummary.active_count=0 при любых данных — типизированный
контракт систематически врал.
Patch: _ACTIVE_STATUSES = frozenset({"Строящиеся"}). Заодно обновил два
unit-теста которые кодировали баг (использовали "sales"/"construction"
в моках, тестировали логику против сломанного словаря). Теперь моки
матчат реальную prod-форму.
51/51 competitors-тестов зелёные. ruff clean.
Closes#1213
Phase B продолжает hot-path latency cut'ы analyze_parcel поверх Phase A (PR #1194):
1. get_air_quality_cached в weather_cache (третий TTL-слот рядом с forecast/climate):
- hit-TTL 1h (hourly bucket Open-Meteo), negative-TTL 5min.
- timeout 5s → 2s.
- single-flight под per-cache threading.Lock.
- Снимает 0.26с на cache-hit и до 5с при DNS-fail (тот же антипаттерн,
что был у forecast/climate до Phase A).
2. _neighbors_summary: 2 SQL → 1 statement.
- WITH neighbors AS (... LIMIT 30), overlaps AS (... LIMIT 5)
- + COALESCE(json_agg(row_to_json(...)), '[]'::json) в основном SELECT.
- Один round-trip вместо двух (~-47 ms на каждый analyze).
- Семантика идентична: те же ST_DWithin(geom, point, 100m) для neighbors,
ST_Intersects + ST_Area для overlaps. Anti-fakes guards (_COST_PER_M2_MIN/MAX,
float() try/except, >50 m² overlap) сохранены.
Формат возвращаемых dict не меняется (frontend контракт). 5 новых юнит-тестов в
test_weather_cache.py (TestAirQualityCache: hot-hit, negative, empty-hourly,
separate-from-weather, ttl-expires). 7 файлов API-тестов обновили patch-цель
_fetch_air_quality_sync → get_air_quality_cached.
Refs #1130
Два внешних HTTP-вызова в analyze (_fetch_weather_sync + _fetch_seasonal_weather_sync)
сидели без кэша. На проде в private network с restricted egress DNS до *.open-meteo.com
фейлит и каждый replay висит полный timeout (5s + 15s = до 20с лишних на analyze).
Профиль cProfile одного replay: 2.38с в httpx + 1.73с в SSL recv.
Вынес логику в app/services/weather_cache.py:
- TTL hit: forecast 6h, climate normals 7d. Negative-cache: 5min (DNS-fail не повторяет
timeout на каждый analyze, восстановление подхватывается через ~5 минут).
- Ключ — округлённые до 0.01° (~1 км) координаты.
- Single-flight: per-cache threading.Lock + check-then-fetch-then-store.
- Тайм-ауты сокращены: forecast 5s→2s, climate 15s→3s.
Контракт для caller'а не изменился: None по-прежнему допустим (env_ok-флаг в
_build_environmental_score). 7 файлов API-тестов обновили patch-цели на новые имена
get_weather_cached / get_seasonal_weather_cached. 11 новых юнит-тестов в
tests/services/test_weather_cache.py (single-flight через threading.Barrier(16), TTL
expire через monkeypatch _now, изоляция forecast vs climate).
Refs #1130
Add the LLM prose-composition path for the parcel-forecast chat, layered
over the deterministic Step-1 fallback which stays the safety net.
- chat/tools.py: 5 read-only section tools (exec_summary, product_recommendation,
forecast, risks, scenarios) — pure slices of the loaded report dict, no DB/
recompute, graceful on missing sections. market_now (raw analyze blob) and meta
are deliberately NOT exposed -> highest-PII data cannot reach the LLM.
- chat/safe_payload.py: the §19 gate — single place that builds the outbound
SafePayload from a section-aggregate allowlist; honors is_confidential hard-block.
- chat/orchestrator.py: manual tool-call loop with call-cap/termination, real
grounded_in provenance; any LLMResult.ok=False (disabled/timeout/rate_limited/
redaction_refused/call_cap/provider_error/empty) degrades to the deterministic answer.
- llm/prompts.py: versioned chat_system@v1 — answer only from sections, never
fabricate numbers, advisory tone, decline out-of-scope.
- api/v1/chat.py: branch on settings.llm_enabled; sync complete bridged via
run_in_threadpool. Default-off -> deterministic path, no provider built.
- Tests: fake provider only (no network), planted-secret redaction-boundary +
per-reason fallback + call-cap + numbers-from-report coverage.
Refs #957
Step 1 of #957. Answers parcel questions by reading the already-persisted
§22 SiteFinderReport (latest_run_for, schema "1.0", read-only) and returning
templated RU answers with engine numbers verbatim (§16, never fabricated).
Intent routing (explicit or RU keyword match) -> per-section renderers;
graceful on partial/missing sections and pending (no run / DB error) without
500s. Works with llm_enabled=False (llm_used always False); LLM composition
is Step 2. Mounted off /api/v1/chat so rbac_guard auto-requires an authed
known user.
Refs #957
Foundation PR: unified "our projects" source the §25.3 overlap engine (PR2)
will consume. Two origins normalized to OwnProject (class/timing/price/unit-mix):
- current <- domrf_kn_objects filtered by settings.own_developer_ids (numeric
prefix of composite dev_id; empty -> [] graceful, no DB hit, no hardcoded id)
- future <- new manual-entry own_planned_project entity (migration 148)
Adds OWN_DEVELOPER_IDS config (comma-sep -> list[int], default []),
own_planned_project table (range/unit_mix CHECKs via IMMUTABLE helper, generated
geom), /api/v1/own-projects CRUD (created_by from X-Authenticated-User), and
get_own_portfolio(db). Per-source graceful degradation; psycopg-v3 CAST clean.
Does not touch special_indices.py or parcels.py (out of scope).
Refs #1169
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
Insight = аналитик пишет free-form intel про локацию/участок с пометкой
«непублично» (is_confidential, §7.13/§8.9). CRUD под /api/v1/insights;
created_by из X-Authenticated-User (не из тела — спуфинг автора невозможен).
Зеркало custom_pois: raw-SQL service + Pydantic v2 schemas, sync def handlers
(threadpool, off event loop).
- data/sql/145_insight.sql: table insight (idempotent, geom GENERATED из lon/lat,
индексы district/cad_num/is_confidential/created_by/GIST); аддитивно расширяет
CHECK audit_log.action += insight_write (тот же constraint ck_audit_log_action,
to_regclass-guarded, superset — не ломает #962-аудит; НЕ правит applied 144).
- audit_middleware.classify_path: method-aware insight_write для POST/PUT/DELETE
/api/v1/insights (GET-чтения не аудируются); обратно совместимо с parcels.
- ACL: backend rbac_guard hard-блокирует только /api/v1/admin/*; pilot отсекается
frontend RouteGuard + Caddy (как parcels/custom_pois). is_confidential — стораемая
пометка без per-record backend-гейта (analyst видит, per #962-политике).
- tests: 28 insight (CRUD+filters+required+#261 commit) + 5 audit. 86 зелёных, ruff.
Part B (Location first-class, §8.2) — отдельный follow-up.
Refs #948.
render_report_docx (python-docx) mirrors report_md/report_pdf section order &
content, reuses report_pdf pure helpers (DRY), graceful on thin/empty report.
Widen /forecast/export format Literal to include docx → Word attachment.
Add python-docx dep + regenerate uv.lock (uv sync --frozen passes). Part of #959.
render_report_telegram_summary (pure, no new deps, DRY-reuses report_pdf str
helpers) + `tg` format on GET /{cad}/forecast/export → inline text/plain snippet
(no attachment, copy-paste-ready). md/json unchanged; no-run 404, bad format 422.
Graceful on thin/empty reports. Part of EPIC #959.
render_report_markdown (pure, no new runtime deps) reuses report_pdf's str
helpers (DRY), + GET /{cad_num}/forecast/export?format=md|json. No forecast run
→ 404; graceful on thin/empty reports; GFM-safe table escaping. PDF/XLSX already
existed; this adds the cheapest no-dep formats. Part of EPIC #959.
#992: attach response_model=AnalyzeResponse to POST /{cad_num}/analyze. Model
uses extra="allow" so any result_payload key not explicitly modeled is preserved
in the 200 response (no silent drops that would break Site Finder), and ALL
fields are Optional so the #93 202 fetch-stub serializes without a 500.
#994: add GET /{cad_num}/runs (light summary list, empty 200 not 404) and
GET /runs/{run_id} (full row incl result, 404 if missing), backed by
list_runs_for/get_run in analysis_runs.repository (psycopg v3 CAST(:x AS type)).
Routes ordered before /{parcel_id} so /runs/{run_id} is not shadowed.
Closes#992. Closes#994. Refs #961.
weighted_avg_velocity was a naive mean despite the name — a 500-flat ЖК weighed
the same as a 20-flat one. Now count-weighted by flats_total (sql.md AVG
principle): Σ(velocity*flats_total)/Σ(flats_total). Competitors with unknown
flats_total are excluded from weights; if sizes are unknown for ALL, graceful
fallback to the simple mean (den>0 guard). Field name + API contract UNCHANGED
(zero consumer ripple — traced: only CompetitorsSummary, no frontend ref).
Tests: equal sizes → weighted==naive (existing 6.0 stays); NEW test with
500-flat@40 + 20-flat@2 → 38.54 (not naive 21.0), proving the weighting.
LIVE BUG (from #994, already merged): the analysis_runs persist call-site in
analyze_parcel extracted district via result_payload["district"]["district_name"]
— but a district dict without that key (real data: partial district lookup)
raised KeyError OUTSIDE the best-effort SAVEPOINT (extraction is at the call
site, not inside persist_analysis_run) → 500 on the LIVE /analyze endpoint.
Fix: .get("district_name") → None instead of raising. Caught by reviving the
analyze test suite (below).
Test-infra (the suite was un-runnable, which is why the bug shipped):
- Lazy-import WeasyPrint in layout_tz_pdf.py + trade_in_pdf.py (matches the
existing report_pdf.py / snapshot_pdf.py pattern). The eager top-level imports
made `app.main` (→ parcels/trade_in routers) un-importable on hosts without
WeasyPrint native libs (e.g. macOS dev), breaking pytest COLLECTION of the
whole api/v1 suite. WeasyPrint is still imported when a PDF is actually rendered.
- tests/conftest.py: autouse fixture clears app.dependency_overrides after each
test (anti-leak — a leaked get_db override caused real-DB connection attempts).
- test_parcel_by_bbox.py: rewrite get_db mocking from patch("...get_db") (a no-op
— FastAPI Depends holds the original ref) to app.dependency_overrides[get_db],
add explicit X-Authenticated-User header (RBAC gate), patch latest_run_dates,
+ a new test asserting last_analysis_date from a latest run (#994). 5/5 green.
NOTE: reviving collectability exposes PRE-EXISTING rot in other api/v1 suites
(analyze/admin/best_layouts: RBAC-header + stale-assertion/mock drift) — those
are NOT regressions from this PR (they were uncollectable before) and are
tracked separately for a deliberate suite-rehab + CI test-gate effort.
Refs #994.
Persist each successful POST /analyze into analysis_runs (migration 127):
best-effort SAVEPOINT-wrapped INSERT in a thin analysis_runs repository, then
explicit db.commit() (get_db has no commit-on-success; SAVEPOINT RELEASE alone
does not persist — without the commit the row rolls back on request teardown,
a silent no-op caught in code review). A persist failure never breaks the
response or poisons the session. Result serialized via jsonable_encoder→
json.dumps (handles embedded Pydantic models/dates); confidence/status
normalized to satisfy the 127 CHECK constraints (unknown→NULL/'complete').
Populate by-bbox last_analysis_date from v_analysis_runs_latest via a single
batch query (no N+1), replacing the #307 placeholder. The read is best-effort
wrapped too — a view-missing deploy window or future drift must not 500 the
map (falls back to last_analysis_date=None).
Additive only — analyze response shape unchanged. Tests: by-bbox suite patches
latest_run_dates (mock-db returns same rows for any query) + new test asserts
ISO last_analysis_date from a latest run. Analyze mock-suite unaffected
(graceful side_effect overflow + best-effort persist absorb the extra INSERT).
Incidental: ruff-format fixed one pre-existing f-string spacing (line ~2391)
the format hook flags on touch.
Closes#994. Refs #961.
Resolve conflict in best_layouts.py: keep SF-01 inline velocity
(deals_window numerator) + add SF-05 sold_pct clamp at 100% and
is_oversold flag from main.
Раньше _VELOCITY_DIVISORS делил агрегаты mv_layout_velocity (24 мес)
на 4/12 для quarter/year, не меняя реальное окно данных. Теперь
inline SQL из objective_corpus_room_month с CAST(:window_interval AS interval).
velocity_per_month = deals_window / months_in_window (1.0/3.0/12.0).
Разные time_window → разные строки из БД → разный mix/velocity/jk_count.
Closes (epic part) #271 item 1
Symptom: sold_pct_of_supply = sum_deals_24mo / supply_count_snapshot * 100
yields >100% (e.g. 199%) for fast-selling small formats due to incompatible
time windows (24-month deals vs point-in-time supply snapshot).
Option A (hotfix): clamp to 100.0, expose is_oversold=True for UI badge.
- best_layouts.py: compute sold_pct_raw, clamp with min(..., 100.0),
set is_oversold = raw > 100%
- parcel.py TopLayoutRow: add is_oversold: bool field
- best-layouts.ts TopLayoutRow: add is_oversold: boolean
- BestLayoutsBlock.tsx: show warn badge ">100%" when is_oversold=True
- tests: two new cases — raw 199% clamps + is_oversold=True;
raw 50% passes through + is_oversold=False
Closes (epic part) #271 item 5
Проблема: POST /custom-pois возвращал 201 с id=N, но row не сохранялась
в БД — user_custom_pois оставалась пустой после request close.
Симптом возникает если service-функция не вызывает db.commit() явно:
get_db.finally → db.close() без commit → psycopg3 rollbacks pending tx.
Код (#257) содержит db.commit() во всех 3 мутациях, но тесты полностью
мокировали service layer через patch() — regression в commit не ловился.
Fix: добавлены 6 service-level тестов с паттерном "два сеанса":
- MagicMock(db) + db.commit.assert_called_once() для create/update/delete
- db.commit.assert_not_called() для not-found и empty-payload случаев
Closes#261
TASK A (#29 G2): add parcel_meta to analyze response
- New ParcelMeta Pydantic schema in app/schemas/parcel.py
- SELECT from cad_parcels WHERE cad_num=:c in analyze_parcel() (step 9f)
- Returns permitted_use_established_by_document, land_record_category_type,
land_record_subtype, cost_value; None when row absent
- Tests: test_analyze_parcel_meta.py (found + not-found cases)
TASK B (#232 G3): cad_zouit fallback in _get_zouit_overlaps
- When nspd_quarter_dumps has zouit_count==0, fall back to ST_Intersects
query on cad_zouit (3483 rows, GIST indexed)
- Overlaps tagged with source='cad_zouit'; format compatible with NSPD path
- gate_verdict.py: BLOCKER_TYPE_ZONE_KEYWORDS tuple for keyword-based
classification (охранная зона / трубопровод / электр / газ -> blocker;
СЗЗ -> warning); NSPD subcategory path preserved backward-compat
- Tests: 6 new test cases in test_gate_verdict.py covering cad_zouit path
and backward-compat for NSPD subcategory path
Updated db.execute call sequence in test_analyze_*.py (index shift +1 at pos 10).
Wraps refresh_all / refresh_year Celery tasks behind the existing
AdminTokenAuth gate so the table can be populated on demand after
first deploy instead of waiting for the monthly beat (1st of month).
- TriggerEkburgPermitsRequest: year int|None, ge=2022 le=2030
- year=None -> refresh_all.apply_async() (scope all_years_2022_2026)
- year=N -> refresh_year.apply_async(args=[N]) (scope year_N)
- 4 smoke tests: all/year/invalid_year/no_token
domrf_kn_flats.status is NULL in ~99.8% of rows, so WHERE status='sold'
always returned 0 rows and avg_price_per_m2 was always None. Drop the
filter; AVG over all rows with price_per_m2 IS NOT NULL is semantically
correct for a complex-level price estimate.
Adds regression test test_competitors_avg_price_populated (Issue #227).
