feat(chat): LLM tool-loop + §19 redaction for parcel chat (#957 Step2+3) #1173

Merged

bot-backend merged 1 commit from feat/chat-957-llm-loop into main

2026-06-08 12:53:04 +00:00

bot-backend commented

2026-06-08 12:45:26 +00:00

Collaborator

Summary

Step 2 (+ folded Step 3) of #957 — the LLM prose-composition path for the parcel chat, layered over the Step-1 deterministic fallback. Dormant until llm_enabled=True + a prod OPENAI_API_KEY; default-off → deterministic path, no provider constructed, no network.

§19 safety invariant (load-bearing)

safe_payload.build_chat_payload is the single outbound SafePayload constructor; the only complete() call in the chat package goes through it. fields can be populated only by the 5 read-only section tools (tools.py). The §22 report has 8 sections — the tools expose just 5; market_now (built from the raw analyze blob — competitors/prices/deals, highest-PII) and meta are deliberately NOT exposed, so they cannot reach the LLM. A planted-secret test puts phone+ФИО in market_now and proves it never appears in any outbound message. is_confidential=True → RedactionRefusedError → fallback with zero provider calls.

Tool-loop + grounding (§16)

Manual tool-call loop (no agent magic): complete → execute requested section slicers against the already-loaded report dict (no DB, no recompute, no engine math) → re-call until done or call_index hits llm_max_calls_per_request → compose. Numbers come from the report sections; the LLM only composes RU prose around them. grounded_in.sections = sections the model actually requested (honest provenance).

Fallback-always

Every LLMResult.ok=False reason (disabled / no_api_key / timeout / rate_limited / provider_error / redaction_refused / call_cap / empty_response) degrades to the Step-1 deterministic render_answer (llm_used=False + fallback_reason). Empty LLM prose is guarded (never returned).

Test plan

uv run pytest tests/services/chat/ tests/api/v1/test_chat.py -q → 77 passed (fake providers only, zero network)
ruff check + mypy (changed source files) → clean
Safety tests: planted-secret no-leak, market_now/meta unreachable, is_confidential→refused→fallback (0 provider calls), llm_enabled=False→no-provider tripwire, call-cap termination, per-reason fallback, numbers-from-report-not-invented
code-reviewer ✅ APPROVE (all 4 blocker classes provably absent)
live activation (you): rotate the exposed key → set OPENAI_API_KEY in prod backend+worker env → LLM_ENABLED=true; then smoke POST /chat/ask with a real §22 run.

Scope

Only services/chat/* + additive llm/prompts.py template + api/v1/chat.py. parcels.py / special_indices.py / own_portfolio.py untouched; no pyproject/uv.lock drift.

#957 chat is functionally complete (deterministic live now; LLM path dormant-until-key). Remaining: frontend chat UI (separate frontend track).

Refs #957

## Summary Step 2 (+ folded Step 3) of #957 — the **LLM prose-composition path** for the parcel chat, layered over the Step-1 deterministic fallback. Dormant until `llm_enabled=True` + a prod `OPENAI_API_KEY`; default-off → deterministic path, **no provider constructed, no network**. ### §19 safety invariant (load-bearing) `safe_payload.build_chat_payload` is the **single** outbound `SafePayload` constructor; the only `complete()` call in the chat package goes through it. `fields` can be populated **only** by the 5 read-only section tools (`tools.py`). The §22 report has 8 sections — the tools expose just **5**; **`market_now` (built from the raw `analyze` blob — competitors/prices/deals, highest-PII) and `meta` are deliberately NOT exposed**, so they cannot reach the LLM. A planted-secret test puts phone+ФИО in `market_now` and proves it never appears in any outbound message. `is_confidential=True` → `RedactionRefusedError` → fallback with **zero** provider calls. ### Tool-loop + grounding (§16) Manual tool-call loop (no agent magic): `complete` → execute requested section slicers against the **already-loaded** report dict (no DB, no recompute, no engine math) → re-call until done or `call_index` hits `llm_max_calls_per_request` → compose. Numbers come from the report sections; the LLM only composes RU prose around them. `grounded_in.sections` = sections the model actually requested (honest provenance). ### Fallback-always Every `LLMResult.ok=False` reason (disabled / no_api_key / timeout / rate_limited / provider_error / redaction_refused / call_cap / empty_response) degrades to the Step-1 deterministic `render_answer` (`llm_used=False` + `fallback_reason`). Empty LLM prose is guarded (never returned). ## Test plan - [x] `uv run pytest tests/services/chat/ tests/api/v1/test_chat.py -q` → **77 passed** (fake providers only, zero network) - [x] `ruff check` + `mypy` (changed source files) → clean - [x] Safety tests: planted-secret no-leak, `market_now`/`meta` unreachable, is_confidential→refused→fallback (0 provider calls), llm_enabled=False→no-provider tripwire, call-cap termination, per-reason fallback, numbers-from-report-not-invented - [x] code-reviewer ✅ APPROVE (all 4 blocker classes provably absent) - [ ] live activation (you): rotate the exposed key → set `OPENAI_API_KEY` in prod backend+worker env → `LLM_ENABLED=true`; then smoke `POST /chat/ask` with a real §22 run. ## Scope Only `services/chat/*` + additive `llm/prompts.py` template + `api/v1/chat.py`. `parcels.py` / `special_indices.py` / `own_portfolio.py` untouched; no pyproject/uv.lock drift. ## Next #957 chat is functionally complete (deterministic live now; LLM path dormant-until-key). Remaining: frontend chat UI (separate frontend track). Refs #957

bot-backend added 1 commit 2026-06-08 12:45:27 +00:00

feat(chat): LLM tool-loop + §19 redaction wiring for #957 (Step 2+3)

CI / changes (push) Successful in 6s

Details

CI / frontend-tests (push) Has been skipped

Details

CI / changes (pull_request) Successful in 5s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

Deploy / changes (push) Successful in 5s

Details

Deploy / build-frontend (push) Has been skipped

Details

Deploy / build-backend (push) Successful in 2m41s

Details

CI / backend-tests (push) Successful in 6m24s

Details

CI / backend-tests (pull_request) Successful in 6m23s

Details

Deploy / build-worker (push) Successful in 3m26s

Details

Deploy / deploy (push) Successful in 1m12s

Details

fceaaf9a2c

Add the LLM prose-composition path for the parcel-forecast chat, layered
over the deterministic Step-1 fallback which stays the safety net.

- chat/tools.py: 5 read-only section tools (exec_summary, product_recommendation,
  forecast, risks, scenarios) — pure slices of the loaded report dict, no DB/
  recompute, graceful on missing sections. market_now (raw analyze blob) and meta
  are deliberately NOT exposed -> highest-PII data cannot reach the LLM.
- chat/safe_payload.py: the §19 gate — single place that builds the outbound
  SafePayload from a section-aggregate allowlist; honors is_confidential hard-block.
- chat/orchestrator.py: manual tool-call loop with call-cap/termination, real
  grounded_in provenance; any LLMResult.ok=False (disabled/timeout/rate_limited/
  redaction_refused/call_cap/provider_error/empty) degrades to the deterministic answer.
- llm/prompts.py: versioned chat_system@v1 — answer only from sections, never
  fabricate numbers, advisory tone, decline out-of-scope.
- api/v1/chat.py: branch on settings.llm_enabled; sync complete bridged via
  run_in_threadpool. Default-off -> deterministic path, no provider built.
- Tests: fake provider only (no network), planted-secret redaction-boundary +
  per-reason fallback + call-cap + numbers-from-report coverage.

Refs #957