Kjell Tore Guttormsen
37547fe292
The context sets, the packaged knowledge bases and the example bundles are replaced by one fictitious example set about IT operations in an invented organisation: three context sets (serverrom-2027, driftsavtale-2027 and the two-base drift-og-avtale-2027), two synthetic knowledge bases under src/portfolio_optimiser/data/kunnskapsbaser and two example bundles under src/portfolio_optimiser/data/bundles. Numbers, codes and structural values in tests and fixtures are kept; names, ids and wording change. Dated measurement documents that only recorded runs on the replaced material are deleted. Gate figures measured on the new set are not comparable with earlier ones. The exclusion gate from the previous commit is green: 0 tracked files hit outside the shared/ subtree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
48 lines
2.8 KiB
Markdown
48 lines
2.8 KiB
Markdown
# Spike C findings — blocking deterministic hybrid validator (B1)
|
|
|
|
**Assumption:** a blocking deterministic hybrid-validator can *structurally* block an
|
|
out-of-range proposal from ever reaching the expert — and a valid proposal yields a
|
|
risk distribution (P10/P50/P90) with a capped self-repair loop.
|
|
|
|
## Result — CONFIRMED (fully deterministic, no MAF, no endpoint)
|
|
|
|
Three stages over a typed Pydantic IR (`SavingsProposal`):
|
|
|
|
1. **IR schema gate.** Field validators + a cross-field `@model_validator` reject
|
|
malformed proposals at construction: negative quantities, and a claimed saving that
|
|
exceeds the affected items' own total (both proven to raise `ValidationError`).
|
|
2. **PuLP solver-in-the-loop.** A real **CBC** solve (PuLP's bundled binary, found at
|
|
`solverdir/cbc/osx/i64/cbc` on this Intel mac) bounds the maximum feasible saving at
|
|
`MAX_SAVING_FRACTION = 0.30` of the affected items' cost. CBC absence would
|
|
**escalate** (`CbcUnavailable`) — there is no silent LP-relaxation fallback (R2).
|
|
*(PuLP 3.x deprecates `PULP_CBC_CMD`; PuLP 4.0 will require `pip install pulp[cbc]` +
|
|
`COIN_CMD`. The spike silences the cosmetic warning — a Fase 2 migration note.)*
|
|
3. **Monte Carlo.** Seeded stdlib `random` + `statistics.quantiles` over uncertain
|
|
unit-costs give **P10 ≤ P50 ≤ P90** of the feasible saving, and are reproducible under
|
|
the fixed seed.
|
|
|
|
**Structural block (the B1 headline):** an out-of-range proposal returns a `Rejection`,
|
|
a *different type* from `ValidatedProposal` that carries **no percentiles** — accessing
|
|
`.p50` on it raises `AttributeError`, so a rejected proposal can never be consumed as
|
|
validated. A valid proposal (claim well under the feasible cap) returns a
|
|
`ValidatedProposal` with the percentiles. **`self_repair` is capped** at `max_attempts`
|
|
and hard-stops (verified it calls the generator exactly N times and no more, B4).
|
|
|
|
### Worked example (project KONTOR-IT-E1, codes 05.2 + 03.1)
|
|
|
|
- Affected items' total ≈ **1 482 500 NOK**; 30% feasible cap ≈ **444 750 NOK**.
|
|
- Claim **200 000** → `ValidatedProposal` with ordered P10/P50/P90 around the cap.
|
|
- Claim **800 000** → constructs (≤ total) but exceeds P90 feasible → **`Rejection`**.
|
|
|
|
## Token use
|
|
|
|
**0 — validator is fully deterministic; live LLM generation is gated.** The
|
|
`generate_via_llm` entry point `skip`s without a LOCAL endpoint; all validator / solver /
|
|
Monte-Carlo logic is exercised on crafted IR directly, with **zero** model tokens spent.
|
|
|
|
## Implication for Fase 2
|
|
|
|
The obligatory, blocking validator is realizable exactly as specified: typed IR +
|
|
real solver + risk percentiles + a type-level rejection that cannot leak downstream.
|
|
Fase 2 should keep the `Rejection`/`ValidatedProposal` type split (structural block, not
|
|
an advisory flag) and the CBC-absent escalate contract.
|