The live consumption measurement on N100:2023, N200:2024 and N500:2024, ordered
as P1 (20260908T113200Z-46497613). Three paid arms in A-form
(--prepass-payload), one per bundle, NOK 0.21 of a NOK 5 cap, 0 x 429. No file
under src/ touched; no production code in this commit.
ALL FIVE KNOWN-POSITIVES HELD BEFORE ANYTHING WAS PAID FOR. The three
sha256-tree refs reproduce vegnormal-okf's values character for character;
okf.navigate_bundle -- our own code -- reaches exactly 446 / 1133 / 270
concepts with 0 unfollowed links; the pre-pass on okf a37d5ce delivers the gold
concept at RANK 1 of 8 on all three with the flagless default command;
okf_contract_check exits 0 three times; the cheap client probe is green.
okf's three opt-in flags were NOT used: the order allowed them only if the gold
came back below_k, and it never did.
THE VERDICT IS 0 OF 3, and the three reasons are different and separately
owned. Criterion: (a) the model answers with the gold requirement's central
condition AND (b) cites the right concept id AND (c) zero hallucinated values.
(c) is 0 on all three -- the model quoted only what it had. (a) is YES only on
N100, where it reproduced the body sentence verbatim. On N200 and N500, 0 of 6
keywords from each gold body appear in any answer.
THE MAIN FINDING, MEASURED, NOT GUESSED (okf owns it): the payload's excerpt
carries `text` and no `title`/`req_number`, so for N100 and N500 the
requirement number the question is ASKED BY appears nowhere in the model's
context -- not in the gold excerpt, not anywhere in the delivered cut. The
ranking finds the right document ON that number and the delivery then drops it.
Our own read_file returns the whole 774-character file including
`req_number: Krav 3.3.1-13`; the excerpt is 98 characters of body. The declared
cut is strictly less informative than our own navigation on precisely the key
the question uses. This is the structural twin of S7c: there the locks bought
the BYTES and not the STRUCTURE; here the ranking buys the RIGHT DOCUMENT and
not the KEY.
THE SECOND FINDING IS OURS, and the order asked for it by name: po has no
lookup door in A-form. `run.py:1243` is hard-coded (`Find a cost-saving measure
for {project.id}`), none of run_project's 26 parameters carries a question, a
prompt, an objective or a task (measured with inspect.signature), and
`--explore` is refused together with `--prepass-payload` (rc 1, zero model
calls). The question reaches the model only as the pre-pass's declared
`question` line -- verbatim in 2 of 6 / 2 of 5 / 2 of 6 prompts -- while the
task line says something else. On N200 and N500 the model followed the task it
was actually given and used a different delivered concept. That is the form,
not a misconfiguration, and choosing what to do about it is the operator's.
(b) IS SCORED ON THE MODEL'S REFERENCE, NOT ON THE STAMP, and that is a
sharpening of the order's criterion rather than a softening. The stamp cites
the delivered concepts MECHANICALLY: `stamp intersect delivered` is 8 of 8 on
all three and would be 8 of 8 for a run in which the model read nothing. A
measure that can only come out one way is this repository's vacuous-gate class,
so the load-bearing reading is what the model actually named -- and there the
answer is no on all three.
A THIRD FINDING, reported and not fixed: N200's run VALIDATED a proposal whose
cost codes were `03418c46` (a real requirement id used as a cost line) and
`03423b12` (0 matches among the bundle's 1137 files -- invented). It passed
because the run was un-anchored and the validator's stage 0 was skipped: the
first LIVE instance of exactly what --require-cost-baseline (F4, session 100)
refuses. The run said so itself, in plain text, on its own notice line.
TWO THINGS GOT CONFIRMED LIVE ON THREE FRESH CORPORA, neither of them ordered.
`{run_id}-parse-failures.json` is absent from all three outboxes, so the
derived structured-output grammar (Fase 1b finding 1b) was accepted by the live
endpoint on corpora it had never been tested against -- closing one of that
row's stated honesty limits. And Step 5's informed refinement fires: attempts 2
and 3 carry "your previous proposal was REJECTED by the deterministic
validator" verbatim, and the claim falls monotonically (3 000 000 -> 1 500 000
-> 600 000 on N100; 350 000 -> 210 000 -> 90 000 on N500).
S7a-3's slack was paid for here for the first time: all three bases declare
`bundle_id` in the root index while mounted under a different directory name.
Before session 81 that was REFUSED, so none of the three could have been opened
as delivered.
HONESTY LIMITS, stated: one run is one run -- three arms, one question each,
one model, no repetition, so nothing here measures variance. (a) on N100 is not
evidence that the chain answers lookups: that gold requirement is
cost-shaped, so answering the task and answering the question coincide, and the
coincidence is identified rather than hidden. (c) = 0 covers the PROSE; the
structured proposal invents cost codes, which is structurally required with no
baseline. The denominator vocabulary was used 0 times in 17 replies, but no arm
was ever in the position the marker exists for, so that is an unanswered
denominator and not evidence it does not work. None of the three bundles was
reviewed on its subject matter; (a) compares against the concept body only.
Suite after `git add`: 1511 passed / 5 skipped, ruff and mypy clean, golden
demo-transcript.stdout BYTE-UNCHANGED (shasum -a 1 of the CONTENT =
ea8c534773acdbe41ae68f2c55724d69aaf8be4f, never the git blob id). Leak check
after staging: 0 hits for the real host in staged content and in every tracked
file. Verdicts sent FYI to vegnormal-okf (which owns nothing here -- the
bundles are clean) and to llm-ingestion-okf (one field).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kjell Tore Guttormsen
e7ffe9e196