docs(front-page): the gate numbers the gate actually prints, and a breaking point that was measured
Four claims on the front page were false on this commit, and one of them was a number no division ever produced. **The retrieval gate.** README reported it RED on rows 3, 4, 5, 7, 8 and 9, with row 3 at 2 of 5 and row 4 at 3 of 6. Run on this commit it is RED on rows 5, 7, 8 and 9, with row 3 at 5 of 5 and row 4 at 6 of 6: `f81683e` made a withheld concept carry the rule that actually decided it, and `05cb190` gave the payload a `coverage` block, and neither updated the table. Row 8 is `0 of 3 | NOT RUN` on the default run and was published as `44 of 64 questions`, which is what it scores the day all three private sets are handed to it -- now labelled with the day and the machine rather than printed as a row. The same four figures were stale in `CLAUDE.md`. **The breaking point in a generated skill.** `int(LIMIT / per_withheld) if per_withheld else 0` printed `At roughly 0 concepts the bookkeeping alone reaches the 120000-byte limit` whenever the generation run withheld nothing -- the absence of a measurement, rendered as one, and read as a bundle that breaks before it holds anything. A run with no withheld entry has no slope to extrapolate from, so the sentence is withheld with its reason. The shipped `skills/okf-consume/SKILL.md` is generated with the question its `references/README.md` names, withholds nothing, and carried exactly that `0`; it is regenerated. Two arms in the test, because one would pass on an empty set: the bundles that withhold something must still state a positive figure. The sentence for that arm also stopped saying `**4 bytes** for 3 concepts` where the 4 bytes were the cost of 0 withheld entries. It is now `for N of M concepts`, which moves two generated skills' line counts and therefore the published comparison: 280 of 312 and 310 -> 281 of 313 and 311, re-measured, with the 62 differing lines unchanged. **Four tools.** A single-bundle server exposes three: `okf_list` is absent where there is nothing to list. README's table already said so in a cell; the heading and the CHANGELOG did not. **What `--accounting` accounts for.** The account is over the element classes each format's vocabulary names, verified against `accounting._READERS` rather than against the report: a file whose suffix has no reader is accounted at file level only, `.docx` reads `document.xml` and `footnotes.xml` (so headers, footers, endnotes and comments are outside), `.pptx` reads the slides (so speaker notes are outside), `.xlsx` reads the worksheets (so cell comments are outside and a cell contributes its cached value, never its formula), and `.rtf` skips its header and footer groups. A hidden slide or sheet IS counted -- it lives in the same part as a visible one. Nothing is built for this; the list is what `0 unaccounted` does not claim. Gates re-run on the commit: retrieval `GATE RED: rows 5, 7, 8, 9` (exit 1), MCP `GATE RED: rows 2` (exit 1), both matching what is now written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
bf697bfcad
commit
d300338e4d
9 changed files with 203 additions and 754 deletions
|
|
@ -363,3 +363,55 @@ def test_a_project_skill_still_passes_the_contract_checker(tmp_path: Path) -> No
|
|||
payload = okf_consume.build_payload(GOLDEN, question="hva er kravet til pris?")
|
||||
report = okf_contract_check.check(written.read_text(encoding="utf-8"), payload)
|
||||
assert report.findings == ()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bundle", BUNDLES, ids=lambda path: path.name)
|
||||
def test_the_breaking_point_is_a_measurement_or_it_is_withheld(
|
||||
bundle: Path, tmp_path: Path
|
||||
) -> None:
|
||||
"""`0 concepts` was a division that never happened, printed as a number.
|
||||
|
||||
The figure is EXTRAPOLATED from what one `withheld` entry costs, so a
|
||||
generation run that withheld nothing has no slope to extrapolate from:
|
||||
`per_withheld` was `0.0`, the guard returned the literal `0`, and the
|
||||
document told its reader the bundle's bookkeeping fills a 120000-byte
|
||||
budget at zero concepts -- before the bundle holds anything at all.
|
||||
|
||||
Driven from both sides so a generator that simply stopped stating the
|
||||
figure would fail: the bundle that withholds nothing must say it could not
|
||||
measure it, and a bundle that withholds something must still print a
|
||||
positive count.
|
||||
"""
|
||||
written = _generate(bundle, tmp_path / "out")
|
||||
text = written.read_text(encoding="utf-8")
|
||||
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
|
||||
assert payload["withheld"], "the known-positive arm withheld nothing to extrapolate from"
|
||||
assert "**0 concepts**" not in text
|
||||
stated = re.search(r"At roughly\s+\*\*(\d+) concepts\*\*", text)
|
||||
assert stated is not None, "a bundle that withheld something states no figure"
|
||||
assert int(stated.group(1)) > 0
|
||||
|
||||
|
||||
def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The arm the SHIPPED skill is on, and the one that was wrong.
|
||||
|
||||
`okf skill --example-question "Hva sier veiledningen om krav?"` delivers
|
||||
all three concepts of the golden bundle, so `withheld` is empty and there
|
||||
is no per-entry cost. The question is part of what the shipped file is
|
||||
(`skills/okf-consume/references/README.md`), which is why the defect was
|
||||
in the repository rather than only reachable in theory.
|
||||
"""
|
||||
written = okf_skill.generate(
|
||||
GOLDEN,
|
||||
out=tmp_path / "out",
|
||||
question="Hva sier veiledningen om krav?",
|
||||
force=True,
|
||||
)
|
||||
text = written.read_text(encoding="utf-8")
|
||||
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
|
||||
assert payload["withheld"] == [], "the premise of this arm no longer holds"
|
||||
assert "**0 concepts**" not in text
|
||||
assert "breaking point could not be measured" in text
|
||||
assert "At roughly" not in text
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue