1
0
Fork 0

docs(okf): measure the tags/description gap against the pinned SPEC and corpus

Order 20260906T213322Z: measure and propose, no src, no bump. Measured against
SPEC _okf-canonical @ ad30107 and the corpus _okf-upstream @ 3fcbb9f (denominator
53, extracted at the pin because the work tree had moved on to 9a15b13).

Three claims in the tags/description entry are stale against the 1.3.0 parser.
The parser does have a sequence type -- _consume_block_list parses block lists of
scalars -- what it lacks is tolerance for the flush-left indentation 36/53 of the
corpus uses. Removing tags alone now lets 6/53 pass, not the 4/53 recorded, and
the entry's headline claim does not hold at all: description alone unblocks 0/53,
and with all three surface forms closed 44/53 still stop on generated as a
top-level block mapping. The entry oversells its own reach by a factor of seven.

The SPEC has no depth rule. no-nesting-past-depth-1 is entirely ours; the
conformance floor is only "a parseable YAML frontmatter block" (SS11.1).

Candidates were measured by normalizing the surface form onto a shape the parser
already accepts, then importing parse_frontmatter -- the predicate is never
re-implemented, only the input's spelling is rewritten. P1 (scalar flow sequence)
takes the corpus from 0/53 to 6/53 without spending depth-1; P2 and P3 buy 0/53
each and 6/53 stacked on P1. Recommendation is P1 alone, and the note says out
loud that this does not open the corpus.

The label "punkt 44" is not grep-able: it came from a count-after-insertion, and
the entry is today the 12th of 45 at docs/LIMITATIONS.md:126.

No src change, no version bump, no push. 868 passed, coverage 130/130 + 6/6 gaps,
redos sweep exit 0 -- all after git add.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-06 23:58:02 +02:00
commit 6e7c8d2b98
5 changed files with 579 additions and 0 deletions

45
scratchpad/measure.py Normal file
View file

@ -0,0 +1,45 @@
"""Measure the pinned OKF corpus against the CURRENT parser. No src/ changes."""
import os, sys, collections
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from llm_ingestion_guard import okf
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
def docs():
out = []
for dirpath, _dirnames, filenames in os.walk(ROOT):
for fn in sorted(filenames):
if not fn.endswith(".md"):
continue
if fn in ("index.md", "log.md"):
continue
out.append(os.path.join(dirpath, fn))
return sorted(out)
def main():
paths = docs()
ok, fail = [], []
reasons = collections.Counter()
for p in paths:
with open(p, encoding="utf-8") as fh:
text = fh.read()
try:
okf.parse_frontmatter(text)
ok.append(p)
except okf.OKFFrontmatterError as exc:
fail.append((p, str(exc)))
reasons[str(exc).split(":")[0]] += 1
print("denominator: %d documents" % len(paths))
print("parse OK : %d" % len(ok))
print("parse FAIL : %d" % len(fail))
print("--- failure classes ---")
for r, c in reasons.most_common():
print("%4d %s" % (c, r))
print("--- passing documents (positive control) ---")
for p in ok:
print(" " + os.path.relpath(p, ROOT))
print("--- first 8 failures verbatim ---")
for p, e in fail[:8]:
print(" %s\n %s" % (os.path.relpath(p, ROOT), e))
main()