Kjell Tore Guttormsen
6e7c8d2b98
Order 20260906T213322Z: measure and propose, no src, no bump. Measured against SPEC _okf-canonical @ ad30107 and the corpus _okf-upstream @ 3fcbb9f (denominator 53, extracted at the pin because the work tree had moved on to 9a15b13). Three claims in the tags/description entry are stale against the 1.3.0 parser. The parser does have a sequence type -- _consume_block_list parses block lists of scalars -- what it lacks is tolerance for the flush-left indentation 36/53 of the corpus uses. Removing tags alone now lets 6/53 pass, not the 4/53 recorded, and the entry's headline claim does not hold at all: description alone unblocks 0/53, and with all three surface forms closed 44/53 still stop on generated as a top-level block mapping. The entry oversells its own reach by a factor of seven. The SPEC has no depth rule. no-nesting-past-depth-1 is entirely ours; the conformance floor is only "a parseable YAML frontmatter block" (SS11.1). Candidates were measured by normalizing the surface form onto a shape the parser already accepts, then importing parse_frontmatter -- the predicate is never re-implemented, only the input's spelling is rewritten. P1 (scalar flow sequence) takes the corpus from 0/53 to 6/53 without spending depth-1; P2 and P3 buy 0/53 each and 6/53 stacked on P1. Recommendation is P1 alone, and the note says out loud that this does not open the corpus. The label "punkt 44" is not grep-able: it came from a count-after-insertion, and the entry is today the 12th of 45 at docs/LIMITATIONS.md:126. No src change, no version bump, no push. 868 passed, coverage 130/130 + 6/6 gaps, redos sweep exit 0 -- all after git add.
59 lines
2 KiB
Python
59 lines
2 KiB
Python
"""Shape census of the pinned OKF corpus: which surface form each key is written in."""
|
|
import os, re, sys, collections
|
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
|
|
from llm_ingestion_guard import okf
|
|
|
|
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
|
|
|
|
def docs():
|
|
out = []
|
|
for dirpath, _d, fns in os.walk(ROOT):
|
|
for fn in sorted(fns):
|
|
if fn.endswith(".md") and fn not in ("index.md", "log.md"):
|
|
out.append(os.path.join(dirpath, fn))
|
|
return sorted(out)
|
|
|
|
def split_fm(text):
|
|
lines = text.split("\n")
|
|
if not lines or lines[0].strip() != "---":
|
|
return None, None
|
|
for i in range(1, len(lines)):
|
|
if lines[i].strip() == "---":
|
|
return lines[1:i], lines[i + 1:]
|
|
return None, None
|
|
|
|
def key_shape(fm, key):
|
|
for i, raw in enumerate(fm):
|
|
m = re.match(r"^(%s):(.*)$" % re.escape(key), raw)
|
|
if not m:
|
|
continue
|
|
val = m.group(2).strip()
|
|
follow = fm[i + 1] if i + 1 < len(fm) else ""
|
|
if val.startswith("["):
|
|
return "flow-sequence"
|
|
if val.startswith("{"):
|
|
return "flow-mapping"
|
|
if val:
|
|
if follow[:1] in (" ", "\t") and not follow.strip().startswith("- "):
|
|
return "scalar+continuation"
|
|
return "single-line-scalar"
|
|
if follow.strip().startswith("- "):
|
|
return "block-seq-indented" if follow[:1] in (" ", "\t") else "block-seq-flush"
|
|
if follow[:1] in (" ", "\t"):
|
|
return "block-mapping"
|
|
return "empty"
|
|
return "ABSENT"
|
|
|
|
def main():
|
|
paths = docs()
|
|
print("denominator: %d" % len(paths))
|
|
for key in ("tags", "description", "generated", "verified", "sources"):
|
|
c = collections.Counter()
|
|
for p in paths:
|
|
fm, _ = split_fm(open(p, encoding="utf-8").read())
|
|
c[key_shape(fm, key)] += 1
|
|
print("\n%s:" % key)
|
|
for shape, n in c.most_common():
|
|
print(" %-22s %d/%d" % (shape, n, len(paths)))
|
|
|
|
main()
|