OKF v0.2 writes its whole trust and provenance layer as mappings, and T2 gave
the mapping class no expressible form. A consumer measured 0 of 53 upstream
concepts through the gate on 0.3.4, 1.0.0 and 1.1.0. That was a contract
collision, not a calibration setting: SPEC.md @ 62432a09 uses flow mappings in
its own 5.1/5.2 examples, and 11 carries a hard MUST for consumers ("MUST treat
a bare `verified` mapping as a one-element list") that presupposes they parse.
Admitted: a flow mapping, as a value or as a block-list item, whose every key is
on a nine-name allowlist and whose every leaf is a plain scalar run through the
unchanged dangerous-value and mapping-construct predicates. The form is safe
because the allowlist inspects every key -- the blanket refusal was the
enforcement, not the point.
Refused, each on its own rule and ground-truthed against PyYAML 6.0.3: a key off
the allowlist, a nested collection, a quoted leaf, a duplicate key, an empty or
unclosed mapping, trailing junk, and {a:b} (which PyYAML reads as the KEY a:b).
A refused mapping still raises rather than degrading into a string, so the 1.1.0
type-confusion defect is not reopened, and the block, dotted and inline-colon
routes still raise.
`resource` is deliberately off the allowlist though SPEC.md 5.1 names it inside
a `sources` entry: it is a pointer rather than a label and the only key T3
exists for, so admitting it would let `executor: { resource: skills/run.md }`
carry an executable-code pointer through in typed clothes -- the door-C route
closed in 1.1.0. It costs nothing today, because the conformant carrier for
sources[].resource is a block sequence of block mappings, which stays refused.
Mapping leaves are scanned like every other frontmatter value (T1). Coverage
matrix 130/130 (new row: the off-allowlist key). No exported surface, detector
behaviour or calibration changed.
T2 gives the mapping class no expressible form by design. Two routes escaped
that by parsing "successfully" into the wrong TYPE instead of raising:
sources:\n - uri: https://e.com/a -> the string 'uri: https://e.com/a'
attester: resource: attesters/x.py -> the string 'resource: attesters/x.py'
Only the first was documented (LIMITATIONS.md:43). The inline second colon was
found by measurement while closing it -- a real YAML parser refuses that line
outright, ours accepted it. Shipping the list half alone would have left a
LIMITATIONS rewrite that overclaims.
Same consequence either way: a pointer parked in a degraded mapping rides
through in a key the `resource` allowlist never inspects, and mode-b
import_bundle wrote the merged concept verbatim (WARN). Both now FAIL_SECURE at
T2, before the allowlist is reached.
The boundary is where YAML puts it, ground-truthed against PyYAML 6.0.3 rather
than reasoned: ": " and a trailing ":" are exactly the two shapes where a plain
scalar becomes a mapping. A colon carrying neither a space nor a line end opens
no mapping -- domain:security and https://e.com:8443/a still parse -- and a
quoted scalar is still a scalar. Over-blocking a conformant bundle is itself a
failure mode, so the seven admitted shapes get rows of their own.
Iron Law: the four rejected rows and both import_bundle rows were written
first and seen red (7 failures, each DID NOT RAISE) before okf.py was touched.
Suite 792 -> 802. LIMITATIONS stays at 35: the bullet is reworded, not
retired -- the restricted grammar is still a limitation, the silent misparse
is no longer part of it.
`5667063` moved test_output.py's ReDoS bounds off the wall clock, because a
loaded machine steals wall seconds without adding any cycles and two rows
failed at 2.24s / 3.66s against a 2.0s bound while census had the CPU. The
remaining ten bounds in five other files still ran on `time.monotonic()` and
carried the same defect. They now share ONE clock.
The clock is IMPORTED, not copied: `tests/redos_clock.py`. Five private copies
would leave four of them unpinned -- the instrument test
(test_the_redos_clock_ignores_time_this_process_did_not_spend) can only pin the
implementation it calls, and the suite already holds that rule for the code it
measures.
Every ported row was verified the only way a time bound can be: the vulnerable
form patched back in, red demanded, `git checkout --` after. Measured against
the 2.0s bound (3.0s for the url arm):
active_content long-attr `{0,63}` -> `*` RED
neutralize long-attr same patch RED
output gate long-attr same patch 12.41s
okf link graph `[^\]\[]` -> `[^\]]` 6.91s
sanitize comment str.find -> `<!--.*?-->` 17.56s
lexicon md-link-anchor-text 319.14s
lexicon md-link-anchor-url 8.55s
lexicon md-link-ref-comment 37.82s
Two rows did not go red, for two different reasons.
test_sanitize.py::test_legitimate_comment_heavy_document is the legitimate SIDE
of a separation, not a second pin on the defect: closed comments never withhold
the required literal, so the lazy form runs it in 0.016s. Recorded in place.
test_lexicon.py::test_redos_pathological_subagent_input_returns_fast is DEAD --
the same zero-signal shape the `<a ` carrier had, found by the same method. The
seed form is `(?:.*?\s+)?` (llm-security 7.8.0, injection-patterns.mjs:84) and
this repo has never carried it: the bounded `{0,12}?` port is in the pattern
table's first commit. Patched in by hand at the row's own size: shipped 0.135s
vs seed 0.113s, separation 1.2x. Not the keyword gate either -- a variant that
reaches the inner branch stays linear over four doublings (exponent ~1.0),
because the nesting is one lazy run inside an OPTIONAL group, never a repeated
one. Left standing with the measurement written into it; picking a new carrier
is an operator call, like the wall-clock row above it.
The dead sibling row named in STATE is fixed: test_active_content.py's
long-attribute row swaps carrier `<a ` -> `<script `, for the reason `5667063`
established on its composed-gate twin -- 0.7.0's own no-URL narrowing put `<a>`
in `_URL_AFFORDANCE_TAGS`, so the tag returns inert BEFORE its body reaches the
arm the row guards. Re-measured here, not inherited: `<a ` 0.041s and NO
findings against the vulnerable form; `<script ` 19.349s against 0.052s
shipped, 373x apart.
`test_pathological_input_returns_within_a_bound` deliberately keeps its wall
clock (operator decision): it claims to catch a hang, and only a wall clock
catches one.
792 tests, 129/129, 6/6.
`scan_active_content` called directly and `okf.link_graph` were the two surfaces
still reading attacker-supplied text with no cap — the first reached by an
adapter that wants the active-content classes alone, the second running a
`findall` over every body in a bundle. Both are detection-shaped, so they
truncate and flag rather than raise the way the transform surfaces do: what a
detector shortens is its own coverage, not the caller's content.
Truncation is only honest if it is visible, so neither goes quiet: the scanner
emits `active:oversize-input` (LLM10), and `link_graph` records
`(from_id, body_length)` in `LinkGraphResult.truncated` — the field that lets a
caller tell "no links past here" from "no links read past here".
Reached through `scan_output`, the text is already under that surface's cap and
`max_scan_chars` is now passed down, so the flag is raised once, there.
The generalised sweep found what 0.3.2's hand-written rows missed. All three are
the documented class -- a run in front of a required literal that never arrives,
so every start position rescans the tail -- and all three are worse than the
0.3.3 findings, because `sanitize`, `neutralize`, `scan_active_content` and the
okf link graph apply NO input cap. `scan_lexicon`/`scan_output` are the only
entry points that do, so there is no ceiling to extrapolate to.
sanitize._HTML_COMMENT_RE `<!--`*100_000 20.1s, exponent 1.96-2.14
active_content.URL_IN_TEXT_RE `<a `+`A`*100_000 12.99s / 14.9s, exponent ~2.0
okf._MD_LINK_RE `[`*100_000 7.1s, exponent 1.99-2.05
Each fix is the one the pattern's own shape allows, not a copied choice:
- The comment stripper drops the regex for `str.find`. Excluding `<` would lose
every comment containing markup; bounding the run would be a carrier bypass
of the exact construct the stripper exists to remove.
- `URL_IN_TEXT_RE` bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
Bounding is safe *here* only because it is a defanger inside a tag already
flagged `active:raw-html`. A lookbehind was measured too and rejected: it
drops `-http://evil.com`, a one-character evasion. Bounded: 0.185s at 1M.
- `_MD_LINK_RE` excludes `[`, matching `active_content.MD_LINK_RE` exactly,
including the nested-label trade already documented there.
`sanitize` claimed "no catastrophic backtracking" in a comment; that claim was
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
676 tests (+10), coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
The okf destination run gets no row: `[^)\s]+` cannot fail, so a row for it
could never go red.
A consumer measured our frontmatter gate against 0.2.0 and asked us to confirm
against a newer guard, correctly noting that a divergence would be a version
difference rather than a contradiction. Measured at 0.3.1: no divergence. Both
their results reproduce exactly.
Two things are sharper than what we documented yesterday:
ALL THREE ROUTES TO A MAPPING FAIL, each on a different rule -- flow {k: v} on the
disallowed value-start indicator, block on the nested-mapping check, dotted keys on
the key pattern. So the mapping CLASS has no expressible form; it is not a choice
between two shapes where one is better. That matters because OKF v0.2's `generated`
IS a mapping (`by` is required when it is present), so it cannot be expressed at
all. Pinned by a test asserting the three failures stay distinct.
T2 RUNS ON DOOR C ONLY. parse_frontmatter is referenced nowhere in the door A/B
persist path, so the same frontmatter that FAIL_SECUREs through import_bundle
passes screen_output unremarked. The grammar bounds what a consumer can RECEIVE,
never what a producer can EMIT -- which is the question a consumer's emitter design
was blocked on.
Also corrects one of our own rows: we reported "sources block list" as a single
FAIL_SECURE case. The parser distinguishes three shapes -- flat scalars parse
correctly, one key per item misparses silently to a string, two keys per item hard-
rejects. A one-element `verified:` block list passes today.
631 -> 642 passed. README item count synced.
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.
docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.
LIMITATIONS.md brought current with the field measurements:
- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
selector. No parameter-level remedy covers any two, which moves this from a
conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
where two keys hard-reject, so a pointer can ride past the resource allowlist.
Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
both backward-breaking migration targets are nested. Fail-secure, but a
compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.
New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
A received OKF bundle MAY legitimately carry index.md (directory listing, read
first under progressive disclosure) and log.md (update history) at any level
(spec §3.1/§6/§7). import_bundle previously hard-rejected those basenames in the
T4 path gate, so a conformant third-party bundle was over-blocked in full
(FAIL_SECURE) — and because the reject fired before scan_concept, index.md's
body (the highest-priority injection surface) was never scanned.
import_bundle now defaults allow_reserved=True: reserved basenames are scanned
as structural files (path-safety checks — traversal / absolute / backslash / .md
— still apply). The shadow-reject (an *upload* masquerading as index.md) is
preserved: the front-end passes allow_reserved=False so a materialized upload
landing on a reserved basename is still refused. That front-end opt-in was
required to keep the shadow-reject once the default flipped (not in the plan's
Filer set; traced from the code).
- okf.py: validate_concept_path/_validate_concept/import_bundle gain the
keyword; validate_concept_path default stays False (strict standalone).
- tests: +3 (legit index/log admit; injection in index.md body caught;
okf_version frontmatter admits). Per-concept-iteration test switched to a
traversal vector; mode-b showcase's index.md surface reframed from
reserved-name-reject to index.md-body-scan.
- README honest-limits + CLAUDE.md context note the mode-b/upload distinction.
Suite: 341 -> 344 passed. Core invariant intact (dependencies=[]).
