The previous commit said the two corpus counts differ by harvester over the same
knowledge base. Checked the provenance rather than leaving it asserted: 2401
traces to ms-ai-architect's 2026-07-26 message, 2400 to their 07-31 re-run, so
the attribution was right -- but the knowledge base was not held constant. Their
file count moved 389 -> 394 between the runs.
That inverts how the delta reads. A larger snapshot returning one fewer distinct
URL is not a rounding difference between two scripts; two variables moved at
once, so neither count explains the other and the one-URL gap is not evidence of
anything in particular.
Strengthens the do-not-merge rule instead of weakening it. The earlier wording
would have let a reader treat the counts as near-identical measurements of one
thing, which is the merge it was written to prevent.
ms-ai-architect re-measured after retracting their entropy refutation (their
original finding scored whole filenames as one unit, so it was never a finding
about our rule). The retraction confirms the branch; the re-measurement produced
a number we did not have, and it goes against us.
The sharpest legitimate token in their 2400-URL corpus is a 78-character
percent-escaped lovdata title at H=4.301. That is 0.099 from the 4.4 floor where
this bullet previously implied 0.36 -- the mechanism was already ours, but the
worked example understated how thin the margin gets in real Norwegian
government paths.
Verified against our own code before recording, not reconstructed: our real
_URL_TOKEN_RE emits it as a single 78-character token, our shannon_entropy
returns 4.301, and is_ordinary_url returns False. Their three calibration
examples also reproduce to three decimals here.
Threshold stays. Their own conclusion too -- moot per URL, since the % rule
already disqualifies it before entropy is consulted.
Records the 2400-vs-2401 provenance split explicitly. The counts come from two
different harvesters over the same knowledge base and differ by one URL; the
neighbouring hex bullet cites 2401. Left unstated, a later reader would
reasonably merge them into one corpus figure, which is exactly the
prose-vs-measurement conflation this file exists to prevent.
okf delivered the code read we asked for: materialize.py:71 is
`[^a-z0-9]+ -> "-"`, a whitelist, so their generated-path percent-escape
exposure is structurally zero. Verified independently against their source
rather than taken on the message: the regex is where they said, both their
worked examples reproduce against the real implementation, and the only
urllib.parse.quote in their tree builds a sqlite file: URI that never touches
a filename.
That retires the "still open for the one consumer whose slugger we have not
seen" clause. The Loekkene example already in this bullet was linkedin-studio's
run output, not a prediction -- checked before rewriting, because the two
clauses read as if they contradicted each other and only one was stale.
Adds the honest half that closing the question exposes: we have now read two
whitelist sluggers and zero encodeURIComponent-class ones, so the
"produces escapes systematically" arm of this axis is still a prediction from
the transform, not a field observation.
The {tenant} placeholder result goes on the generated-path side, deliberately
not appended to the %7B sentence -- that one is a link-corpus harvesting
artifact, and merging them would re-conflate the two axes this bullet exists
to separate.
No test is owed here: the claim pins okf's regex, which this suite cannot
assert. It is verifiable only by re-reading their source.
A consumer corrected a claim we shipped this morning. We wrote that both
zero-measuring corpora were English, and used that to explain the third corpus's
10 Norwegian hits as a sampling bias. One of those two is a Norwegian repo with
Norwegian content, so the stated explanation was wrong.
Their measurement separates two axes we had conflated. For third-party LINK
corpora -- URLs collected from other people's sites -- language does predict, and
the Norwegian legal/government sources are where the escapes were. For GENERATED
paths the predictor is slugger class: a whitelist slugger ([^a-z0-9]+ -> "-")
cannot emit an escape in any language because it discards the character before
anything encodes it, while encodeURIComponent produces them systematically as soon
as titles are non-ASCII. Their own slugger returns a structural zero on Norwegian
input, not a statistical one.
So non-ASCII language is a confounder for encode-vs-whitelist, and slugger class is
the thing that is actually testable at a consumer -- a one-line code read rather
than a corpus census. Reframed accordingly, with the earlier conflation named so
the correction is visible rather than silently rewritten.
A consumer measured our frontmatter gate against 0.2.0 and asked us to confirm
against a newer guard, correctly noting that a divergence would be a version
difference rather than a contradiction. Measured at 0.3.1: no divergence. Both
their results reproduce exactly.
Two things are sharper than what we documented yesterday:
ALL THREE ROUTES TO A MAPPING FAIL, each on a different rule -- flow {k: v} on the
disallowed value-start indicator, block on the nested-mapping check, dotted keys on
the key pattern. So the mapping CLASS has no expressible form; it is not a choice
between two shapes where one is better. That matters because OKF v0.2's `generated`
IS a mapping (`by` is required when it is present), so it cannot be expressed at
all. Pinned by a test asserting the three failures stay distinct.
T2 RUNS ON DOOR C ONLY. parse_frontmatter is referenced nowhere in the door A/B
persist path, so the same frontmatter that FAIL_SECUREs through import_bundle
passes screen_output unremarked. The grammar bounds what a consumer can RECEIVE,
never what a producer can EMIT -- which is the question a consumer's emitter design
was blocked on.
Also corrects one of our own rows: we reported "sources block list" as a single
FAIL_SECURE case. The parser distinguishes three shapes -- flat scalars parse
correctly, one key per item misparses silently to a string, two keys per item hard-
rejects. A one-element `verified:` block list passes today.
631 -> 642 passed. README item count synced.
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.
docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.
LIMITATIONS.md brought current with the field measurements:
- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
selector. No parameter-level remedy covers any two, which moves this from a
conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
where two keys hard-reject, so a pointer can ride past the resource allowlist.
Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
both backward-breaking migration targets are nested. Fail-secure, but a
compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.
New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
Two consumers measured the 0.3.1 URL-shape rule against real corpora on the
same day. Record what they found, and pin the dispositions both of them
inferred wrongly.
Percent-escape FP: zero occurrences across both corpora (0/347 external URLs
in a 527-document vendor-docs corpus, 0/81 in a capture store). This bounds
the shape rather than closing it -- a consumer that slugs filenames from
titles produces %20 systematically, and that corpus is still unmeasured.
Query FP: the over-block that actually occurs. The two corpora hit disjoint
benign populations -- 16/16 publisher-authored utm_* tracking versus 35/35
content identity (?v=, ?channel_id=) where the parameter IS the resource.
An allowlist keyed on tracking-parameter names resolves the first entirely
and the second not at all, so no parameter-level remedy covers both. That
is input to the 0.4.0 axis-separation scope, not a fix here.
Both consumers reported dispositions they had inferred rather than run, and
both were wrong: a query-carrying link is MEDIUM, so it is held or warned,
never hard-failed. tests/test_wiring.py now pins that, plus the active-tag
gate -- counting raw HTML tags overcounts what the gate flags, since
formatting markup is inert and only name/on*=/URL-attr tags are active.
593 passed.
v0.3.0 made the untrusted upload path unusable: measured on both doors, an
ordinary remote image fail_secure'd and an ordinary link/autolink/refdef
quarantined, so only documents without external references persisted.
Two independent defects compounded; neither fix works alone:
1. `markdown-image: HIGH` fired on any external image. The exfil primitive is a
URL that moves bytes outward, not an image. `is_ordinary_url` now grades on
shape - http(s)/protocol-relative, no query, no userinfo, no percent-escape,
no opaque host label or path segment -> LOW; anything data-carrying keeps the
carrier's severity. raw-html and data: URIs stay HIGH unconditionally.
Opacity reuses entropy's primitives; floors calibrated against real doc URLs
(worst legit token H=4.08, exfil segments 4.36-4.54) and frozen in
calibration.
2. The quarantine_default floor fired on ANY finding, a premise that broke when
every ordinary link became a finding. It now fires at MEDIUM+ - a no-op for
every detector that shipped before 0.3.0 (no LOW/INFO exists), which is what
makes this a patch rather than a minor.
The corpus blind spot that let this pass 522 green tests is closed: the FP
corpus carries realistic markdown and is asserted on the OUTPUT gate under
PRESET_USER_UPLOAD, with a counter-corpus of exfil-shaped URLs that must still
block. Beaconing and short opaque segments are conceded in LIMITATIONS and
asserted by the coverage matrix rather than papered over.
No new public API; no new preset (0.4.0 work); allow_reserved default unchanged.
Answers the gap that the README said what it does NOT stop (a long limitations
section) but never plainly listed what it DOES. Add a 'What it protects against'
section high up: attack classes grouped by OWASP anchor (LLM01 injection + 83
lexicon classes + carriers, LLM02 egress, LLM05 EchoLeak, LLM06 agency, LLM10
fail-secure, OKF T1-T6, container front-end), each driven by a live coverage-matrix
payload. Move the full honest-limitations list to docs/LIMITATIONS.md; README keeps
a high-impact summary + link. Net: protection and limits read in balance, 261 -> 216
lines. Coverage 126/126 and 522 tests unchanged; every class listed is real.
