The scanners cap by truncating: they return findings, so reading a prefix costs
detection in the tail and nothing else. The three transform surfaces return
*content*, where the same move is not available — a shortened document is silent
data loss, and a transformed prefix followed by an untransformed tail is a
bypass, since the attacker chooses where in the document the payload sits.
So they fail secure instead. Above MAX_INPUT_CHARS (1 000 000) sanitize, fence
and neutralize raise OversizeInputError. sanitize is step 1 of prepare_input and
only ever removes, so that one refusal bounds the whole input path.
OversizeInputError subclasses ContractViolation: a pipeline already bracketing
its quarantined stage keeps failing closed rather than meeting a type it has
never heard of. It inherits the alert-routable property too — sizes in the
message, refusing surface in details, no input in either.
Invariant now pinned across all three: returned text is always fully
transformed, or not returned at all.
Still uncapped and recorded in LIMITATIONS: scan_active_content called directly
(through scan_output it inherits that cap) and the okf link graph. Both are
detection-shaped, so truncate-and-flag transfers unchanged — mechanical, not
policy.
699 tests (+23), coverage 128/128 + 6/6, ReDoS sweep 0 candidates / 150.
0.3.3 swept 83 lexicon patterns arm by arm and left the other ten regex-bearing
modules on 0.3.2's hand-written rows. Generalising the sweep found three more,
and the two on the input path matter more than the count suggests: `sanitize` is
step 1 of `prepare_input`, and `MAX_SCAN_CHARS` is applied in `scan_lexicon` and
`scan_output` only, so there was no cap to extrapolate to. That missing input
cap is now a documented residual of its own -- extending it changes the contract
for existing callers and is not something to smuggle into a ReDoS patch.
Version synced in all four places + CHANGELOG. LIMITATIONS 30 -> 31 items;
the 0.3.3 entry claiming the lexicon sweep's scope is corrected in place, since
"the output path" was never the whole surface either.
676 tests, coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
Version synced across the four locked points (pyproject, __version__, README
badge, README install pin) + CHANGELOG.
Corrections this release carries, both measured rather than reasoned:
- docs/LIMITATIONS.md said the script-tag change removed "the last"
quadratic-backtracking site on the output path. It did not. Corrected, with
the 334.7s gate measurement that falsifies it.
- README's coverage line claimed 126/126 classes; the matrix reports 128/128.
Stale since before v0.3.2. Test badge was 642, actual 666.
New residual recorded (LIMITATIONS, now 30 items, README synced): the sweep
flags on timing above a 1.5ms noise floor at N=8000, so an arm hiding under it
could still cost ~23s at the cap. What this supports is "no arm worse than ~23s",
not "no quadratic arm remains" -- and the blind spot is demonstrated, since a
generic-payload pass found only one of the two patterns.
666 tests green, coverage matrix 128/128 with 6/6 gaps holding, exit 0.
Version sync in all four places (pyproject, __version__, README badge, README
install pin) plus the changelog entry for cff0437.
Two claims that were about to ship in the release note did not survive being
measured against a v0.3.1 worktree, and are corrected in LIMITATIONS first:
- "Report-only, so the cost is a review, not a block" was wrong twice. HIGH under
a low-trust preset is fail_secure, not a review -- report-only means the text is
never mutated, not that a finding cannot block. And the new script-tag false
positive costs no consumer a disposition at all: any text containing a literal
`<script>` already produced active:raw-html at HIGH on 0.3.1, so the same prose
disposed fail_secure under PRESET_USER_UPLOAD before this change and after it.
The fail-open that was closed is narrower for the same reason -- it existed only
in scan_lexicon called on its own; through either composed gate, raw-html
already caught the unclosed tag. What changed is the label, not the outcome.
- "The realistic long value is still caught by egress:jwt-token" was true and
hid the part that matters. Measured at the 257-char boundary: a generic long
password still trips entropy:base64-blob at CRITICAL, disposition unchanged.
A JWT used as a DB password is the case that moves -- its remaining detections
top out below CRITICAL, so the any-tier block is lost and PRESET_TRUSTED_SOURCE
drops from fail_secure to quarantine_review. PRESET_USER_UPLOAD still
fail_secures. Recorded as a behaviour change in the changelog, not buried.
The first probe for that boundary used a 300-char run of "A" and found nothing on
either version: all-same-char values are suppressed as placeholders. The probe was
wrong, not the pattern.
662 passed, coverage matrix exit 0, LIMITATIONS still 29 items = README's 29.
The output gate claimed LLM10 self-safety on the grounds that its patterns have
no nested quantifiers. True, and irrelevant: nesting is not what makes these
blow up. A run in front of a REQUIRED literal, reachable from a short anchor, is
enough -- crafted input repeats the anchor and never supplies the literal, so
every start position rescans the tail. Quadratic, not exponential, and the
max_scan_chars cap does not help: it bounds the input, and quadratic work on a
bounded input is still hours.
Measured, not argued. `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone; the
composed gate on that payload took 458.7s, extrapolating to ~5.7 hours at the
1_000_000-char input the gate itself accepts. Size-matched ordinary prose runs
0.31s, so the separation is 18x-660x -- unlike the blob in the neighbouring
test, which is the *faster* side of prose and never exercised backtracking.
Two fixes, chosen per pattern rather than uniformly:
- active_content + lexicon JSON (15 runs): exclude the character that opens the
pattern's own anchor (`[` for markdown, `<` for tags), so a run cannot reach
past the next start position and the per-start costs telescope. Verified to
cost no recall: long URLs, long alt text, and `<` inside a quoted attribute
all still match. Bounding instead would have been linear too but wrong here --
the content is attacker-controlled, so padding past a bound would be a
one-line bypass of the EchoLeak class this table exists to catch.
- connstr egress (4 runs): bound the password at MAX_CONNSTR_VALUE. The
exclusion fix is unavailable -- the anchor character is `/` and passwords
containing `/` are the common case (measured: they match today). The residual
miss is a credential over 256 chars; a token that long is still caught by
egress:jwt-token.
hybrid-xss:script-tag had neither option: its run is the script BODY, which may
legitimately contain `<`. It now matches the opening tag and drops the
`</script>` requirement. That also closes a fail-open -- `<script>alert(1)`
unclosed was silently missed -- at the cost of flagging prose that merely
mentions `<script>`, now documented.
Found by the composed-gate test staying red after every individual scanner was
already linear: the lexicon's six html-obfuscation patterns were the remaining
813x. A per-scanner test alone would have shipped that.
662 passed (was 642), and faster than before the fix.
Yesterday's-shape correction, one commit old. e25de56 explained the 2400-vs-2401
gap by saying the corpus "grew from 389 files to 394 between the two runs", so
the larger snapshot returned one fewer URL -- two variables moving at once.
The consumer self-corrected the same day and the correction reproduces here
against their tree, not against their message:
find skills -path '*/references/*' -name '*.md' 389
find skills -name 'SKILL.md' 5
find skills -name '*.md' 394
394 is every .md under skills/; 389 is the references/** path the measurement
actually scoped to. One snapshot counted two ways, not two snapshots taken at
different times -- and the 5 SKILL.md files contributed no unique URLs.
So the corpus never grew, only the script moved, and the one-URL gap is back to
unexplained (the original harvester is gone from a scratchpad). Everything else
in their measurement stands unchanged: 2400 distinct URLs, 0 firings on the
entropy branch, worst legitimate token H=4.301 at len=78.
The do-not-merge rule is unaffected and now rests on less: two counts of the
same knowledge base, one URL apart, no account of why. Never summed, never
quoted as one figure.
27 items unchanged, README stays in sync.
The previous commit said the two corpus counts differ by harvester over the same
knowledge base. Checked the provenance rather than leaving it asserted: 2401
traces to ms-ai-architect's 2026-07-26 message, 2400 to their 07-31 re-run, so
the attribution was right -- but the knowledge base was not held constant. Their
file count moved 389 -> 394 between the runs.
That inverts how the delta reads. A larger snapshot returning one fewer distinct
URL is not a rounding difference between two scripts; two variables moved at
once, so neither count explains the other and the one-URL gap is not evidence of
anything in particular.
Strengthens the do-not-merge rule instead of weakening it. The earlier wording
would have let a reader treat the counts as near-identical measurements of one
thing, which is the merge it was written to prevent.
ms-ai-architect re-measured after retracting their entropy refutation (their
original finding scored whole filenames as one unit, so it was never a finding
about our rule). The retraction confirms the branch; the re-measurement produced
a number we did not have, and it goes against us.
The sharpest legitimate token in their 2400-URL corpus is a 78-character
percent-escaped lovdata title at H=4.301. That is 0.099 from the 4.4 floor where
this bullet previously implied 0.36 -- the mechanism was already ours, but the
worked example understated how thin the margin gets in real Norwegian
government paths.
Verified against our own code before recording, not reconstructed: our real
_URL_TOKEN_RE emits it as a single 78-character token, our shannon_entropy
returns 4.301, and is_ordinary_url returns False. Their three calibration
examples also reproduce to three decimals here.
Threshold stays. Their own conclusion too -- moot per URL, since the % rule
already disqualifies it before entropy is consulted.
Records the 2400-vs-2401 provenance split explicitly. The counts come from two
different harvesters over the same knowledge base and differ by one URL; the
neighbouring hex bullet cites 2401. Left unstated, a later reader would
reasonably merge them into one corpus figure, which is exactly the
prose-vs-measurement conflation this file exists to prevent.
okf delivered the code read we asked for: materialize.py:71 is
`[^a-z0-9]+ -> "-"`, a whitelist, so their generated-path percent-escape
exposure is structurally zero. Verified independently against their source
rather than taken on the message: the regex is where they said, both their
worked examples reproduce against the real implementation, and the only
urllib.parse.quote in their tree builds a sqlite file: URI that never touches
a filename.
That retires the "still open for the one consumer whose slugger we have not
seen" clause. The Loekkene example already in this bullet was linkedin-studio's
run output, not a prediction -- checked before rewriting, because the two
clauses read as if they contradicted each other and only one was stale.
Adds the honest half that closing the question exposes: we have now read two
whitelist sluggers and zero encodeURIComponent-class ones, so the
"produces escapes systematically" arm of this axis is still a prediction from
the transform, not a field observation.
The {tenant} placeholder result goes on the generated-path side, deliberately
not appended to the %7B sentence -- that one is a link-corpus harvesting
artifact, and merging them would re-conflate the two axes this bullet exists
to separate.
No test is owed here: the claim pins okf's regex, which this suite cannot
assert. It is verifiable only by re-reading their source.
A consumer corrected a claim we shipped this morning. We wrote that both
zero-measuring corpora were English, and used that to explain the third corpus's
10 Norwegian hits as a sampling bias. One of those two is a Norwegian repo with
Norwegian content, so the stated explanation was wrong.
Their measurement separates two axes we had conflated. For third-party LINK
corpora -- URLs collected from other people's sites -- language does predict, and
the Norwegian legal/government sources are where the escapes were. For GENERATED
paths the predictor is slugger class: a whitelist slugger ([^a-z0-9]+ -> "-")
cannot emit an escape in any language because it discards the character before
anything encodes it, while encodeURIComponent produces them systematically as soon
as titles are non-ASCII. Their own slugger returns a structural zero on Norwegian
input, not a statistical one.
So non-ASCII language is a confounder for encode-vs-whitelist, and slugger class is
the thing that is actually testable at a consumer -- a one-line code read rather
than a corpus census. Reframed accordingly, with the earlier conflation named so
the correction is visible rather than silently rewritten.
A consumer measured our frontmatter gate against 0.2.0 and asked us to confirm
against a newer guard, correctly noting that a divergence would be a version
difference rather than a contradiction. Measured at 0.3.1: no divergence. Both
their results reproduce exactly.
Two things are sharper than what we documented yesterday:
ALL THREE ROUTES TO A MAPPING FAIL, each on a different rule -- flow {k: v} on the
disallowed value-start indicator, block on the nested-mapping check, dotted keys on
the key pattern. So the mapping CLASS has no expressible form; it is not a choice
between two shapes where one is better. That matters because OKF v0.2's `generated`
IS a mapping (`by` is required when it is present), so it cannot be expressed at
all. Pinned by a test asserting the three failures stay distinct.
T2 RUNS ON DOOR C ONLY. parse_frontmatter is referenced nowhere in the door A/B
persist path, so the same frontmatter that FAIL_SECUREs through import_bundle
passes screen_output unremarked. The grammar bounds what a consumer can RECEIVE,
never what a producer can EMIT -- which is the question a consumer's emitter design
was blocked on.
Also corrects one of our own rows: we reported "sources block list" as a single
FAIL_SECURE case. The parser distinguishes three shapes -- flat scalars parse
correctly, one key per item misparses silently to a string, two keys per item hard-
rejects. A one-element `verified:` block list passes today.
631 -> 642 passed. README item count synced.
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.
docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.
LIMITATIONS.md brought current with the field measurements:
- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
selector. No parameter-level remedy covers any two, which moves this from a
conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
where two keys hard-reject, so a pointer can ride past the resource allowlist.
Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
both backward-breaking migration targets are nested. Fail-secure, but a
compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.
New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
Two consumers measured the 0.3.1 URL-shape rule against real corpora on the
same day. Record what they found, and pin the dispositions both of them
inferred wrongly.
Percent-escape FP: zero occurrences across both corpora (0/347 external URLs
in a 527-document vendor-docs corpus, 0/81 in a capture store). This bounds
the shape rather than closing it -- a consumer that slugs filenames from
titles produces %20 systematically, and that corpus is still unmeasured.
Query FP: the over-block that actually occurs. The two corpora hit disjoint
benign populations -- 16/16 publisher-authored utm_* tracking versus 35/35
content identity (?v=, ?channel_id=) where the parameter IS the resource.
An allowlist keyed on tracking-parameter names resolves the first entirely
and the second not at all, so no parameter-level remedy covers both. That
is input to the 0.4.0 axis-separation scope, not a fix here.
Both consumers reported dispositions they had inferred rather than run, and
both were wrong: a query-carrying link is MEDIUM, so it is held or warned,
never hard-failed. tests/test_wiring.py now pins that, plus the active-tag
gate -- counting raw HTML tags overcounts what the gate flags, since
formatting markup is inert and only name/on*=/URL-attr tags are active.
593 passed.
v0.3.0 made the untrusted upload path unusable: measured on both doors, an
ordinary remote image fail_secure'd and an ordinary link/autolink/refdef
quarantined, so only documents without external references persisted.
Two independent defects compounded; neither fix works alone:
1. `markdown-image: HIGH` fired on any external image. The exfil primitive is a
URL that moves bytes outward, not an image. `is_ordinary_url` now grades on
shape - http(s)/protocol-relative, no query, no userinfo, no percent-escape,
no opaque host label or path segment -> LOW; anything data-carrying keeps the
carrier's severity. raw-html and data: URIs stay HIGH unconditionally.
Opacity reuses entropy's primitives; floors calibrated against real doc URLs
(worst legit token H=4.08, exfil segments 4.36-4.54) and frozen in
calibration.
2. The quarantine_default floor fired on ANY finding, a premise that broke when
every ordinary link became a finding. It now fires at MEDIUM+ - a no-op for
every detector that shipped before 0.3.0 (no LOW/INFO exists), which is what
makes this a patch rather than a minor.
The corpus blind spot that let this pass 522 green tests is closed: the FP
corpus carries realistic markdown and is asserted on the OUTPUT gate under
PRESET_USER_UPLOAD, with a counter-corpus of exfil-shaped URLs that must still
block. Beaconing and short opaque segments are conceded in LIMITATIONS and
asserted by the coverage matrix rather than papered over.
No new public API; no new preset (0.4.0 work); allow_reserved default unchanged.
Answers the gap that the README said what it does NOT stop (a long limitations
section) but never plainly listed what it DOES. Add a 'What it protects against'
section high up: attack classes grouped by OWASP anchor (LLM01 injection + 83
lexicon classes + carriers, LLM02 egress, LLM05 EchoLeak, LLM06 agency, LLM10
fail-secure, OKF T1-T6, container front-end), each driven by a live coverage-matrix
payload. Move the full honest-limitations list to docs/LIMITATIONS.md; README keeps
a high-impact summary + link. Net: protection and limits read in balance, 261 -> 216
lines. Coverage 126/126 and 522 tests unchanged; every class listed is real.
