The scanners cap by truncating: they return findings, so reading a prefix costs
detection in the tail and nothing else. The three transform surfaces return
*content*, where the same move is not available — a shortened document is silent
data loss, and a transformed prefix followed by an untransformed tail is a
bypass, since the attacker chooses where in the document the payload sits.
So they fail secure instead. Above MAX_INPUT_CHARS (1 000 000) sanitize, fence
and neutralize raise OversizeInputError. sanitize is step 1 of prepare_input and
only ever removes, so that one refusal bounds the whole input path.
OversizeInputError subclasses ContractViolation: a pipeline already bracketing
its quarantined stage keeps failing closed rather than meeting a type it has
never heard of. It inherits the alert-routable property too — sizes in the
message, refusing surface in details, no input in either.
Invariant now pinned across all three: returned text is always fully
transformed, or not returned at all.
Still uncapped and recorded in LIMITATIONS: scan_active_content called directly
(through scan_output it inherits that cap) and the okf link graph. Both are
detection-shaped, so truncate-and-flag transfers unchanged — mechanical, not
policy.
699 tests (+23), coverage 128/128 + 6/6, ReDoS sweep 0 candidates / 150.
0.3.3 swept 83 lexicon patterns arm by arm and left the other ten regex-bearing
modules on 0.3.2's hand-written rows. Generalising the sweep found three more,
and the two on the input path matter more than the count suggests: `sanitize` is
step 1 of `prepare_input`, and `MAX_SCAN_CHARS` is applied in `scan_lexicon` and
`scan_output` only, so there was no cap to extrapolate to. That missing input
cap is now a documented residual of its own -- extending it changes the contract
for existing callers and is not something to smuggle into a ReDoS patch.
Version synced in all four places + CHANGELOG. LIMITATIONS 30 -> 31 items;
the 0.3.3 entry claiming the lexicon sweep's scope is corrected in place, since
"the output path" was never the whole surface either.
676 tests, coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
The generalised sweep found what 0.3.2's hand-written rows missed. All three are
the documented class -- a run in front of a required literal that never arrives,
so every start position rescans the tail -- and all three are worse than the
0.3.3 findings, because `sanitize`, `neutralize`, `scan_active_content` and the
okf link graph apply NO input cap. `scan_lexicon`/`scan_output` are the only
entry points that do, so there is no ceiling to extrapolate to.
sanitize._HTML_COMMENT_RE `<!--`*100_000 20.1s, exponent 1.96-2.14
active_content.URL_IN_TEXT_RE `<a `+`A`*100_000 12.99s / 14.9s, exponent ~2.0
okf._MD_LINK_RE `[`*100_000 7.1s, exponent 1.99-2.05
Each fix is the one the pattern's own shape allows, not a copied choice:
- The comment stripper drops the regex for `str.find`. Excluding `<` would lose
every comment containing markup; bounding the run would be a carrier bypass
of the exact construct the stripper exists to remove.
- `URL_IN_TEXT_RE` bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
Bounding is safe *here* only because it is a defanger inside a tag already
flagged `active:raw-html`. A lookbehind was measured too and rejected: it
drops `-http://evil.com`, a one-character evasion. Bounded: 0.185s at 1M.
- `_MD_LINK_RE` excludes `[`, matching `active_content.MD_LINK_RE` exactly,
including the nested-label trade already documented there.
`sanitize` claimed "no catastrophic backtracking" in a comment; that claim was
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
676 tests (+10), coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
The okf destination run gets no row: `[^)\s]+` cannot fail, so a row for it
could never go red.
0.3.3 swept the 83 lexicon patterns arm-by-arm and left the other tables on
0.3.2's hand-written rows -- the class of sweep that misses arms. Generalise the
generator over all eleven regex-bearing modules and make the collector mechanical
on both axes: walk each module namespace for compiled patterns (a pattern added
later is swept without anyone listing it) and derive each one's call mode by
grepping the module source, since `.sub()`/`.finditer()` visit every start
position where `.match()` cannot. Patterns reached only through a helper
parameter get the worst mode, marked `*`, so the fallback can over-measure but
never miss.
Sweeps 137 patterns across 11 tables where 0.3.3 covered 83 in one.
The generator, not just its results. It synthesises an almost-match sample from
each regex's own skeleton and uses every token-boundary prefix as a repeating
unit, so `[`, `[system]` and `[system](` are each probed as separate arms.
Committed because the next step depends on it: 0.3.2 swept the other detector
tables with hand-crafted rows, and this session showed that class of sweep
misses arms -- a generic-payload pass found only one of the two patterns here.
Docstring carries the caveat the script cannot enforce: the ratio is computed
from two points, so a hit near the noise floor is a coin flip. Post-fix it flags
sub-agent:delegate-bypass at x2.8; measured over four doublings the exponent is
0.96-1.05, i.e. linear, 0.2s at the cap. Re-measure before concluding.
Not part of the package -- docs/, stdlib-only, no effect on the wheel.
Version synced across the four locked points (pyproject, __version__, README
badge, README install pin) + CHANGELOG.
Corrections this release carries, both measured rather than reasoned:
- docs/LIMITATIONS.md said the script-tag change removed "the last"
quadratic-backtracking site on the output path. It did not. Corrected, with
the 334.7s gate measurement that falsifies it.
- README's coverage line claimed 126/126 classes; the matrix reports 128/128.
Stale since before v0.3.2. Test badge was 642, actual 666.
New residual recorded (LIMITATIONS, now 30 items, README synced): the sweep
flags on timing above a 1.5ms noise floor at N=8000, so an arm hiding under it
could still cost ~23s at the cap. What this supports is "no arm worse than ~23s",
not "no quadratic arm remains" -- and the blind spot is demonstrated, since a
generic-payload pass found only one of the two patterns.
666 tests green, coverage matrix 128/128 with 6/6 gaps holding, exit 0.
The input-path duty `8deca93` scoped. All 83 lexicon patterns measured arm by
arm; two are quadratic, same shape 0.3.2 fixed -- a run in front of a required
literal that may cross the pattern's own opening anchor. Exponent 1.98 over five
points, so quadratic, not exponential.
markdown:link-anchor-injection `[` 1.91s @8k ~8.3h at the cap
markdown:link-anchor-injection `[system](` 0.006s @8k ~89s at the cap
markdown:link-ref-comment `[//]: # (` 0.22s @8k ~1.0h at the cap
Not input-path-only: `scan_lexicon` runs on the output path, so `scan_output("["
* 100_000)` took 334.7s. 0.3.2's "last quadratic site on the output path" was
false when written -- its sweep drove `[` only through `scan_active_content`.
Fix is anchor exclusion, not bounding (bounding attacker-controlled content is a
one-line bypass). The excluded char is `(`, not the obvious `[`: excluding `[`
drops `[//]: # (see [x] then ignore this)`, which no other pattern catches. The
anchors contain `(` too, so it telescopes at zero measured recall cost.
N is per row deliberately. The URL arm ran 0.9s UNFIXED at N=100_000 -- under the
2.0s bound, so that row could not have failed. Measured at N=300_000 instead,
where crafted (8.10s) and legitimate (0.926s) separate 8.8x.
Version sync in all four places (pyproject, __version__, README badge, README
install pin) plus the changelog entry for cff0437.
Two claims that were about to ship in the release note did not survive being
measured against a v0.3.1 worktree, and are corrected in LIMITATIONS first:
- "Report-only, so the cost is a review, not a block" was wrong twice. HIGH under
a low-trust preset is fail_secure, not a review -- report-only means the text is
never mutated, not that a finding cannot block. And the new script-tag false
positive costs no consumer a disposition at all: any text containing a literal
`<script>` already produced active:raw-html at HIGH on 0.3.1, so the same prose
disposed fail_secure under PRESET_USER_UPLOAD before this change and after it.
The fail-open that was closed is narrower for the same reason -- it existed only
in scan_lexicon called on its own; through either composed gate, raw-html
already caught the unclosed tag. What changed is the label, not the outcome.
- "The realistic long value is still caught by egress:jwt-token" was true and
hid the part that matters. Measured at the 257-char boundary: a generic long
password still trips entropy:base64-blob at CRITICAL, disposition unchanged.
A JWT used as a DB password is the case that moves -- its remaining detections
top out below CRITICAL, so the any-tier block is lost and PRESET_TRUSTED_SOURCE
drops from fail_secure to quarantine_review. PRESET_USER_UPLOAD still
fail_secures. Recorded as a behaviour change in the changelog, not buried.
The first probe for that boundary used a 300-char run of "A" and found nothing on
either version: all-same-char values are suppressed as placeholders. The probe was
wrong, not the pattern.
662 passed, coverage matrix exit 0, LIMITATIONS still 29 items = README's 29.
The output gate claimed LLM10 self-safety on the grounds that its patterns have
no nested quantifiers. True, and irrelevant: nesting is not what makes these
blow up. A run in front of a REQUIRED literal, reachable from a short anchor, is
enough -- crafted input repeats the anchor and never supplies the literal, so
every start position rescans the tail. Quadratic, not exponential, and the
max_scan_chars cap does not help: it bounds the input, and quadratic work on a
bounded input is still hours.
Measured, not argued. `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone; the
composed gate on that payload took 458.7s, extrapolating to ~5.7 hours at the
1_000_000-char input the gate itself accepts. Size-matched ordinary prose runs
0.31s, so the separation is 18x-660x -- unlike the blob in the neighbouring
test, which is the *faster* side of prose and never exercised backtracking.
Two fixes, chosen per pattern rather than uniformly:
- active_content + lexicon JSON (15 runs): exclude the character that opens the
pattern's own anchor (`[` for markdown, `<` for tags), so a run cannot reach
past the next start position and the per-start costs telescope. Verified to
cost no recall: long URLs, long alt text, and `<` inside a quoted attribute
all still match. Bounding instead would have been linear too but wrong here --
the content is attacker-controlled, so padding past a bound would be a
one-line bypass of the EchoLeak class this table exists to catch.
- connstr egress (4 runs): bound the password at MAX_CONNSTR_VALUE. The
exclusion fix is unavailable -- the anchor character is `/` and passwords
containing `/` are the common case (measured: they match today). The residual
miss is a credential over 256 chars; a token that long is still caught by
egress:jwt-token.
hybrid-xss:script-tag had neither option: its run is the script BODY, which may
legitimately contain `<`. It now matches the opening tag and drops the
`</script>` requirement. That also closes a fail-open -- `<script>alert(1)`
unclosed was silently missed -- at the cost of flagging prose that merely
mentions `<script>`, now documented.
Found by the composed-gate test staying red after every individual scanner was
already linear: the lexicon's six html-obfuscation patterns were the remaining
813x. A per-scanner test alone would have shipped that.
662 passed (was 642), and faster than before the fix.
PLAN.md's test-plan line read as if both scan paths carried ReDoS coverage.
f74245f measured that the output path's "pathological" payload is the FASTER
side against size-matched ordinary prose (0.93x / 0.96x, order swapped), so it
does not exercise catastrophic backtracking at all.
Leaving the line unscoped would be the same defect c0899e9 just retracted in
LIMITATIONS.md: a committed claim resting on a premise now known to be false.
Names which path carries which duty and says plainly that a crafted payload for
the output regexes does not exist. No new limitation bullet -- 27 stays 27.
Yesterday's-shape correction, one commit old. e25de56 explained the 2400-vs-2401
gap by saying the corpus "grew from 389 files to 394 between the two runs", so
the larger snapshot returned one fewer URL -- two variables moving at once.
The consumer self-corrected the same day and the correction reproduces here
against their tree, not against their message:
find skills -path '*/references/*' -name '*.md' 389
find skills -name 'SKILL.md' 5
find skills -name '*.md' 394
394 is every .md under skills/; 389 is the references/** path the measurement
actually scoped to. One snapshot counted two ways, not two snapshots taken at
different times -- and the 5 SKILL.md files contributed no unique URLs.
So the corpus never grew, only the script moved, and the one-URL gap is back to
unexplained (the original harvester is gone from a scratchpad). Everything else
in their measurement stands unchanged: 2400 distinct URLs, 0 firings on the
entropy branch, worst legitimate token H=4.301 at len=78.
The do-not-merge rule is unaffected and now rests on less: two counts of the
same knowledge base, one URL apart, no account of why. Never summed, never
quoted as one figure.
27 items unchanged, README stays in sync.
Measured this session, the reason the bound was flaky at all:
size-matched ordinary prose 3.50-4.20s (cold up to ~4.2s)
the "pathological" blob 3.35-3.86s warm, 4.70s cold
ratio patho/ordinary 0.93x, and 0.96x with the order swapped
The blob is the FASTER side. The test therefore never proved what its comment
claimed -- it measured throughput on 1MB of text, not catastrophic backtracking,
and could only fail when the process was cold or the machine loaded. That is
exactly how it failed: 4.39 / 4.46 / 4.92s against a 5.0s bound.
Catastrophic backtracking IS covered, elsewhere and properly, by
test_lexicon.py::test_redos_pathological_subagent_input_returns_fast, which
crafts against a known-bad nested `.*?` pattern (0.137s against a 2.0s bound --
14x headroom, measured, left untouched).
So: raise the bound rather than make it relative. A relative bound was the other
option on the table and the measurement killed it -- with a ratio below 1.0 you
would need k >= 3 to clear the noise, and an assertion with 3x headroom over a
case already under 1.0 can never fire. Vacuous, plus it would add a second
cold-start asymmetry and double the wall clock.
10.0s is ~2.4x the slowest observed legitimate run (ordinary prose, cold). The
comment now carries the measurements, says the name overstates the payload, and
warns that the 1_000_200-char size is deliberate -- 200 over the max_scan_chars
default, so the test also exercises the truncate-and-flag oversize path.
No production code changed. 642 passed.
The previous commit said the two corpus counts differ by harvester over the same
knowledge base. Checked the provenance rather than leaving it asserted: 2401
traces to ms-ai-architect's 2026-07-26 message, 2400 to their 07-31 re-run, so
the attribution was right -- but the knowledge base was not held constant. Their
file count moved 389 -> 394 between the runs.
That inverts how the delta reads. A larger snapshot returning one fewer distinct
URL is not a rounding difference between two scripts; two variables moved at
once, so neither count explains the other and the one-URL gap is not evidence of
anything in particular.
Strengthens the do-not-merge rule instead of weakening it. The earlier wording
would have let a reader treat the counts as near-identical measurements of one
thing, which is the merge it was written to prevent.
ms-ai-architect re-measured after retracting their entropy refutation (their
original finding scored whole filenames as one unit, so it was never a finding
about our rule). The retraction confirms the branch; the re-measurement produced
a number we did not have, and it goes against us.
The sharpest legitimate token in their 2400-URL corpus is a 78-character
percent-escaped lovdata title at H=4.301. That is 0.099 from the 4.4 floor where
this bullet previously implied 0.36 -- the mechanism was already ours, but the
worked example understated how thin the margin gets in real Norwegian
government paths.
Verified against our own code before recording, not reconstructed: our real
_URL_TOKEN_RE emits it as a single 78-character token, our shannon_entropy
returns 4.301, and is_ordinary_url returns False. Their three calibration
examples also reproduce to three decimals here.
Threshold stays. Their own conclusion too -- moot per URL, since the % rule
already disqualifies it before entropy is consulted.
Records the 2400-vs-2401 provenance split explicitly. The counts come from two
different harvesters over the same knowledge base and differ by one URL; the
neighbouring hex bullet cites 2401. Left unstated, a later reader would
reasonably merge them into one corpus figure, which is exactly the
prose-vs-measurement conflation this file exists to prevent.
okf delivered the code read we asked for: materialize.py:71 is
`[^a-z0-9]+ -> "-"`, a whitelist, so their generated-path percent-escape
exposure is structurally zero. Verified independently against their source
rather than taken on the message: the regex is where they said, both their
worked examples reproduce against the real implementation, and the only
urllib.parse.quote in their tree builds a sqlite file: URI that never touches
a filename.
That retires the "still open for the one consumer whose slugger we have not
seen" clause. The Loekkene example already in this bullet was linkedin-studio's
run output, not a prediction -- checked before rewriting, because the two
clauses read as if they contradicted each other and only one was stale.
Adds the honest half that closing the question exposes: we have now read two
whitelist sluggers and zero encodeURIComponent-class ones, so the
"produces escapes systematically" arm of this axis is still a prediction from
the transform, not a field observation.
The {tenant} placeholder result goes on the generated-path side, deliberately
not appended to the %7B sentence -- that one is a link-corpus harvesting
artifact, and merging them would re-conflate the two axes this bullet exists
to separate.
No test is owed here: the claim pins okf's regex, which this suite cannot
assert. It is verifiable only by re-reading their source.
A consumer corrected a claim we shipped this morning. We wrote that both
zero-measuring corpora were English, and used that to explain the third corpus's
10 Norwegian hits as a sampling bias. One of those two is a Norwegian repo with
Norwegian content, so the stated explanation was wrong.
Their measurement separates two axes we had conflated. For third-party LINK
corpora -- URLs collected from other people's sites -- language does predict, and
the Norwegian legal/government sources are where the escapes were. For GENERATED
paths the predictor is slugger class: a whitelist slugger ([^a-z0-9]+ -> "-")
cannot emit an escape in any language because it discards the character before
anything encodes it, while encodeURIComponent produces them systematically as soon
as titles are non-ASCII. Their own slugger returns a structural zero on Norwegian
input, not a statistical one.
So non-ASCII language is a confounder for encode-vs-whitelist, and slugger class is
the thing that is actually testable at a consumer -- a one-line code read rather
than a corpus census. Reframed accordingly, with the earlier conflation named so
the correction is visible rather than silently rewritten.
The test badge still said 577 after two commits that added 65 tests, and the
coverage-matrix claim said 126 where CORE_CASES now holds 134 payloads. Both
verified by running rather than by reading: pytest reports 642, and
len(coverage.CORE_CASES) is 134.
A consumer measured our frontmatter gate against 0.2.0 and asked us to confirm
against a newer guard, correctly noting that a divergence would be a version
difference rather than a contradiction. Measured at 0.3.1: no divergence. Both
their results reproduce exactly.
Two things are sharper than what we documented yesterday:
ALL THREE ROUTES TO A MAPPING FAIL, each on a different rule -- flow {k: v} on the
disallowed value-start indicator, block on the nested-mapping check, dotted keys on
the key pattern. So the mapping CLASS has no expressible form; it is not a choice
between two shapes where one is better. That matters because OKF v0.2's `generated`
IS a mapping (`by` is required when it is present), so it cannot be expressed at
all. Pinned by a test asserting the three failures stay distinct.
T2 RUNS ON DOOR C ONLY. parse_frontmatter is referenced nowhere in the door A/B
persist path, so the same frontmatter that FAIL_SECUREs through import_bundle
passes screen_output unremarked. The grammar bounds what a consumer can RECEIVE,
never what a producer can EMIT -- which is the question a consumer's emitter design
was blocked on.
Also corrects one of our own rows: we reported "sources block list" as a single
FAIL_SECURE case. The parser distinguishes three shapes -- flat scalars parse
correctly, one key per item misparses silently to a string, two keys per item hard-
rejects. A one-element `verified:` block list passes today.
631 -> 642 passed. README item count synced.
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.
docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.
LIMITATIONS.md brought current with the field measurements:
- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
selector. No parameter-level remedy covers any two, which moves this from a
conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
where two keys hard-reject, so a pointer can ride past the resource allowlist.
Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
both backward-breaking migration targets are nested. Fail-secure, but a
compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.
New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
Two consumers measured the 0.3.1 URL-shape rule against real corpora on the
same day. Record what they found, and pin the dispositions both of them
inferred wrongly.
Percent-escape FP: zero occurrences across both corpora (0/347 external URLs
in a 527-document vendor-docs corpus, 0/81 in a capture store). This bounds
the shape rather than closing it -- a consumer that slugs filenames from
titles produces %20 systematically, and that corpus is still unmeasured.
Query FP: the over-block that actually occurs. The two corpora hit disjoint
benign populations -- 16/16 publisher-authored utm_* tracking versus 35/35
content identity (?v=, ?channel_id=) where the parameter IS the resource.
An allowlist keyed on tracking-parameter names resolves the first entirely
and the second not at all, so no parameter-level remedy covers both. That
is input to the 0.4.0 axis-separation scope, not a fix here.
Both consumers reported dispositions they had inferred rather than run, and
both were wrong: a query-carrying link is MEDIUM, so it is held or warned,
never hard-failed. tests/test_wiring.py now pins that, plus the active-tag
gate -- counting raw HTML tags overcounts what the gate flags, since
formatting markup is inert and only name/on*=/URL-attr tags are active.
593 passed.
okf asked outright whether the v1.0 gate requires 'the tagged artifact resolves
0.3.1' or 'the fixture set passes against 0.3.1'. It is the second, and the
answer is locked here so a later session cannot quietly upgrade the ask into a
release request against their operator.
They refused to widen their published range to run the measurement, and the
refusal is better than the ask it answers: widening a published range in order
to generate the evidence that would justify widening it is circular, and a red
result would then have meant unwinding a release to fix a test setup. The
measurement has to be able to say no. Their scratch-venv method -- outside the
project environment, --no-deps, guard installed from the v0.3.1 tag, resolved
version asserted from importlib.metadata -- satisfies the gate exactly.
Both were given to named consumers who are building on current behaviour, and
both were about to live only in STATE.md -- which is gitignored, so a promise
kept there does not survive a machine. Breaking either silently is a release
defect, not a preference.
1. Any change to how PRESET_USER_UPLOAD grades ordinary links and remote images
requires notice to linkedin-studio before it ships; they pin v0.3.1 at wiring.
Applies to 0.4.0: the axis separation must change disposition, not grading.
2. Closing the relative-target asymmetry requires notice to llm-ingestion-okf;
it is a Door C property rather than a guard gap, but closing it changes what
arrives at their persist gate.
Also records a mechanism we had not modelled: okf reports percent-escapes
reaching filenames through a slugger, so %20 is manufactured systematically from
document titles rather than being incidental as it is in the documentation URLs
the FP was calibrated against.
okf pins llm-ingestion-guard >=0.2,<0.3 in [project].dependencies and pins the
uv source to tag v0.2.0 -- verified at their v0.4.0 tag and at their HEAD. The
cap excludes 0.3.1, so the committed Door B+C fixture run cannot reach the
version the gate names. It does not fail either: uv resolves v0.2.0, the
constraint holds, and the fixtures come back green because v0.2.0 never had the
regression. Green from an unmodified tree is the one outcome that proves nothing
and reads as proof.
The gate now requires the result to name the resolved guard version, and records
that transitive arrival does not count as an integration -- okf v0.4.0 makes the
guard a mandatory runtime dependency, so a consumer can acquire us without
choosing us, and today lands on v0.2.0 without the 0.3.0 hardening or the 0.3.1
fix. Silent under-defend, not a break.
v0.3.0 made the untrusted upload path unusable: measured on both doors, an
ordinary remote image fail_secure'd and an ordinary link/autolink/refdef
quarantined, so only documents without external references persisted.
Two independent defects compounded; neither fix works alone:
1. `markdown-image: HIGH` fired on any external image. The exfil primitive is a
URL that moves bytes outward, not an image. `is_ordinary_url` now grades on
shape - http(s)/protocol-relative, no query, no userinfo, no percent-escape,
no opaque host label or path segment -> LOW; anything data-carrying keeps the
carrier's severity. raw-html and data: URIs stay HIGH unconditionally.
Opacity reuses entropy's primitives; floors calibrated against real doc URLs
(worst legit token H=4.08, exfil segments 4.36-4.54) and frozen in
calibration.
2. The quarantine_default floor fired on ANY finding, a premise that broke when
every ordinary link became a finding. It now fires at MEDIUM+ - a no-op for
every detector that shipped before 0.3.0 (no LOW/INFO exists), which is what
makes this a patch rather than a minor.
The corpus blind spot that let this pass 522 green tests is closed: the FP
corpus carries realistic markdown and is asserted on the OUTPUT gate under
PRESET_USER_UPLOAD, with a counter-corpus of exfil-shaped URLs that must still
block. Beaconing and short opaque segments are conceded in LIMITATIONS and
asserted by the coverage matrix rather than papered over.
No new public API; no new preset (0.4.0 work); allow_reserved default unchanged.
v0.3.0 makes the untrusted upload path refuse ordinary documents. Measured on
both doors under PRESET_USER_UPLOAD: one ordinary markdown image reaches
fail_secure, one ordinary link/autolink/refdef reaches quarantine_review, and
only a document with no external references persists.
Root cause is three independent defects that compound. Severity tracks
construct type rather than URL shape, so every external image is HIGH when the
exfiltration primitive is actually a URL that carries data outward. The
quarantine floor fires on any finding at all, a premise that broke once every
link became a finding. And the false-positive corpus could not have caught
either: it holds no markdown links or images, asserts only under
PRESET_TRUSTED_SOURCE where everything warns anyway, and runs _scan_input, so
the output gate where active_content lives is never exercised by it.
Both code changes are required together -- verified that fixing severity alone
still quarantines via the floor, and fixing the floor alone still fail-secures
on a HIGH image.
Records the version reasoning too: 0.3.1 is honest as a patch because the
lexicon carries no LOW/INFO patterns (40 high, 22 medium, 21 critical) and no
other detector emits LOW, so raising the floor to MEDIUM+ is a no-op for every
finding that existed before 0.3.0. The new middle preset stays out of this
release; it is additive API and belongs in 0.4.0.
PLAN-v1.md said consumer integration was NOT part of the v1.0 sequence, and
Session G listed A-F as its only dependency -- both of which are now false.
The gate change lived only in STATE.md, which is local-only and overwritten
each session, so the plan was the wrong side of the truth.
Records: 1.0.0 now gates on the first real integration coming back green
(llm-ingestion-okf step 4 against v0.3.0), not on our own suite; the reasoning
(522 self-authored tests prove the code matches the design, not that the
design survives contact -- our first external contact found both a
tag-vs-main divergence and three undocumented behaviour changes); and that
one validated `integrated` outweighs five `planned` declarations, so we do not
wait on repos that may never arrive.
Also fixes Session G's now-stale mechanics: its CHANGELOG entry can no longer
"list A-F" because A/A2/B already shipped under [0.3.0]; the version-sync file
list gains the README status line and install pin; and verification gains an
anonymous clean-venv install check against the new tag, since README must
never advertise a command we have not run.
Version-sync across pyproject, __init__, README badge/status/install pin and
CHANGELOG, and cut [Unreleased] to [0.3.0].
Why minor: 0772daf made allow_reserved default to True on okf.import_bundle,
so a consumer who upgrades without touching their code gets a LOOSENED gate --
a reserved index.md/log.md in a received bundle is now scanned and may become
mergeable, where v0.2.0 rejected it unconditionally. Under 0.x a >=0.2,<0.3
pin absorbs a 0.2.1 silently but stops at 0.3.0, which is the signal such a
change should send. Found by llm-ingestion-okf, our first real downstream
consumer, against main.
Development Status stays 3 - Alpha and the README keeps "the public API may
still change": this ships the hardened surface (Sessions A/A2/B) to consumers
who are still pinned at v0.2.0 and therefore have none of it. It is not the
v1.0 freeze -- that stays gated on the first real integration coming back
green, which is exactly what 0.3.0 makes possible.
Verified: 522 passed; coverage matrix exit 0; version string present in all
four files; no dangling 0.2.0 outside CHANGELOG history.
A downstream consumer (llm-ingestion-okf) found that main had diverged from
v0.2.0 on okf.import_bundle: 0772daf adds allow_reserved defaulting to True,
so a reserved index.md/log.md in a received bundle is now scanned rather than
unconditionally rejected. Their tests pin the v0.2.0 reject, so a 0.2.1 cut
from main would have changed downstream behaviour silently.
Checking that turned up a wider hole: [Unreleased] documented only the
coverage matrix and the docs work, while three commits since the tag change
what an unchanged caller observes -- 0772daf (loosens), 4d53765 (active
content now reaches decide(), HIGH/MEDIUM with compound escalation) and
f4e89d2 (base64-decoded plaintext now hits secret-egress). None were listed,
so CHANGELOG could not answer "is this bump safe?".
Records all three under Changed, and states that the next release is 0.3.0 --
under 0.x a >=0.2,<0.3 pin absorbs a 0.2.1 silently but stops at 0.3.0, which
is exactly the signal a gate-loosening change should send.
The Install section promised `pip install llm-ingestion-guard`, but the
distribution is not on PyPI — a downstream consumer following the README
would fail. Replace it with the channel that actually works, verified in a
throwaway venv against the public mirror:
pip install "llm-ingestion-guard @ git+https://.../@v0.2.0"
Anonymous HTTPS read on the open/ mirror is confirmed, so consumer CI needs
no deploy key. Also states the two things a consumer must know up front:
a git URL is a PEP 508 direct reference (exact tag, not a >=0.2,<0.3 range;
range pinning arrives with a Forgejo PyPI registry at the first patch release
or second consumer), and vendoring is unsupported because it severs the
security patch channel.
Answers the install-channel question raised by llm-ingestion-okf via coord.
Answers the gap that the README said what it does NOT stop (a long limitations
section) but never plainly listed what it DOES. Add a 'What it protects against'
section high up: attack classes grouped by OWASP anchor (LLM01 injection + 83
lexicon classes + carriers, LLM02 egress, LLM05 EchoLeak, LLM06 agency, LLM10
fail-secure, OKF T1-T6, container front-end), each driven by a live coverage-matrix
payload. Move the full honest-limitations list to docs/LIMITATIONS.md; README keeps
a high-impact summary + link. Net: protection and limits read in balance, 261 -> 216
lines. Coverage 126/126 and 522 tests unchanged; every class listed is real.
Rewrite the README value proposition to make the necessity land on mechanism,
not adjectives: write-time ingestion is the trust boundary query-time guardrails
structurally cannot see; an OKF/LLM-wiki has no format-level authenticity, so the
ingestion pipeline IS the trust boundary. Add a first-class 'OKF / LLM-wiki
support (shipped)' section for import_bundle mode-b (per-concept gates). Fix stale
test badge (357 -> 522) and drop the brittle module count. Tighten prose; retain
all honest-limitations items (a shipped control). Every symbol/preset/command
verified against v0.2 code.
Self-contained brief a consumer repo can plan an inclusion from: what the
guard is (write-time, not query-time), the two bookends + 8-step contract,
the shipped OKF adapter (import_bundle mode-b, per-concept gates), how to
verify (coverage matrix -> 126 classes), how to depend (stdlib-only core),
and a planning checklist for WHEN/WHERE to wire it (untrusted boundary, not
first-party onboarding). Every claim verified against v0.2 code.
Add a single declarative manifest proving, in one place, every vulnerability
class the guard stops — and the documented gaps it does not. This is the
real-case validation gate ahead of any v1.0 freeze (v1.0 stays parked until
verified on real cases).
- src/llm_ingestion_guard/coverage.py: stdlib-only manifest (CORE_CASES) +
narrated runner. `python -m llm_ingestion_guard.coverage` prints
class -> OWASP -> expected -> observed -> verdict; exit 0 iff every caught
class is caught and every documented gap holds. 126 caught classes + 4 gaps.
Lexicon cases are generated from load_lexicon() via a payload dict, so a
pattern with no payload fails loudly at import (self-verifying).
- tests/test_coverage_matrix.py: asserts total recall, that every documented
gap still holds, and completeness (every lexicon id + every OWASP anchor has
a case). Adds the full 25-pattern LLM02 secret-egress set (fixtures assembled
from split tokens so no secret shape sits in source) and the container-layer
front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink).
- README + CHANGELOG: point to the runnable matrix.
+165 tests (357 -> 522). No core dependency added.
Reframe .pdf from a 'known gap'/TODO to a deliberate concession across the
honest-limits and OKF-upload docs. A top-level .pdf drop is already refused as
an unsupported format (inbox_frontend.py raises on the else branch), not
half-scanned; adding a PDF parser + reportlab (solely to author white-on-white
test fixtures) is disproportionate for a dev-scoped showcase, and the OCR /
font-render stego carriers a PDF would smuggle are out of scope regardless.
- README honest-limits: .pdf = concession, not TODO; only the numeric CSV FP
remains a known gap.
- docs/PLAN.md: upload table .pdf row marked 'conceded'; honest-scope paragraph
names .pdf; assertions tightened to 'every accepted format'.
Closes the last format gap before v1.0 freeze (Session G). No code touched;
357 tests green.
- README: tests badge 275->357; status v0.1->v0.2 (repo is 0.2.0; the v1.0
bump belongs to the Session G freeze, not this docs pass); add three
honest-limits — lone-HIGH-in-trusted-prose->WARN, vacuous quarantine-floor,
Cyrillic/Latin homoglyph-mix false positive.
- docs/BRIEF.md: drop "No code yet" pre-implementation framing -> implemented v0.2.
- docs/OKF-INGESTION-BRIEF.md 4: correct cross-link control language —
absolute https / references/ targets are spec-permitted, not rejected.
- Add SECURITY.md (private Forgejo disclosure) + CONTRIBUTING.md (stdlib-only
core, Iron-Law TDD, no trailers, Forgejo-only invariants).
Session D: move every calibration constant (entropy floors 5.4/128, 5.1/64,
4.7/40 + shape floors; MAX_SCAN_CHARS; rot13-min; cognitive-load lengths
2000/2500; disposition ranks; active-content severities) into one documented
calibration.py, so a parallel Node/TS port can mirror exactly the same numbers.
Pure refactor, zero behavior change: calibration is a leaf module (imports only
report.Severity) that entropy/lexicon/disposition/active_content now source
their thresholds from. MAX_SCAN_CHARS is re-exported from lexicon so output.py
and existing callers are unaffected. The 347 pre-existing tests pass unmodified;
new test_calibration.py freezes the values and asserts each detector actually
reads its threshold from calibration (identity-checked, not a dead copy).
Replace the unverified/absolute novelty statement with the defensible
four-part-contract form, verified against a focused adversarial PyPI+GitHub
survey (2026-07-15):
- BRIEF §11: 'assumed, not verified' -> verified survey with sources. Names
aig-guardian (real, query-time; blurs only the minimal-dep-library
differentiator), GuardLLM (nearest neighbour, runtime hardening, no
scan-before-persist / capability isolation / fail-secure), and ipi-scanner
(orphaned placeholder repo, recorded for honesty not as prior art).
- README: differentiator moved from 'library vs hosted/model' to the full
four-part write-time contract.
- PLAN §27-31: drop the unverifiable 'the first' superlative.
Also promotes the v1.0 session plan (PLAN-v1.md) and the cross-model review
(review-2026-07.md) into docs/ on the open/ mirror, referenced by PLAN.md's
re-sequencing addendum.
Output gate step 3 now runs scan_secret_egress over every decoded base64
blob's plaintext, not only scan_lexicon. A base64-wrapped credential that
formerly vanished (decode fed the lexicon, which has no secret patterns)
now surfaces as decoded:egress:* carrying the blob offset. Evidence stays
length-only, so the decoded finding never leaks the secret value.
Hex-wrapped secrets remain a documented honest-limit (entropy exposes
decoded plaintext for base64 only). README honest-limits + CLAUDE.md
Kontekst updated; 3 tests added (347 passed, was 344).
A received OKF bundle MAY legitimately carry index.md (directory listing, read
first under progressive disclosure) and log.md (update history) at any level
(spec §3.1/§6/§7). import_bundle previously hard-rejected those basenames in the
T4 path gate, so a conformant third-party bundle was over-blocked in full
(FAIL_SECURE) — and because the reject fired before scan_concept, index.md's
body (the highest-priority injection surface) was never scanned.
import_bundle now defaults allow_reserved=True: reserved basenames are scanned
as structural files (path-safety checks — traversal / absolute / backslash / .md
— still apply). The shadow-reject (an *upload* masquerading as index.md) is
preserved: the front-end passes allow_reserved=False so a materialized upload
landing on a reserved basename is still refused. That front-end opt-in was
required to keep the shadow-reject once the default flipped (not in the plan's
Filer set; traced from the code).
- okf.py: validate_concept_path/_validate_concept/import_bundle gain the
keyword; validate_concept_path default stays False (strict standalone).
- tests: +3 (legit index/log admit; injection in index.md body caught;
okf_version frontmatter admits). Per-concept-iteration test switched to a
traversal vector; mode-b showcase's index.md surface reframed from
reserved-name-reject to index.md-body-scan.
- README honest-limits + CLAUDE.md context note the mode-b/upload distinction.
Suite: 341 -> 344 passed. Core invariant intact (dependencies=[]).
Close the EchoLeak wiring hole (CVE-2025-32711 class): markdown images/
links, reference definitions, autolinks, raw active HTML and data: URIs
now surface as report-only findings (active:*, OWASP LLM05) in
scan_output step 6, so screen_output and okf.import_bundle dispose of
them instead of admitting them with findings=[].
- new active_content.py: canonical home of the shared pattern table +
scan_active_content; neutralize refactored to import it (mutating API
and behavior unchanged, all neutralize tests pass as-is)
- images/links flagged only for absolute/protocol-relative URLs:
relative in-bundle links are legitimate wiki/OKF mechanism (principle 5)
- evidence carries defanged URLs only (hxxps://evil[.]example)
- EchoLeak vectors planted in both showcases; detach proofs cover them
- README export list + checklist step 6, CLAUDE.md context line updated
Suite: 321 -> 341 passed. Core invariant intact (dependencies=[]).
Two structural regions the office extractors missed, no new dependency:
- .docx table cells (they live outside doc.paragraphs);
- grouped .pptx shapes (add_group_shape moves a shape inside the group, so the
extractor recurses through MSO_SHAPE_TYPE.GROUP to reach it).
Each proven by an injection-in-region REJECT plus a clean-region ADMIT. Tests
310 -> 314.
Concede the binary layer that survives text extraction as out-of-scope (a
shipped control): VBA/macros, OLE/embedded objects, OCR-needed image text,
font/render stego, encrypted files. Point to the two-stage OKF inbox showcase as
the worked example, and list the known gaps (.xlsx/.pdf deferred, .docx tables,
grouped .pptx shapes, numeric CSV -/+ FP).
The pptx payload hides where an audience watching the slides does not look. The
extractor surfaces all three regions into the concept text so the stage-2 scan
catches the injection:
- speaker notes (slide.notes_slide.notes_text_frame.text);
- off-slide (off-canvas) text boxes (shape.text_frame.text, position-agnostic);
- image/shape alt-text (cNvPr@descr, read off the XML — python-pptx 1.0.2 has no
stable public accessor across shape types).
Detach-proof: same visible slide without notes ADMITs. python-pptx imported
lazily (dev/showcase-scoped, not a core dep). Tests 305 -> 310.
The docx payload hides where a human reviewing the file in Word does not look.
The extractor surfaces all three regions into the concept text so the stage-2
scan catches the injection:
- hidden/vanish runs (w:vanish) — still runs, so paragraph.text includes them;
- review comments (doc.comments[].text);
- core metadata properties (subject/keywords/comments/title/category/author).
Detach-proof: the same visible body WITHOUT the hidden run ADMITs, so it is the
extractor surfacing the hidden region that caught it, not merely 'a docx'.
python-docx is imported lazily (dev/showcase-scoped, not a core dep). Tests
300 -> 305.
