`5667063` moved test_output.py's ReDoS bounds off the wall clock, because a
loaded machine steals wall seconds without adding any cycles and two rows
failed at 2.24s / 3.66s against a 2.0s bound while census had the CPU. The
remaining ten bounds in five other files still ran on `time.monotonic()` and
carried the same defect. They now share ONE clock.
The clock is IMPORTED, not copied: `tests/redos_clock.py`. Five private copies
would leave four of them unpinned -- the instrument test
(test_the_redos_clock_ignores_time_this_process_did_not_spend) can only pin the
implementation it calls, and the suite already holds that rule for the code it
measures.
Every ported row was verified the only way a time bound can be: the vulnerable
form patched back in, red demanded, `git checkout --` after. Measured against
the 2.0s bound (3.0s for the url arm):
active_content long-attr `{0,63}` -> `*` RED
neutralize long-attr same patch RED
output gate long-attr same patch 12.41s
okf link graph `[^\]\[]` -> `[^\]]` 6.91s
sanitize comment str.find -> `<!--.*?-->` 17.56s
lexicon md-link-anchor-text 319.14s
lexicon md-link-anchor-url 8.55s
lexicon md-link-ref-comment 37.82s
Two rows did not go red, for two different reasons.
test_sanitize.py::test_legitimate_comment_heavy_document is the legitimate SIDE
of a separation, not a second pin on the defect: closed comments never withhold
the required literal, so the lazy form runs it in 0.016s. Recorded in place.
test_lexicon.py::test_redos_pathological_subagent_input_returns_fast is DEAD --
the same zero-signal shape the `<a ` carrier had, found by the same method. The
seed form is `(?:.*?\s+)?` (llm-security 7.8.0, injection-patterns.mjs:84) and
this repo has never carried it: the bounded `{0,12}?` port is in the pattern
table's first commit. Patched in by hand at the row's own size: shipped 0.135s
vs seed 0.113s, separation 1.2x. Not the keyword gate either -- a variant that
reaches the inner branch stays linear over four doublings (exponent ~1.0),
because the nesting is one lazy run inside an OPTIONAL group, never a repeated
one. Left standing with the measurement written into it; picking a new carrier
is an operator call, like the wall-clock row above it.
The dead sibling row named in STATE is fixed: test_active_content.py's
long-attribute row swaps carrier `<a ` -> `<script `, for the reason `5667063`
established on its composed-gate twin -- 0.7.0's own no-URL narrowing put `<a>`
in `_URL_AFFORDANCE_TAGS`, so the tag returns inert BEFORE its body reaches the
arm the row guards. Re-measured here, not inherited: `<a ` 0.041s and NO
findings against the vulnerable form; `<script ` 19.349s against 0.052s
shipped, 373x apart.
`test_pathological_input_returns_within_a_bound` deliberately keeps its wall
clock (operator decision): it claims to catch a hang, and only a wall clock
catches one.
792 tests, 129/129, 6/6.
Two changes that had to ship together, because they co-occur.
`active:raw-html-link` (MEDIUM) splits the click-required carriers out of
`active:raw-html`. The same URL was LOW as `[t](url)` and HIGH as
`<a href="url">` — an asymmetry produced by syntax, not by affordance, on a
carrier the markdown path has graded MEDIUM since 0.3.1. The event-handler test
runs first, so `<a onclick=...>` stays HIGH. The url-attribute branch stays HIGH
too: a name outside the active set has unknown rendering, and grading
`<Card src=...>` as a link would be reasoning rather than measurement.
The no-URL narrowing makes `</a>`, `<Frame>`, `<video />` and `<img alt=...>`
without `src` inert — `<base />`'s argument from 0.6.0 applied to the rest of the
name branch. It tests for the URL attribute's PRESENCE, not for a readable value,
so the fail-secure gap `_url_attr_is_external` leaves open is not reopened here.
WHY TOGETHER: the narrowing strips a document's `</a>`/`<Frame>` and what remains
is the `<a href=...>` the split grades down, so each alone leaves the document
blocked by the other's residue.
`active_tag_class` is now the classification point and `is_active_tag` wraps it.
The census patches the former: a boolean could only express a narrowing, never a
regrade, so every carrier candidate would have measured equal to PRODUCTION —
silently, and in the direction that reads as "no change helps".
TWO COSTS, BOTH RECORDED RATHER THAN GLOSSED:
- The split TIGHTENS the trusted tier. One finding becomes two, and >=2 findings
at MEDIUM+ trip the compound overlay, so a document carrying both an `<img src>`
and an `<a href>` goes WARN -> quarantine_review on PRESET_TRUSTED_SOURCE. On
that preset it is the only direction the split can move anything. The census
now reports a TIGHTENS column on both trust tiers against the previously
shipped row — "frees N" without "tightens M" is a one-sided number.
- `count` drops on documents containing `</a>`, a published field moving under a
meaning that did not change.
MEASURED: reference-corpus (389) 54 -> 53 fail_secure, tightens 0/0, and the
census `PRODUCTION` row equals its `C1 + D` candidate row for row. The census
also reproduces 133/3/13/108/25 exactly, so it is calibrated against every
published historical number. The two wiki corpora are NOT yet re-measured; the
tree says so explicitly in the docstring, LIMITATIONS and CHANGELOG rather than
carrying probe numbers as fact.
791 tests (was 759), coverage 129/129, 6/6 documented gaps holding. Version
bumped to 0.7.0 across every surface; no tag is set until the measurement lands.
`is_active_tag`'s URL-attribute branch was a presence test: any element carrying
`href=`/`src=`/`action=` graded HIGH regardless of where the URL pointed. An MDX
`<Card href="/en/agent-sdk/quickstart">` reaches no attacker-controlled host, and
neither does APIM policy XML's `<set-header>`. It now requires an external target
-- the rule the markdown paths have applied since 0.3.1. `<base>` left the active
name set in the same change: HTML's `<base>` has its whole affordance in an `href`
the attribute branch still catches, and APIM's attribute-less `<base />` is inert.
Measured before and after in ONE session against one corpus state, because two of
the three corpora are living and a split would mix this with re-harvest drift:
reference-corpus 389 docs 133 -> 108 (ceiling 107)
vendor-harvest 187 docs 100 -> 98 (ceiling 62)
generated-notes 550 docs 90 -> 88 (ceiling 49)
96% of the achievable reduction in reference-corpus, 5% in the wiki corpora. The
two classes had to be measured TOGETHER -- alone they free 3 and 13 documents,
together 25, because a document carrying one usually carries the other.
The second surface: `neutralize` imported `is_active_tag` by name, so this would
have silently narrowed the opt-in mutator too -- and no test discriminated the two
halves, since every `neutralize:raw-html` payload stays active under any narrowing
considered. That test is written first here. The predicates are now separate
symbols; the mutator keeps defanging anything, because over-defanging is auditable
and blocks nothing while under-defanging hands a human a live construct.
Behaviour change: a document whose only finding was one of these classes now WARNs
instead of holding. Detection is unchanged -- 128/128 classes, 6/6 gaps hold.
Self-safety: reading an attribute VALUE needs a pattern the presence test lacks. It
reuses the same literal alternation so no new run shape enters the table; its
`_REDOS_PAYLOADS` row denies the `=` the pattern requires, since a unit supplying it
matches at once and never exercises the run (the lexicon's `script-tag` row is the
cautionary case). 0.031-0.046s across five attack shapes at 100_000 chars against a
2.0s bound; `docs/redos-sweep.py` reports 0 candidates of 152. An attribute the
presence test saw but the value parser cannot read counts as external -- fail secure.
`docs/rawhtml-census.py` gains a PRODUCTION row that re-measures the shipped
predicate rather than a hypothesis, so a published number and the code cannot drift
apart unnoticed. README's limitation count moves 34 -> 33.
727 passed (was 717).
`scan_active_content` called directly and `okf.link_graph` were the two surfaces
still reading attacker-supplied text with no cap — the first reached by an
adapter that wants the active-content classes alone, the second running a
`findall` over every body in a bundle. Both are detection-shaped, so they
truncate and flag rather than raise the way the transform surfaces do: what a
detector shortens is its own coverage, not the caller's content.
Truncation is only honest if it is visible, so neither goes quiet: the scanner
emits `active:oversize-input` (LLM10), and `link_graph` records
`(from_id, body_length)` in `LinkGraphResult.truncated` — the field that lets a
caller tell "no links past here" from "no links read past here".
Reached through `scan_output`, the text is already under that surface's cap and
`max_scan_chars` is now passed down, so the flag is raised once, there.
The generalised sweep found what 0.3.2's hand-written rows missed. All three are
the documented class -- a run in front of a required literal that never arrives,
so every start position rescans the tail -- and all three are worse than the
0.3.3 findings, because `sanitize`, `neutralize`, `scan_active_content` and the
okf link graph apply NO input cap. `scan_lexicon`/`scan_output` are the only
entry points that do, so there is no ceiling to extrapolate to.
sanitize._HTML_COMMENT_RE `<!--`*100_000 20.1s, exponent 1.96-2.14
active_content.URL_IN_TEXT_RE `<a `+`A`*100_000 12.99s / 14.9s, exponent ~2.0
okf._MD_LINK_RE `[`*100_000 7.1s, exponent 1.99-2.05
Each fix is the one the pattern's own shape allows, not a copied choice:
- The comment stripper drops the regex for `str.find`. Excluding `<` would lose
every comment containing markup; bounding the run would be a carrier bypass
of the exact construct the stripper exists to remove.
- `URL_IN_TEXT_RE` bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
Bounding is safe *here* only because it is a defanger inside a tag already
flagged `active:raw-html`. A lookbehind was measured too and rejected: it
drops `-http://evil.com`, a one-character evasion. Bounded: 0.185s at 1M.
- `_MD_LINK_RE` excludes `[`, matching `active_content.MD_LINK_RE` exactly,
including the nested-label trade already documented there.
`sanitize` claimed "no catastrophic backtracking" in a comment; that claim was
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
676 tests (+10), coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
The okf destination run gets no row: `[^)\s]+` cannot fail, so a row for it
could never go red.
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.
docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.
LIMITATIONS.md brought current with the field measurements:
- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
selector. No parameter-level remedy covers any two, which moves this from a
conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
where two keys hard-reject, so a pointer can ride past the resource allowlist.
Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
both backward-breaking migration targets are nested. Fail-secure, but a
compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.
New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
v0.3.0 made the untrusted upload path unusable: measured on both doors, an
ordinary remote image fail_secure'd and an ordinary link/autolink/refdef
quarantined, so only documents without external references persisted.
Two independent defects compounded; neither fix works alone:
1. `markdown-image: HIGH` fired on any external image. The exfil primitive is a
URL that moves bytes outward, not an image. `is_ordinary_url` now grades on
shape - http(s)/protocol-relative, no query, no userinfo, no percent-escape,
no opaque host label or path segment -> LOW; anything data-carrying keeps the
carrier's severity. raw-html and data: URIs stay HIGH unconditionally.
Opacity reuses entropy's primitives; floors calibrated against real doc URLs
(worst legit token H=4.08, exfil segments 4.36-4.54) and frozen in
calibration.
2. The quarantine_default floor fired on ANY finding, a premise that broke when
every ordinary link became a finding. It now fires at MEDIUM+ - a no-op for
every detector that shipped before 0.3.0 (no LOW/INFO exists), which is what
makes this a patch rather than a minor.
The corpus blind spot that let this pass 522 green tests is closed: the FP
corpus carries realistic markdown and is asserted on the OUTPUT gate under
PRESET_USER_UPLOAD, with a counter-corpus of exfil-shaped URLs that must still
block. Beaconing and short opaque segments are conceded in LIMITATIONS and
asserted by the coverage matrix rather than papered over.
No new public API; no new preset (0.4.0 work); allow_reserved default unchanged.
Close the EchoLeak wiring hole (CVE-2025-32711 class): markdown images/
links, reference definitions, autolinks, raw active HTML and data: URIs
now surface as report-only findings (active:*, OWASP LLM05) in
scan_output step 6, so screen_output and okf.import_bundle dispose of
them instead of admitting them with findings=[].
- new active_content.py: canonical home of the shared pattern table +
scan_active_content; neutralize refactored to import it (mutating API
and behavior unchanged, all neutralize tests pass as-is)
- images/links flagged only for absolute/protocol-relative URLs:
relative in-bundle links are legitimate wiki/OKF mechanism (principle 5)
- evidence carries defanged URLs only (hxxps://evil[.]example)
- EchoLeak vectors planted in both showcases; detach proofs cover them
- README export list + checklist step 6, CLAUDE.md context line updated
Suite: 321 -> 341 passed. Core invariant intact (dependencies=[]).
