No data file changed and no pattern moved. A published figure was wrong. llm-ingestion-pipeline-security flagged it (coord, 2026-08-11T19:51:55Z) by measuring the hybrid-xss:iframe-src row themselves instead of citing ours, and got roughly 4x our number. They attributed the gap to measurement surface - their composed scan_lexicon() against our standalone regex - and said no re-measure was needed. Re-measured anyway, because the claim was about OUR number. Their diagnosis does not hold: our standalone 100 000-char figure is 7.86 s against their composed 8.95 s, so the two surfaces differ by far less than the error did. Standalone, Python 3.14.0, same unit the document claims: iframe-src [^>]* 822.7 ms @ 32k 51 477.4 ms @ 256k published 119.6 ms @ 32k 16 857 ms @ 256k script-tag [^>]* 87.4 ms @ 32k 5 222.6 ms @ 256k published 21 ms @ 32k 5 440 ms @ 256k So the Python script-tag figure at 256k reproduces and the one at 32k does not, and the iframe-src pair reproduces at neither point. Error ratios are not constant, so a single mis-sized input does not explain it, and the original harness lived in a previous session's scratchpad and is gone. Recorded as NOT DIAGNOSABLE rather than given a plausible cause - a guessed cause would read like a finding. Superseded figures are struck in place rather than overwritten. Anyone who cited the old number needs to be able to discover that they did. The 0.4.0 decision does not depend on this. Every corrected figure is larger, the shape is unchanged (quadratic, x4 per doubling), and both [^><]* forms stay flat. The 0.4.0 CHANGELOG section still quotes the old figure and is left alone: that section is the record of what was released, not a live claim. Verified: JSON well-formed, specs normative, charter clean, README vendoring examples and prose moved to v0.4.1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLEZ4XCSnSrQUFA8SzkQB4
496 lines
32 KiB
Markdown
496 lines
32 KiB
Markdown
# Lexicon port divergence — commons vs the Python guard
|
||
|
||
**Status: informative.** Nothing here is normative and nothing here changes a data file. It
|
||
records a measured disagreement between two ports of one source table, so the decision can be
|
||
taken where each table is tested. Under this repository's behaviour-preservation invariant,
|
||
a divergence found here is **reported, not fixed**.
|
||
|
||
That invariant held through the closure recorded below. Two rows this document measured did change
|
||
in `lexicon/injection-lexicon.json` `0.8.0` — and not because this document found them wrong. The
|
||
finding was routed to the runtime that owns the value, that runtime re-measured it, decided, and
|
||
asked commons to carry the result. The measurement travelled; the authority did not.
|
||
|
||
Produced 2026-08-09. Every number below came from a command; the scripts live in the session
|
||
scratchpad rather than in this repository, because executable code here would breach the
|
||
charter. They are reproducible from the method column.
|
||
|
||
**Revised the same day, after `llm-security`'s source became readable and both runtimes
|
||
replied.** Four things changed, and three of them are corrections to this file rather than
|
||
new results:
|
||
|
||
1. The claim that **neither runtime misses an attack** is **retracted**. It does. See
|
||
*What this does not show* — the measurement behind that claim unioned pattern tables
|
||
belonging to two different runtimes and read the result as a statement about each.
|
||
2. One of the 13 divergences does not reach report level, so **12** is the number that
|
||
changes what a report says. The 13 still blocks `conformance/`.
|
||
3. The `hybrid` **severity is resolved** to `high` — the reported hint was correct, and the
|
||
citation behind it was not.
|
||
4. The **pattern id space is ratified** by both runtimes.
|
||
|
||
Corrections are marked in place rather than edited away, because a reader who saw the first
|
||
version needs to know which sentence moved.
|
||
|
||
## What was compared
|
||
|
||
| Side | Artefact | Version |
|
||
| --- | --- | --- |
|
||
| commons | [`lexicon/injection-lexicon.json`](../lexicon/injection-lexicon.json) | file `version` as committed |
|
||
| guard | `llm-ingestion-pipeline-security` `src/llm_ingestion_guard/injection_lexicon.json` | lexicon `version` 1.0, repo v0.3.4, commit `0bf0729` |
|
||
|
||
Both are **ports of the same file**: `llm-security/scanners/lib/injection-patterns.mjs`. The
|
||
guard's JSON says so in its own `note` field — *"Injection lexicon ported from llm-security
|
||
injection-patterns.mjs. Single source of truth."* Commons extracted the same table from an
|
||
operator dump of that module.
|
||
|
||
That is what makes the comparison worth running. These are not two different detectors that
|
||
happen to overlap; they are two transcriptions of one table, and where they disagree, they
|
||
disagree about what the same source says.
|
||
|
||
## Result
|
||
|
||
| Measure | Method | Result |
|
||
| --- | --- | --- |
|
||
| Pattern count, both sides | count entries | 83 and 83 |
|
||
| Label correspondence | match commons `label` to guard `desc`, em-dash normalised to hyphen | **83/83** |
|
||
| Regex source byte-identical | string compare | **64/83** |
|
||
| Differing regex text | string compare | 19 |
|
||
| — of those, provably equivalent | unescape commons' JS-isms (`\/` → `/`, `\uXXXX` → literal) and compare for string identity | **6/6 identical** |
|
||
| — of those, behaviourally divergent | differential match-set comparison (offsets + matched text), targeted corpus per pattern family | **13**, each with a concrete witness input |
|
||
| — of those, divergent at REPORT level | re-check whether a sibling pattern raises an equivalent finding on the same witness | **12** — one of the 13 is a label-set difference only |
|
||
| Total input comparisons | count | 401 |
|
||
| Flags | compare declared flags | **0 differences** |
|
||
| Severity / family | commons family vs guard `severity` | **8 differences** (all `hybrid`) — resolved, see *Severity* below |
|
||
|
||
64 identical + 6 escaping-only + 13 divergent = 83.
|
||
|
||
**Read the 13 and the 12 as answering different questions.** Thirteen patterns produce
|
||
different label sets. Twelve of those change what a report would say. The gap is the
|
||
`hybrid-xss: iframe with executable src` row: the guard's version of that one pattern misses
|
||
the witness, but its `hybrid-xss: javascript: URI scheme` pattern fires on the same input at
|
||
the same severity and the same OWASP anchor, so a reader of the guard's report still sees a
|
||
`high` / `LLM01` finding on that payload. Measured, not reasoned: the guard's table matched
|
||
`hybrid-xss:javascript-uri` (high, LLM01) and nothing else; the Node engine matched both
|
||
`hybrid-xss: javascript: URI scheme` and `hybrid-xss: iframe with executable src`. A
|
||
`conformance/expected.json` would have to encode the difference, because a fixture names
|
||
labels — which is why the 13 stood as the number that blocked the corpus. *(Superseded
|
||
2026-08-10: it blocks a fixture written over a **witness** input, and the corpus contains
|
||
none. See* Consequence for `conformance/` *below.)*
|
||
|
||
The 6 escaping-only pairs are equivalent by **string identity after unescaping**, not merely
|
||
by absence of a counter-example: `<\/system>` vs `</system>`, `text\/html` vs `text/html`,
|
||
`(?:https?:\/\/…)` vs `(?:https?://…)`, `(?:~\/)?` vs `(?:~/)?`, and two `\uXXXX`-vs-literal
|
||
character classes. Commons writes code points as escapes by convention; the guard's `note`
|
||
records the same choice for non-ASCII.
|
||
|
||
## The 13, and which side misses what
|
||
|
||
Divergence is **not one-directional**. Each side has inputs the other's pattern does not match.
|
||
|
||
| Pattern | Direction | Witness input |
|
||
| --- | --- | --- |
|
||
| `html-obfuscation: display:none element with content` | commons misses; span differs | attribute run padded past 256 chars |
|
||
| `html-obfuscation: visibility:hidden element` | span only | `<div data-x="<" style="visibility:hidden">x</div>` |
|
||
| `html-obfuscation: off-screen positioned element` | span only | same shape, `position:absolute` |
|
||
| `html-obfuscation: zero font-size element` | span only | same shape, `font-size:0` |
|
||
| `html-obfuscation: zero opacity element` | span only | same shape, `opacity:0` |
|
||
| `html-obfuscation: zero-size overflow-hidden element` | span only | same shape, `height:0;overflow:hidden` |
|
||
| `html-obfuscation: injection in aria-label attribute` | **commons misses** | `<a aria-label="` + 300 × `a` + `ignore">x</a>` |
|
||
| `markdown: suspicious instruction in link-reference comment` | **guard misses** | `[//]: # (a (b) ignore)` |
|
||
| `markdown: injection payload in link anchor text` | **guard misses** | `[click ignore this](http://e.com/a(b))` |
|
||
| `sub-agent: spawn instruction with dangerous capability keywords` | **guard misses** | `create an agent that` + 20 filler words + `execute` |
|
||
| `sub-agent: delegation with safety bypass instruction` | **guard misses** | `delegate to a new agent` + 200 chars + `bypass` |
|
||
| `hybrid-xss: <script> tag in content (agent context XSS)` | **reversed 2026-08-11, see below** | `<script>alert(1)` (unclosed), `<script src=x.js>` |
|
||
| `hybrid-xss: iframe with executable src (agent context XSS)` | **guard misses the label, not the attack** | `<iframe data-x="<" src="javascript:alert(1)">` |
|
||
|
||
The last row is the one of the 13 that does not reach report level. The guard's
|
||
`hybrid-xss:javascript-uri` (`javascript\s*:`, high, LLM01) matches that witness, so the
|
||
payload is still flagged at the same severity and anchor; only the label set differs — one
|
||
finding instead of two. The remaining 12 rows change what a report says.
|
||
|
||
"Span only" means both sides produce a match on the same input but over different extents —
|
||
the guard's match starts at an interior `<`. Whether that matters depends on whether a
|
||
consumer reports offsets or evidence text; it does not change whether a finding is raised.
|
||
|
||
The commons-side misses were confirmed in a real JS engine (Node v25.8.2, `RegExp` built from
|
||
the committed JSON), not only in the Python harness used for the differential.
|
||
|
||
### `hybrid-xss:script-tag`: the row reversed, 2026-08-11
|
||
|
||
At the time this file was written, `<script>alert(1)` (unclosed) and `<script src=x.js>` were
|
||
commons misses only in the sense that commons was a byte-identical extraction of
|
||
`llm-security`, so "commons misses" meant "commons and `llm-security` both miss, the guard
|
||
does not." That symmetry broke on `llm-security`'s side.
|
||
|
||
`llm-security` commit `90f576f` (`fix(llm-security): v8.x-A - close <script> recall hole, add
|
||
whole-table ReDoS gate`) replaced the source pattern
|
||
|
||
```
|
||
<script\b[^>]*>[\s\S]*?<\/script>
|
||
```
|
||
|
||
with
|
||
|
||
```
|
||
<script\b[^>]*>
|
||
```
|
||
|
||
dropping the closing-tag requirement that produced the recall hole this file first measured.
|
||
Both witnesses now match on `llm-security`'s side. Verified directly against the module at
|
||
commit `c67bad3` (2026-08-10, first commit carrying the change) and re-confirmed unmoved at
|
||
`b1ba1fb` (2026-08-11, current public-remote HEAD as fetched this session) — `git diff
|
||
c67bad3..b1ba1fb -- scanners/lib/injection-patterns.mjs` is empty, so no later commit touched
|
||
the pattern a second time.
|
||
|
||
The guard's pattern was never the same construct: `<script\b[^><]*>`, negated-class excluding
|
||
both `>` and `<` (`src/llm_ingestion_guard/injection_lexicon.json:463`, unchanged since
|
||
`de09711`, re-confirmed unmoved through current guard HEAD `0dce50f`, 2026-08-11). The two
|
||
patterns now agree on **shape** — both drop the closing-tag requirement, both are a single
|
||
negated-class run immediately after `<script\b` — but they are still not byte-identical: the
|
||
guard's excluded-character class also excludes `<`, `llm-security`'s does not. That one-byte
|
||
difference is unresolved on both sides (see *Open questions* in the coordination thread); it
|
||
does not reopen the closing-tag question, which is settled.
|
||
|
||
**Commons converged in lexicon `0.7.0` (2026-08-11), and this row is closed on the closing-tag
|
||
axis.** Through `0.6.0` commons was the sole diverger of the three: both `llm-security` and the
|
||
guard matched `<script src=x.js>` and unclosed `<script>alert(1)`, and commons' extracted pattern
|
||
still required `</script>`. Commons now carries `<script\b[^>]*>` — byte-identical to
|
||
`llm-security`'s `RegExp.prototype.source` at `b1ba1fb`, verified by compiling both.
|
||
|
||
The decision was **re-extraction, not revision**, and the distinction is what made it available
|
||
at all under the behaviour-preservation invariant. That invariant forbids commons from correcting
|
||
seed data it believes is wrong; it does not forbid re-reading the source after the source itself
|
||
moved and its owner asked. The lexicon's declared provenance is `llm-security`'s injection table,
|
||
and being loadable verbatim by that table's owner is the one thing it exists to do — the standing
|
||
alternative was a per-pattern override in `llm-security`'s own loader, i.e. a published core that
|
||
its source repository could not load as published. Operator decision, 2026-08-11, on
|
||
`llm-security`'s blocking coord request. Explicitly **not** decided by the 2-of-3 majority: a
|
||
count of implementations is not a mandate over detection data, and the provenance argument would
|
||
have held with the guard on either side.
|
||
|
||
The direction matters for what it cannot break: the new form matches a strict superset of the old
|
||
one, so relative to `0.6.0` it can add matches and cannot add a false negative. The reverse change
|
||
would not have been adoptable on the same reasoning.
|
||
|
||
**The one-byte span difference is CLOSED as of commons `0.8.0`** — all three runtimes now carry
|
||
`[^><]*`. What follows is the account of the divergence while it was open, kept because the table
|
||
below is the measurement that made the closure decidable, and because a consumer diffing against
|
||
`v0.3.0` will find the two changed rows and needs the coordinate.
|
||
|
||
While it was open, the guard excluded `<` from its negated class and the other two did not. The
|
||
three forms turn out to be **totally ordered by what they match** — each a strict superset of the
|
||
next:
|
||
|
||
| Witness | commons `0.7.0` / `llm-security` `[^>]` | guard `[^><]`, and commons from `0.8.0` | commons ≤ `0.6.0` |
|
||
| --- | --- | --- | --- |
|
||
| `<script>steal()</script>` | match | match | match |
|
||
| `<script src=x.js>` | match | match | no match |
|
||
| `<script>alert(1)` | match | match | no match |
|
||
| `<script\n src=x>` | match | match | no match |
|
||
| `<script a="<" >x` | **match** | **no match** | no match |
|
||
| `<script<div>` | **match** | **no match** | no match |
|
||
|
||
Measured by compiling all three forms, not reasoned from the character classes. The last two rows
|
||
are the whole residual divergence.
|
||
|
||
The witness set and the ordering observation are `llm-security`'s, from their coord message of
|
||
2026-08-11T11:01:59Z; both were **reproduced here independently** before being recorded, including
|
||
the strict-superset property in both directions. Their reading of it: `[^><]` refuses to cross a
|
||
`<`, and since anything that reaches `[^>]`-and-not-`[^><]` already contains a literal `<script`
|
||
opening tag, the narrower class buys an empty false-positive surface in exchange for two dropped
|
||
input classes — a recall loss with no precision gain to trade for.
|
||
|
||
**That argument is recorded, not endorsed, and it did not decide commons' form.** Commons carries
|
||
`[^>]` because the lexicon's declared provenance is `llm-security`'s table (see above), not
|
||
because their regex won on merit — the same reasoning would have taken commons to `[^><]` had the
|
||
source been the guard's. `llm-security` asked to hear the guard's reason for `[^><]` — a ReDoS
|
||
bound, or a span-extraction consumer needing the run to stop at `<` — before commons shipped, and
|
||
commons shipped first. That order is deliberate rather than an oversight: a reason that beats this
|
||
argument is a reason for the *guard's own port*, and commons would still track its declared
|
||
source. Put to the guard as an open question 2026-08-11; unanswered at time of writing.
|
||
|
||
**How it resolved, and why the resolution used the same rule rather than overturning it.** Both
|
||
halves of that open question came back the same day. The guard answered that `[^><]` is a
|
||
*measured* ReDoS fix on their side (their commit `cff0437`), not an aesthetic choice — so the
|
||
"reason that beats this argument" the paragraph above asked for turned out to exist. And commons
|
||
then measured its own copy and found the same defect in it: `<script\b[^>]*>` and
|
||
`<iframe\b[^>]*src…` are both quadratic in scan length, the second one having been named by nobody
|
||
until it was counted. `llm-security` re-measured rather than taking the report, reached `[^><]*`
|
||
independently, and asked commons to carry it — because their table is now *built from this file*,
|
||
so there is no local literal left for them to change without forking.
|
||
|
||
The paragraph above said commons "would still track its declared source." That is exactly what
|
||
`0.8.0` does: the declared source chose `[^><]*` and commons followed. The rule did not bend to
|
||
accommodate a better argument — the argument won *at the source*, which is the only place this
|
||
repository lets an argument decide a `pattern` value. The recall cost stated in the two bold rows
|
||
of the table is now paid deliberately: measured across all 90 conformance cases, zero lost a match
|
||
and zero gained one, so the dropped classes remain real and remain unwitnessed by the corpus.
|
||
|
||
## Why they diverge: two different ReDoS mitigations of one table
|
||
|
||
This is not drift, and framing it as a bug in either repository would be wrong.
|
||
|
||
Both ports have been hardened against catastrophic backtracking, by **different strategies**:
|
||
|
||
- **The Node side bounds the run.** `[^"]{0,256}`, `[^>]{1,256}`. Cost: an attacker who pads
|
||
the attribute past 256 characters falls out of the pattern.
|
||
- **The guard excludes the anchor character.** `[^><]`, `[^\]\[]`, `[^)(]`. Cost: content that
|
||
legitimately contains that character stops matching.
|
||
|
||
**Every divergence on the guard's side is documented at source, and traceable to the commit
|
||
that introduced it.** An earlier draft of this file claimed the two sub-agent bounds were not;
|
||
that was wrong, and it was wrong because the search behind it never looked outside the
|
||
CHANGELOG. Both mechanisms are named in `lexicon.py`'s own module docstring:
|
||
|
||
> **Bounded token gaps** — the two sub-agent patterns whose seed form nested `.*?` are ported
|
||
> with `(?:\S+\s+){0,N}?`.
|
||
>
|
||
> **Anchor exclusion** — […] Measured across all 83 patterns arm by arm, two markdown patterns
|
||
> had this defect; both now exclude the anchor character from the run.
|
||
|
||
`git log -S` separates the two: the eight `[^><]` patterns (six html-obfuscation, two
|
||
hybrid-xss) arrived with `cff0437`, *"fix(output): 19 quadratic regex runs on the output path"*
|
||
— the v0.3.2 campaign, whose CHANGELOG describes exactly this remedy (*"exclude the character
|
||
that opens the pattern's own anchor (`[` for markdown, `<` for tags)"*) across a sweep of
|
||
*"150 patterns across 11 tables"*. The `{0,12}` / `{0,120}` sub-agent bounds are older still:
|
||
they arrived with the original port commit `f397cd9`, so they were never a divergence
|
||
introduced later — they are how that table was transcribed in the first place.
|
||
|
||
The v0.3.4 entry also states the measured recall cost, naming precisely the two exceptions this
|
||
comparison rediscovered: *"URLs containing a literal `(` inside a markdown link target and
|
||
comment bodies containing a literal `(` before the keyword."*
|
||
|
||
Worth recording, because it anticipates the criticism the Node side invites: the guard
|
||
considered bounding those runs and **rejected it**, on the grounds that *"the content is
|
||
attacker-controlled, so padding past a bound would be a one-line bypass."* That is the same
|
||
objection the `{0,256}` witness above demonstrates against the Node table. The two projects
|
||
reached opposite conclusions from the same reasoning, which is the substance of the
|
||
disagreement — not an oversight on either side.
|
||
|
||
Neither strategy is free, and neither is obviously right. That is the decision the two owning
|
||
repositories have to take, and it is not commons' to take for them.
|
||
|
||
### The cost of commons' strategy, measured here — 2026-08-11
|
||
|
||
The section above says neither strategy is free and neither is obviously right. That was
|
||
written without a number for **this** side's cost. There is one now, and it is not symmetric
|
||
with the guard's.
|
||
|
||
The prompt came from the guard (coord, 2026-08-11): their `[^><]` is not a preference but a
|
||
measured ReDoS fix (`cff0437`, 2026-07-31), and they asked commons to measure its own form
|
||
rather than take their word. Measured here — Node v25.8.2 and Python 3.14.0, Intel i9-9880H,
|
||
patterns read out of `lexicon/injection-lexicon.json` and run over `'<script '` repeated,
|
||
the unit that denies the pattern the `>` it needs:
|
||
|
||
| chars | `<script\b[^>]*>` (commons, Node) | `<script\b[^><]*>` (guard form, Node) |
|
||
|---|---|---|
|
||
| 16 000 | 25.5 ms | 0.08 ms |
|
||
| 32 000 | 101.3 ms | 0.06 ms |
|
||
| 64 000 | 413.5 ms | 0.12 ms |
|
||
| 128 000 | 1 676.5 ms | 0.24 ms |
|
||
| 256 000 | 6 681.6 ms | 0.41 ms |
|
||
|
||
×4.0 per doubling against ×2 — quadratic against linear, reproduced in Python `re` on the same
|
||
inputs (0.021 s → 5.44 s over the same range, guard form 0.0001 s → 0.0015 s). It is
|
||
scan-position quadratic, not exponential backtracking: `<script` matches at ~n positions and
|
||
`[^>]*` scans to end of input at each one. Nothing here is catastrophic backtracking, and
|
||
nothing here needs a nested quantifier to happen.
|
||
|
||
> **Correction, 2026-08-11 (commons `v0.4.1`) — two of the Python figures above read low, and
|
||
> the `iframe-src` figure below read low by ~3×.** The guard flagged it (coord,
|
||
> 2026-08-11T19:51:55Z) after measuring the `iframe-src` row themselves and getting roughly 4×
|
||
> our published number. They attributed the gap to measurement surface — their composed
|
||
> `scan_lexicon()` against our standalone regex. **That is not the explanation.** Re-measured
|
||
> here standalone, the same surface the numbers above claim, Python 3.14.0, median-stable across
|
||
> repeated runs (the two 256 000-char `iframe-src` runs landed 3 ms apart in 51 s):
|
||
>
|
||
> | chars | `script-tag` `[^>]*` | `script-tag` `[^><]*` | `iframe-src` `[^>]*` | `iframe-src` `[^><]*` |
|
||
> |---|---|---|---|---|
|
||
> | 32 000 | 87.4 ms | 0.17 ms | 822.7 ms | 0.25 ms |
|
||
> | 100 000 | — | — | 7 859.5 ms | — |
|
||
> | 256 000 | 5 222.6 ms | 1.25 ms | 51 477.4 ms | 2.08 ms |
|
||
>
|
||
> So: the Python `script-tag` figure at 256 000 chars **reproduces** (5.44 s published against
|
||
> 5.22 s measured), and the one at 32 000 chars does **not** (0.021 s against 0.087 s). The
|
||
> `iframe-src` pair below reproduces at neither point. The error ratios are not constant, so a
|
||
> single mis-sized input does not explain it, and the original harness lived in a previous
|
||
> session's scratchpad and is gone — **the cause is not diagnosable and is not guessed at here.**
|
||
> Our standalone 100 000-char figure (7.86 s) sits close to the guard's composed 8.95 s, which
|
||
> is why their surface hypothesis does not survive: the two surfaces differ by much less than
|
||
> the error did.
|
||
>
|
||
> **What does not change:** the shape (quadratic, ×4 per doubling), the direction, and the
|
||
> decision. Every corrected figure is *larger*, so the case against the old forms is stronger
|
||
> than the one originally published, and the new forms are flat under both engines. The guard
|
||
> flagged this because a low number could matter if either side later pins a bound — they were
|
||
> right to, and the number they should pin against is the one in this box.
|
||
|
||
**Two of the 83 patterns have this shape, not one.** Counted over the whole file, 8 patterns
|
||
carry a bare `[^>]` and none carries `[^><]`. Six of the 8 are the html-obfuscation family and
|
||
are **bounded** (`[^>]{1,256}`) — measured linear (51 → 164 ms across a 8× input growth), so
|
||
bounding does work as a mitigation, at the recall cost the section above already records. The
|
||
other two are unbounded `[^>]*`:
|
||
|
||
- `hybrid-xss:script-tag` — the row this file already tracks.
|
||
- `hybrid-xss:iframe-src` — **not previously named by anyone**, found here by counting rather
|
||
than by being told. Same shape, same result: 119.6 ms → 16 857 ms over 32 000 → 256 000
|
||
chars of `'<iframe '`. **Both figures are superseded — see the correction box above:
|
||
822.7 ms → 51 477.4 ms on re-measurement, standalone, same unit.**
|
||
|
||
What this does and does not license:
|
||
|
||
- It does **not** license changing either pattern here. Both are byte-identical to
|
||
`llm-security`'s source, and the behaviour-preservation invariant is not suspended because
|
||
the measurement is unwelcome. Commons carries `[^>]` on provenance and would carry `[^><]`
|
||
had the source been the guard's — that was true before this measurement and stays true after.
|
||
- It does mean the finding is **owed to the owning runtime**, which is `llm-security` for both
|
||
patterns. Sent 2026-08-11, with these numbers and the `iframe-src` row they did not have.
|
||
|
||
**Outcome, same day:** the owning runtime reproduced both rows on its own vendored copy,
|
||
measured the exposure at its real entry-point caps (which the numbers above deliberately do not
|
||
cover — a pattern cost is not an exposure), judged it unacceptable, chose `[^><]*` over
|
||
`[^>]{0,256}` on the grounds that a bound is paddable, and asked commons to carry the change.
|
||
Commons `0.8.0` carries it. Note what did **and did not** license that: the measurement above
|
||
still licenses nothing on its own, and the two bullets stand as written. What licensed the
|
||
change is the *owner's decision*, arriving through the one channel this repository accepts for
|
||
moving a `pattern` value. Had they declined, the rows would still read `[^>]*` today and this
|
||
document would record a permanent divergence instead.
|
||
- The guard's own honest disclosure belongs on the record too: their suite passes under both
|
||
forms (717 green), because the DoS row that ought to discriminate uses `'<script>'` as its
|
||
repeating unit, which supplies the `>` and never exercises the run. The invariant is
|
||
documented there and not test-enforced. A conformance case could hold it — but a fixture that
|
||
pins a *timing* property is a different kind of case from anything in this corpus today, and
|
||
minting one is not commons' call to take alone.
|
||
|
||
## Severity: the 8 hybrid patterns
|
||
|
||
**Resolved 2026-08-09. The two sides never disagreed; only the evidence did.**
|
||
|
||
Commons recorded the `hybrid` family with `severity: null` and a note that the seed dump did
|
||
not supply it, so a consumer **MUST NOT** assume one. The guard's port assigned all eight
|
||
`high`. Copying the guard's value would have converted a documented gap into an unverified
|
||
claim, so it was reported instead — and the report was right: the value **is** `high`,
|
||
confirmed at the module, and `lexicon/injection-lexicon.json` 0.5.0 now carries it. The eight
|
||
differences in the table above are closed.
|
||
|
||
The part worth keeping is where the value lives. It is not a field. The engine assigns it by
|
||
pushing `HYBRID_PATTERNS` matches straight into the `high` bucket at
|
||
`injection-patterns.mjs:274-281`. `severity.mjs` contains **no injection-family severity at
|
||
all** — re-measured 2026-08-10 at `b0de0ca`: `CRITICAL_PATTERNS`, `HIGH_PATTERNS`,
|
||
`MEDIUM_PATTERNS` and `HYBRID_PATTERNS` appear there zero times.
|
||
|
||
~~**The guard's port cites `severity.mjs`.** So the guard held the right value behind a
|
||
citation that leads nowhere, and refusing to copy it was right for a better reason than the
|
||
one given at the time: this particular port could not have read what it claimed to.~~
|
||
**Retracted 2026-08-10. The guard's port cites the right file.** This paragraph was never
|
||
measured here; it restated an assertion received from `llm-security`
|
||
(`20260809T201048Z`: *"Guardens port satte riktig verdi, men kunne ikke ha lest den fra fila
|
||
den oppgir"*) as a commons finding. Measured against the guard's own tree: `severity.mjs`
|
||
has **never** appeared in `src/llm_ingestion_guard/injection_lexicon.json` at any point in
|
||
that file's history (`git log -S` returns no commits), and at `0bf0729` — the commit
|
||
`conformance/manifest.json` pins — the only tree-wide occurrence is `docs/PLAN.md:114`,
|
||
correctly attributing the *report* module to `output.mjs` + `severity.mjs`. The guard's only
|
||
source statement for the lexicon is the `note` at `injection_lexicon.json:3`, and it names
|
||
`injection-patterns.mjs`.
|
||
|
||
Refusing to copy the value was still the right call — but for the plain reason, that a port
|
||
is second-hand evidence, not for the sharper one claimed above. The sharper reason was itself
|
||
a wrong citation to a right value, which is the defect this section was written to warn
|
||
about. It survived here because it arrived from a repository that had measured the *other*
|
||
half of the claim correctly, and the correct half carried the incorrect half past review.
|
||
|
||
## What this does not show
|
||
|
||
- ~~**Not that either runtime misses an attack.**~~ **Retracted 2026-08-09. It does.** This
|
||
bullet claimed that every witness payload still produced a finding via
|
||
`active-content: constructs.raw-html`, so no attack went unflagged. The measurement behind
|
||
it was wrong in method, not in arithmetic: the payloads were run against the **union of
|
||
every pattern table this repository holds** — 111 rules across the lexicon,
|
||
`active-content.json` and `secret-egress.json` — and the rescuing hit came from
|
||
`active-content.json`. That table is the **Python guard's**. `llm-security` has no
|
||
active-content table at all. A union of commons tables is not any single runtime's
|
||
coverage, and treating it as one turned two runtimes' combined reach into a claim about
|
||
each of them.
|
||
|
||
Measured properly, through `llm-security`'s own entry point `scanForInjection()` — the
|
||
whole engine, with normalisation, homoglyph folding, the rot13 variant and all four pattern
|
||
arrays, at `b0de0ca`:
|
||
|
||
| Witness | `scanForInjection()` result |
|
||
| --- | --- |
|
||
| `<script>alert(1)` (unclosed) | `found: false` — no finding at all |
|
||
| `<script src=x.js>` | `found: false` — no finding at all |
|
||
| `<a aria-label="` + 300 × `a` + `ignore">` | `found: false` — no finding at all |
|
||
|
||
Controls in the same run behave as expected: `<script>alert(1)</script>` returns `high`
|
||
(hybrid-xss), and the short aria-label variant returns `critical`. So the `{0,256}` window
|
||
is a real evasion window and the `<script>` pattern really does require a closing tag.
|
||
`llm-security` reached the same three results independently and attributes the cause to
|
||
their own v7.8.3 #24 ReDoS hardening, which traded recall for boundedness without seeing
|
||
the window. Three confirmed recall holes, logged there as a v8.x task.
|
||
|
||
What survives from the original bullet is only this: the divergence is *also* in the
|
||
finding set, which is what a `conformance/expected.json` encodes.
|
||
- **Not that the guard misses an attack.** The guard's side of the 13 was re-checked the same
|
||
way, and the one row that looked like a miss (`hybrid-xss: iframe with executable src`) is
|
||
covered by a sibling pattern at the same severity and anchor. Its remaining divergences are
|
||
label-set and span differences.
|
||
- **Not dump-to-module fidelity.** *Superseded 2026-08-09.* Every check above proves the two
|
||
*ports* agree or disagree. Commons' side is now settled separately: the lexicon is verified
|
||
byte-identical to `injection-patterns.mjs` at `b0de0ca`, 83/83, which is recorded in the
|
||
data file rather than here. The guard's fidelity to the module remains its own to
|
||
establish.
|
||
- **Not exhaustive.** The corpus is targeted per pattern family, 401 comparisons. Absence of a
|
||
witness proves nothing except for the 6 escaping-only pairs, which are settled by string
|
||
identity rather than by the corpus.
|
||
|
||
## Consequence for `conformance/`
|
||
|
||
An `expected.json` names findings. Naming a finding needs a stable id, and the two runtimes do
|
||
not have one: the same pattern is `override:ignore-previous` in the guard and
|
||
`override: ignore previous instructions` in the Node table. The guard's JSON happens to carry
|
||
both — `id` and `desc` — which is evidence that a commons-owned id is achievable rather than
|
||
speculative.
|
||
|
||
So the id question is a **prerequisite** for the corpus, not a parallel task: until commons
|
||
owns a pattern id both ports map to, no fixture can be written, including for the 64 patterns
|
||
that are byte-identical. And for the 13 divergent patterns a fixture cannot be authored at all
|
||
without first deciding whose recall cost is the contract.
|
||
|
||
**Resolved for the first half, 2026-08-09 (operator decision).** `lexicon/injection-lexicon.json`
|
||
0.2.0 now carries a commons-owned `id` per pattern, plus an `aliases` object naming what each
|
||
seeding runtime calls it. The id was **adopted verbatim from the guard's port**, not invented
|
||
here — that port already carried both names, so the mapping came from source data. The
|
||
detection data is provably unmoved: stripping `id`, `aliases` and the new `pattern_id_space`
|
||
block reproduces the previous committed file **byte for byte** (23 566 bytes, identical).
|
||
|
||
**Ratified by both runtimes, 2026-08-09.** `llm-security` accepted the id space as-is,
|
||
including the 0.2.0 proposal, and treats an id change as breaking on the same terms; the
|
||
guard confirmed the space its own port supplied. `lexicon/injection-lexicon.json` 0.5.0
|
||
records both. The id is a cross-runtime contract now, not a proposal.
|
||
|
||
~~**The second half of the blocker stands, and it did not get smaller.**~~ **Dissolved
|
||
2026-08-10 by measurement, not by a decision.** This paragraph said the 13 divergent
|
||
patterns had no agreed expected behaviour, so their fixtures could not be authored, and that
|
||
someone would have to pick whose recall cost was the contract.
|
||
|
||
Nobody had to. The question was never asked of the right inputs. Every divergence in the
|
||
table above was found on a **witness** input — an attribute run padded past 256 characters,
|
||
an interior `<`, an unclosed `<script>`. The corpus is built from the seed suite's payloads,
|
||
which are short, unpadded and contain none of those shapes. Run through both runtimes'
|
||
public entry points, all 83 patterns produce **identical lexicon finding sets**, and that
|
||
includes 13 of 13 of the divergent ones on their own case input. Method, commits and counts:
|
||
[`conformance/manifest.json`](../conformance/manifest.json).
|
||
|
||
So the 13 carry no marker in the corpus and no caveat. Marking them would assert a doubt the
|
||
measurement disproves for these inputs, which is a different defect from the one it would
|
||
appear to prevent.
|
||
|
||
**What still stands is everything above this heading.** The divergence is real, it is
|
||
unresolved, and it will reappear the moment a fixture is written over a witness input.
|
||
`llm-security` has decided **not** to adopt the guard's regex strategy at this point: v0.1.0
|
||
is a behaviour-preservation release on their side too, and swapping strategies mid-vendoring
|
||
would void their own golden gate. Both behaviours stay registered as known divergence per
|
||
pattern. Their three confirmed recall holes are logged as a v8.x task; when it lands they
|
||
will say so, and those rows can close then. Until then the correct description of each is
|
||
*"known divergence, `llm-security` side has an open recall hole, measured 2026-08-09; not
|
||
reachable from any input in the v0.1.0 corpus"* — not *"undecided"*, and not *"resolved"*.
|