7ce0ba7 docs(carriers): the third verdict exists, and publishing an alias is what takes it away 302625e fix(conformance): the tag carrier has no output: label, and our blocker claimed it did fe5e6b2 docs(conventions): the merge button is off for a reason, and the reason now lives in the repo 2d86151 fix(divergence): our own iframe number read 3x low, and the reported cause was not the cause daa7ba4 release(0.4.0): two values moved by two mechanisms, and the difference is the release 2eee7e1 feat(lexicon): both unbounded rows narrow to [^><]*, and the mechanism is new here d467324 feat(signatures): the staleness we disclosed is closed by reading the module, not the message 4187715 docs(divergence): our own form has a number now, and it is quadratic 0e765a0 docs(security): the attack surface here is data, so the report route had to say where a wrong entry gets fixed d96fbbf docs(divergence): the span row had one witness; llm-security measured five more git-subtree-dir: scanners/commons git-subtree-split: 7ce0ba706cadd032ec3c16622dcfdb5ce4dc32d6
496 lines
32 KiB
Markdown
496 lines
32 KiB
Markdown
# Lexicon port divergence — commons vs the Python guard
|
||
|
||
**Status: informative.** Nothing here is normative and nothing here changes a data file. It
|
||
records a measured disagreement between two ports of one source table, so the decision can be
|
||
taken where each table is tested. Under this repository's behaviour-preservation invariant,
|
||
a divergence found here is **reported, not fixed**.
|
||
|
||
That invariant held through the closure recorded below. Two rows this document measured did change
|
||
in `lexicon/injection-lexicon.json` `0.8.0` — and not because this document found them wrong. The
|
||
finding was routed to the runtime that owns the value, that runtime re-measured it, decided, and
|
||
asked commons to carry the result. The measurement travelled; the authority did not.
|
||
|
||
Produced 2026-08-09. Every number below came from a command; the scripts live in the session
|
||
scratchpad rather than in this repository, because executable code here would breach the
|
||
charter. They are reproducible from the method column.
|
||
|
||
**Revised the same day, after `llm-security`'s source became readable and both runtimes
|
||
replied.** Four things changed, and three of them are corrections to this file rather than
|
||
new results:
|
||
|
||
1. The claim that **neither runtime misses an attack** is **retracted**. It does. See
|
||
*What this does not show* — the measurement behind that claim unioned pattern tables
|
||
belonging to two different runtimes and read the result as a statement about each.
|
||
2. One of the 13 divergences does not reach report level, so **12** is the number that
|
||
changes what a report says. The 13 still blocks `conformance/`.
|
||
3. The `hybrid` **severity is resolved** to `high` — the reported hint was correct, and the
|
||
citation behind it was not.
|
||
4. The **pattern id space is ratified** by both runtimes.
|
||
|
||
Corrections are marked in place rather than edited away, because a reader who saw the first
|
||
version needs to know which sentence moved.
|
||
|
||
## What was compared
|
||
|
||
| Side | Artefact | Version |
|
||
| --- | --- | --- |
|
||
| commons | [`lexicon/injection-lexicon.json`](../lexicon/injection-lexicon.json) | file `version` as committed |
|
||
| guard | `llm-ingestion-pipeline-security` `src/llm_ingestion_guard/injection_lexicon.json` | lexicon `version` 1.0, repo v0.3.4, commit `0bf0729` |
|
||
|
||
Both are **ports of the same file**: `llm-security/scanners/lib/injection-patterns.mjs`. The
|
||
guard's JSON says so in its own `note` field — *"Injection lexicon ported from llm-security
|
||
injection-patterns.mjs. Single source of truth."* Commons extracted the same table from an
|
||
operator dump of that module.
|
||
|
||
That is what makes the comparison worth running. These are not two different detectors that
|
||
happen to overlap; they are two transcriptions of one table, and where they disagree, they
|
||
disagree about what the same source says.
|
||
|
||
## Result
|
||
|
||
| Measure | Method | Result |
|
||
| --- | --- | --- |
|
||
| Pattern count, both sides | count entries | 83 and 83 |
|
||
| Label correspondence | match commons `label` to guard `desc`, em-dash normalised to hyphen | **83/83** |
|
||
| Regex source byte-identical | string compare | **64/83** |
|
||
| Differing regex text | string compare | 19 |
|
||
| — of those, provably equivalent | unescape commons' JS-isms (`\/` → `/`, `\uXXXX` → literal) and compare for string identity | **6/6 identical** |
|
||
| — of those, behaviourally divergent | differential match-set comparison (offsets + matched text), targeted corpus per pattern family | **13**, each with a concrete witness input |
|
||
| — of those, divergent at REPORT level | re-check whether a sibling pattern raises an equivalent finding on the same witness | **12** — one of the 13 is a label-set difference only |
|
||
| Total input comparisons | count | 401 |
|
||
| Flags | compare declared flags | **0 differences** |
|
||
| Severity / family | commons family vs guard `severity` | **8 differences** (all `hybrid`) — resolved, see *Severity* below |
|
||
|
||
64 identical + 6 escaping-only + 13 divergent = 83.
|
||
|
||
**Read the 13 and the 12 as answering different questions.** Thirteen patterns produce
|
||
different label sets. Twelve of those change what a report would say. The gap is the
|
||
`hybrid-xss: iframe with executable src` row: the guard's version of that one pattern misses
|
||
the witness, but its `hybrid-xss: javascript: URI scheme` pattern fires on the same input at
|
||
the same severity and the same OWASP anchor, so a reader of the guard's report still sees a
|
||
`high` / `LLM01` finding on that payload. Measured, not reasoned: the guard's table matched
|
||
`hybrid-xss:javascript-uri` (high, LLM01) and nothing else; the Node engine matched both
|
||
`hybrid-xss: javascript: URI scheme` and `hybrid-xss: iframe with executable src`. A
|
||
`conformance/expected.json` would have to encode the difference, because a fixture names
|
||
labels — which is why the 13 stood as the number that blocked the corpus. *(Superseded
|
||
2026-08-10: it blocks a fixture written over a **witness** input, and the corpus contains
|
||
none. See* Consequence for `conformance/` *below.)*
|
||
|
||
The 6 escaping-only pairs are equivalent by **string identity after unescaping**, not merely
|
||
by absence of a counter-example: `<\/system>` vs `</system>`, `text\/html` vs `text/html`,
|
||
`(?:https?:\/\/…)` vs `(?:https?://…)`, `(?:~\/)?` vs `(?:~/)?`, and two `\uXXXX`-vs-literal
|
||
character classes. Commons writes code points as escapes by convention; the guard's `note`
|
||
records the same choice for non-ASCII.
|
||
|
||
## The 13, and which side misses what
|
||
|
||
Divergence is **not one-directional**. Each side has inputs the other's pattern does not match.
|
||
|
||
| Pattern | Direction | Witness input |
|
||
| --- | --- | --- |
|
||
| `html-obfuscation: display:none element with content` | commons misses; span differs | attribute run padded past 256 chars |
|
||
| `html-obfuscation: visibility:hidden element` | span only | `<div data-x="<" style="visibility:hidden">x</div>` |
|
||
| `html-obfuscation: off-screen positioned element` | span only | same shape, `position:absolute` |
|
||
| `html-obfuscation: zero font-size element` | span only | same shape, `font-size:0` |
|
||
| `html-obfuscation: zero opacity element` | span only | same shape, `opacity:0` |
|
||
| `html-obfuscation: zero-size overflow-hidden element` | span only | same shape, `height:0;overflow:hidden` |
|
||
| `html-obfuscation: injection in aria-label attribute` | **commons misses** | `<a aria-label="` + 300 × `a` + `ignore">x</a>` |
|
||
| `markdown: suspicious instruction in link-reference comment` | **guard misses** | `[//]: # (a (b) ignore)` |
|
||
| `markdown: injection payload in link anchor text` | **guard misses** | `[click ignore this](http://e.com/a(b))` |
|
||
| `sub-agent: spawn instruction with dangerous capability keywords` | **guard misses** | `create an agent that` + 20 filler words + `execute` |
|
||
| `sub-agent: delegation with safety bypass instruction` | **guard misses** | `delegate to a new agent` + 200 chars + `bypass` |
|
||
| `hybrid-xss: <script> tag in content (agent context XSS)` | **reversed 2026-08-11, see below** | `<script>alert(1)` (unclosed), `<script src=x.js>` |
|
||
| `hybrid-xss: iframe with executable src (agent context XSS)` | **guard misses the label, not the attack** | `<iframe data-x="<" src="javascript:alert(1)">` |
|
||
|
||
The last row is the one of the 13 that does not reach report level. The guard's
|
||
`hybrid-xss:javascript-uri` (`javascript\s*:`, high, LLM01) matches that witness, so the
|
||
payload is still flagged at the same severity and anchor; only the label set differs — one
|
||
finding instead of two. The remaining 12 rows change what a report says.
|
||
|
||
"Span only" means both sides produce a match on the same input but over different extents —
|
||
the guard's match starts at an interior `<`. Whether that matters depends on whether a
|
||
consumer reports offsets or evidence text; it does not change whether a finding is raised.
|
||
|
||
The commons-side misses were confirmed in a real JS engine (Node v25.8.2, `RegExp` built from
|
||
the committed JSON), not only in the Python harness used for the differential.
|
||
|
||
### `hybrid-xss:script-tag`: the row reversed, 2026-08-11
|
||
|
||
At the time this file was written, `<script>alert(1)` (unclosed) and `<script src=x.js>` were
|
||
commons misses only in the sense that commons was a byte-identical extraction of
|
||
`llm-security`, so "commons misses" meant "commons and `llm-security` both miss, the guard
|
||
does not." That symmetry broke on `llm-security`'s side.
|
||
|
||
`llm-security` commit `90f576f` (`fix(llm-security): v8.x-A - close <script> recall hole, add
|
||
whole-table ReDoS gate`) replaced the source pattern
|
||
|
||
```
|
||
<script\b[^>]*>[\s\S]*?<\/script>
|
||
```
|
||
|
||
with
|
||
|
||
```
|
||
<script\b[^>]*>
|
||
```
|
||
|
||
dropping the closing-tag requirement that produced the recall hole this file first measured.
|
||
Both witnesses now match on `llm-security`'s side. Verified directly against the module at
|
||
commit `c67bad3` (2026-08-10, first commit carrying the change) and re-confirmed unmoved at
|
||
`b1ba1fb` (2026-08-11, current public-remote HEAD as fetched this session) — `git diff
|
||
c67bad3..b1ba1fb -- scanners/lib/injection-patterns.mjs` is empty, so no later commit touched
|
||
the pattern a second time.
|
||
|
||
The guard's pattern was never the same construct: `<script\b[^><]*>`, negated-class excluding
|
||
both `>` and `<` (`src/llm_ingestion_guard/injection_lexicon.json:463`, unchanged since
|
||
`de09711`, re-confirmed unmoved through current guard HEAD `0dce50f`, 2026-08-11). The two
|
||
patterns now agree on **shape** — both drop the closing-tag requirement, both are a single
|
||
negated-class run immediately after `<script\b` — but they are still not byte-identical: the
|
||
guard's excluded-character class also excludes `<`, `llm-security`'s does not. That one-byte
|
||
difference is unresolved on both sides (see *Open questions* in the coordination thread); it
|
||
does not reopen the closing-tag question, which is settled.
|
||
|
||
**Commons converged in lexicon `0.7.0` (2026-08-11), and this row is closed on the closing-tag
|
||
axis.** Through `0.6.0` commons was the sole diverger of the three: both `llm-security` and the
|
||
guard matched `<script src=x.js>` and unclosed `<script>alert(1)`, and commons' extracted pattern
|
||
still required `</script>`. Commons now carries `<script\b[^>]*>` — byte-identical to
|
||
`llm-security`'s `RegExp.prototype.source` at `b1ba1fb`, verified by compiling both.
|
||
|
||
The decision was **re-extraction, not revision**, and the distinction is what made it available
|
||
at all under the behaviour-preservation invariant. That invariant forbids commons from correcting
|
||
seed data it believes is wrong; it does not forbid re-reading the source after the source itself
|
||
moved and its owner asked. The lexicon's declared provenance is `llm-security`'s injection table,
|
||
and being loadable verbatim by that table's owner is the one thing it exists to do — the standing
|
||
alternative was a per-pattern override in `llm-security`'s own loader, i.e. a published core that
|
||
its source repository could not load as published. Operator decision, 2026-08-11, on
|
||
`llm-security`'s blocking coord request. Explicitly **not** decided by the 2-of-3 majority: a
|
||
count of implementations is not a mandate over detection data, and the provenance argument would
|
||
have held with the guard on either side.
|
||
|
||
The direction matters for what it cannot break: the new form matches a strict superset of the old
|
||
one, so relative to `0.6.0` it can add matches and cannot add a false negative. The reverse change
|
||
would not have been adoptable on the same reasoning.
|
||
|
||
**The one-byte span difference is CLOSED as of commons `0.8.0`** — all three runtimes now carry
|
||
`[^><]*`. What follows is the account of the divergence while it was open, kept because the table
|
||
below is the measurement that made the closure decidable, and because a consumer diffing against
|
||
`v0.3.0` will find the two changed rows and needs the coordinate.
|
||
|
||
While it was open, the guard excluded `<` from its negated class and the other two did not. The
|
||
three forms turn out to be **totally ordered by what they match** — each a strict superset of the
|
||
next:
|
||
|
||
| Witness | commons `0.7.0` / `llm-security` `[^>]` | guard `[^><]`, and commons from `0.8.0` | commons ≤ `0.6.0` |
|
||
| --- | --- | --- | --- |
|
||
| `<script>steal()</script>` | match | match | match |
|
||
| `<script src=x.js>` | match | match | no match |
|
||
| `<script>alert(1)` | match | match | no match |
|
||
| `<script\n src=x>` | match | match | no match |
|
||
| `<script a="<" >x` | **match** | **no match** | no match |
|
||
| `<script<div>` | **match** | **no match** | no match |
|
||
|
||
Measured by compiling all three forms, not reasoned from the character classes. The last two rows
|
||
are the whole residual divergence.
|
||
|
||
The witness set and the ordering observation are `llm-security`'s, from their coord message of
|
||
2026-08-11T11:01:59Z; both were **reproduced here independently** before being recorded, including
|
||
the strict-superset property in both directions. Their reading of it: `[^><]` refuses to cross a
|
||
`<`, and since anything that reaches `[^>]`-and-not-`[^><]` already contains a literal `<script`
|
||
opening tag, the narrower class buys an empty false-positive surface in exchange for two dropped
|
||
input classes — a recall loss with no precision gain to trade for.
|
||
|
||
**That argument is recorded, not endorsed, and it did not decide commons' form.** Commons carries
|
||
`[^>]` because the lexicon's declared provenance is `llm-security`'s table (see above), not
|
||
because their regex won on merit — the same reasoning would have taken commons to `[^><]` had the
|
||
source been the guard's. `llm-security` asked to hear the guard's reason for `[^><]` — a ReDoS
|
||
bound, or a span-extraction consumer needing the run to stop at `<` — before commons shipped, and
|
||
commons shipped first. That order is deliberate rather than an oversight: a reason that beats this
|
||
argument is a reason for the *guard's own port*, and commons would still track its declared
|
||
source. Put to the guard as an open question 2026-08-11; unanswered at time of writing.
|
||
|
||
**How it resolved, and why the resolution used the same rule rather than overturning it.** Both
|
||
halves of that open question came back the same day. The guard answered that `[^><]` is a
|
||
*measured* ReDoS fix on their side (their commit `cff0437`), not an aesthetic choice — so the
|
||
"reason that beats this argument" the paragraph above asked for turned out to exist. And commons
|
||
then measured its own copy and found the same defect in it: `<script\b[^>]*>` and
|
||
`<iframe\b[^>]*src…` are both quadratic in scan length, the second one having been named by nobody
|
||
until it was counted. `llm-security` re-measured rather than taking the report, reached `[^><]*`
|
||
independently, and asked commons to carry it — because their table is now *built from this file*,
|
||
so there is no local literal left for them to change without forking.
|
||
|
||
The paragraph above said commons "would still track its declared source." That is exactly what
|
||
`0.8.0` does: the declared source chose `[^><]*` and commons followed. The rule did not bend to
|
||
accommodate a better argument — the argument won *at the source*, which is the only place this
|
||
repository lets an argument decide a `pattern` value. The recall cost stated in the two bold rows
|
||
of the table is now paid deliberately: measured across all 90 conformance cases, zero lost a match
|
||
and zero gained one, so the dropped classes remain real and remain unwitnessed by the corpus.
|
||
|
||
## Why they diverge: two different ReDoS mitigations of one table
|
||
|
||
This is not drift, and framing it as a bug in either repository would be wrong.
|
||
|
||
Both ports have been hardened against catastrophic backtracking, by **different strategies**:
|
||
|
||
- **The Node side bounds the run.** `[^"]{0,256}`, `[^>]{1,256}`. Cost: an attacker who pads
|
||
the attribute past 256 characters falls out of the pattern.
|
||
- **The guard excludes the anchor character.** `[^><]`, `[^\]\[]`, `[^)(]`. Cost: content that
|
||
legitimately contains that character stops matching.
|
||
|
||
**Every divergence on the guard's side is documented at source, and traceable to the commit
|
||
that introduced it.** An earlier draft of this file claimed the two sub-agent bounds were not;
|
||
that was wrong, and it was wrong because the search behind it never looked outside the
|
||
CHANGELOG. Both mechanisms are named in `lexicon.py`'s own module docstring:
|
||
|
||
> **Bounded token gaps** — the two sub-agent patterns whose seed form nested `.*?` are ported
|
||
> with `(?:\S+\s+){0,N}?`.
|
||
>
|
||
> **Anchor exclusion** — […] Measured across all 83 patterns arm by arm, two markdown patterns
|
||
> had this defect; both now exclude the anchor character from the run.
|
||
|
||
`git log -S` separates the two: the eight `[^><]` patterns (six html-obfuscation, two
|
||
hybrid-xss) arrived with `cff0437`, *"fix(output): 19 quadratic regex runs on the output path"*
|
||
— the v0.3.2 campaign, whose CHANGELOG describes exactly this remedy (*"exclude the character
|
||
that opens the pattern's own anchor (`[` for markdown, `<` for tags)"*) across a sweep of
|
||
*"150 patterns across 11 tables"*. The `{0,12}` / `{0,120}` sub-agent bounds are older still:
|
||
they arrived with the original port commit `f397cd9`, so they were never a divergence
|
||
introduced later — they are how that table was transcribed in the first place.
|
||
|
||
The v0.3.4 entry also states the measured recall cost, naming precisely the two exceptions this
|
||
comparison rediscovered: *"URLs containing a literal `(` inside a markdown link target and
|
||
comment bodies containing a literal `(` before the keyword."*
|
||
|
||
Worth recording, because it anticipates the criticism the Node side invites: the guard
|
||
considered bounding those runs and **rejected it**, on the grounds that *"the content is
|
||
attacker-controlled, so padding past a bound would be a one-line bypass."* That is the same
|
||
objection the `{0,256}` witness above demonstrates against the Node table. The two projects
|
||
reached opposite conclusions from the same reasoning, which is the substance of the
|
||
disagreement — not an oversight on either side.
|
||
|
||
Neither strategy is free, and neither is obviously right. That is the decision the two owning
|
||
repositories have to take, and it is not commons' to take for them.
|
||
|
||
### The cost of commons' strategy, measured here — 2026-08-11
|
||
|
||
The section above says neither strategy is free and neither is obviously right. That was
|
||
written without a number for **this** side's cost. There is one now, and it is not symmetric
|
||
with the guard's.
|
||
|
||
The prompt came from the guard (coord, 2026-08-11): their `[^><]` is not a preference but a
|
||
measured ReDoS fix (`cff0437`, 2026-07-31), and they asked commons to measure its own form
|
||
rather than take their word. Measured here — Node v25.8.2 and Python 3.14.0, Intel i9-9880H,
|
||
patterns read out of `lexicon/injection-lexicon.json` and run over `'<script '` repeated,
|
||
the unit that denies the pattern the `>` it needs:
|
||
|
||
| chars | `<script\b[^>]*>` (commons, Node) | `<script\b[^><]*>` (guard form, Node) |
|
||
|---|---|---|
|
||
| 16 000 | 25.5 ms | 0.08 ms |
|
||
| 32 000 | 101.3 ms | 0.06 ms |
|
||
| 64 000 | 413.5 ms | 0.12 ms |
|
||
| 128 000 | 1 676.5 ms | 0.24 ms |
|
||
| 256 000 | 6 681.6 ms | 0.41 ms |
|
||
|
||
×4.0 per doubling against ×2 — quadratic against linear, reproduced in Python `re` on the same
|
||
inputs (0.021 s → 5.44 s over the same range, guard form 0.0001 s → 0.0015 s). It is
|
||
scan-position quadratic, not exponential backtracking: `<script` matches at ~n positions and
|
||
`[^>]*` scans to end of input at each one. Nothing here is catastrophic backtracking, and
|
||
nothing here needs a nested quantifier to happen.
|
||
|
||
> **Correction, 2026-08-11 (commons `v0.4.1`) — two of the Python figures above read low, and
|
||
> the `iframe-src` figure below read low by ~3×.** The guard flagged it (coord,
|
||
> 2026-08-11T19:51:55Z) after measuring the `iframe-src` row themselves and getting roughly 4×
|
||
> our published number. They attributed the gap to measurement surface — their composed
|
||
> `scan_lexicon()` against our standalone regex. **That is not the explanation.** Re-measured
|
||
> here standalone, the same surface the numbers above claim, Python 3.14.0, median-stable across
|
||
> repeated runs (the two 256 000-char `iframe-src` runs landed 3 ms apart in 51 s):
|
||
>
|
||
> | chars | `script-tag` `[^>]*` | `script-tag` `[^><]*` | `iframe-src` `[^>]*` | `iframe-src` `[^><]*` |
|
||
> |---|---|---|---|---|
|
||
> | 32 000 | 87.4 ms | 0.17 ms | 822.7 ms | 0.25 ms |
|
||
> | 100 000 | — | — | 7 859.5 ms | — |
|
||
> | 256 000 | 5 222.6 ms | 1.25 ms | 51 477.4 ms | 2.08 ms |
|
||
>
|
||
> So: the Python `script-tag` figure at 256 000 chars **reproduces** (5.44 s published against
|
||
> 5.22 s measured), and the one at 32 000 chars does **not** (0.021 s against 0.087 s). The
|
||
> `iframe-src` pair below reproduces at neither point. The error ratios are not constant, so a
|
||
> single mis-sized input does not explain it, and the original harness lived in a previous
|
||
> session's scratchpad and is gone — **the cause is not diagnosable and is not guessed at here.**
|
||
> Our standalone 100 000-char figure (7.86 s) sits close to the guard's composed 8.95 s, which
|
||
> is why their surface hypothesis does not survive: the two surfaces differ by much less than
|
||
> the error did.
|
||
>
|
||
> **What does not change:** the shape (quadratic, ×4 per doubling), the direction, and the
|
||
> decision. Every corrected figure is *larger*, so the case against the old forms is stronger
|
||
> than the one originally published, and the new forms are flat under both engines. The guard
|
||
> flagged this because a low number could matter if either side later pins a bound — they were
|
||
> right to, and the number they should pin against is the one in this box.
|
||
|
||
**Two of the 83 patterns have this shape, not one.** Counted over the whole file, 8 patterns
|
||
carry a bare `[^>]` and none carries `[^><]`. Six of the 8 are the html-obfuscation family and
|
||
are **bounded** (`[^>]{1,256}`) — measured linear (51 → 164 ms across a 8× input growth), so
|
||
bounding does work as a mitigation, at the recall cost the section above already records. The
|
||
other two are unbounded `[^>]*`:
|
||
|
||
- `hybrid-xss:script-tag` — the row this file already tracks.
|
||
- `hybrid-xss:iframe-src` — **not previously named by anyone**, found here by counting rather
|
||
than by being told. Same shape, same result: 119.6 ms → 16 857 ms over 32 000 → 256 000
|
||
chars of `'<iframe '`. **Both figures are superseded — see the correction box above:
|
||
822.7 ms → 51 477.4 ms on re-measurement, standalone, same unit.**
|
||
|
||
What this does and does not license:
|
||
|
||
- It does **not** license changing either pattern here. Both are byte-identical to
|
||
`llm-security`'s source, and the behaviour-preservation invariant is not suspended because
|
||
the measurement is unwelcome. Commons carries `[^>]` on provenance and would carry `[^><]`
|
||
had the source been the guard's — that was true before this measurement and stays true after.
|
||
- It does mean the finding is **owed to the owning runtime**, which is `llm-security` for both
|
||
patterns. Sent 2026-08-11, with these numbers and the `iframe-src` row they did not have.
|
||
|
||
**Outcome, same day:** the owning runtime reproduced both rows on its own vendored copy,
|
||
measured the exposure at its real entry-point caps (which the numbers above deliberately do not
|
||
cover — a pattern cost is not an exposure), judged it unacceptable, chose `[^><]*` over
|
||
`[^>]{0,256}` on the grounds that a bound is paddable, and asked commons to carry the change.
|
||
Commons `0.8.0` carries it. Note what did **and did not** license that: the measurement above
|
||
still licenses nothing on its own, and the two bullets stand as written. What licensed the
|
||
change is the *owner's decision*, arriving through the one channel this repository accepts for
|
||
moving a `pattern` value. Had they declined, the rows would still read `[^>]*` today and this
|
||
document would record a permanent divergence instead.
|
||
- The guard's own honest disclosure belongs on the record too: their suite passes under both
|
||
forms (717 green), because the DoS row that ought to discriminate uses `'<script>'` as its
|
||
repeating unit, which supplies the `>` and never exercises the run. The invariant is
|
||
documented there and not test-enforced. A conformance case could hold it — but a fixture that
|
||
pins a *timing* property is a different kind of case from anything in this corpus today, and
|
||
minting one is not commons' call to take alone.
|
||
|
||
## Severity: the 8 hybrid patterns
|
||
|
||
**Resolved 2026-08-09. The two sides never disagreed; only the evidence did.**
|
||
|
||
Commons recorded the `hybrid` family with `severity: null` and a note that the seed dump did
|
||
not supply it, so a consumer **MUST NOT** assume one. The guard's port assigned all eight
|
||
`high`. Copying the guard's value would have converted a documented gap into an unverified
|
||
claim, so it was reported instead — and the report was right: the value **is** `high`,
|
||
confirmed at the module, and `lexicon/injection-lexicon.json` 0.5.0 now carries it. The eight
|
||
differences in the table above are closed.
|
||
|
||
The part worth keeping is where the value lives. It is not a field. The engine assigns it by
|
||
pushing `HYBRID_PATTERNS` matches straight into the `high` bucket at
|
||
`injection-patterns.mjs:274-281`. `severity.mjs` contains **no injection-family severity at
|
||
all** — re-measured 2026-08-10 at `b0de0ca`: `CRITICAL_PATTERNS`, `HIGH_PATTERNS`,
|
||
`MEDIUM_PATTERNS` and `HYBRID_PATTERNS` appear there zero times.
|
||
|
||
~~**The guard's port cites `severity.mjs`.** So the guard held the right value behind a
|
||
citation that leads nowhere, and refusing to copy it was right for a better reason than the
|
||
one given at the time: this particular port could not have read what it claimed to.~~
|
||
**Retracted 2026-08-10. The guard's port cites the right file.** This paragraph was never
|
||
measured here; it restated an assertion received from `llm-security`
|
||
(`20260809T201048Z`: *"Guardens port satte riktig verdi, men kunne ikke ha lest den fra fila
|
||
den oppgir"*) as a commons finding. Measured against the guard's own tree: `severity.mjs`
|
||
has **never** appeared in `src/llm_ingestion_guard/injection_lexicon.json` at any point in
|
||
that file's history (`git log -S` returns no commits), and at `0bf0729` — the commit
|
||
`conformance/manifest.json` pins — the only tree-wide occurrence is `docs/PLAN.md:114`,
|
||
correctly attributing the *report* module to `output.mjs` + `severity.mjs`. The guard's only
|
||
source statement for the lexicon is the `note` at `injection_lexicon.json:3`, and it names
|
||
`injection-patterns.mjs`.
|
||
|
||
Refusing to copy the value was still the right call — but for the plain reason, that a port
|
||
is second-hand evidence, not for the sharper one claimed above. The sharper reason was itself
|
||
a wrong citation to a right value, which is the defect this section was written to warn
|
||
about. It survived here because it arrived from a repository that had measured the *other*
|
||
half of the claim correctly, and the correct half carried the incorrect half past review.
|
||
|
||
## What this does not show
|
||
|
||
- ~~**Not that either runtime misses an attack.**~~ **Retracted 2026-08-09. It does.** This
|
||
bullet claimed that every witness payload still produced a finding via
|
||
`active-content: constructs.raw-html`, so no attack went unflagged. The measurement behind
|
||
it was wrong in method, not in arithmetic: the payloads were run against the **union of
|
||
every pattern table this repository holds** — 111 rules across the lexicon,
|
||
`active-content.json` and `secret-egress.json` — and the rescuing hit came from
|
||
`active-content.json`. That table is the **Python guard's**. `llm-security` has no
|
||
active-content table at all. A union of commons tables is not any single runtime's
|
||
coverage, and treating it as one turned two runtimes' combined reach into a claim about
|
||
each of them.
|
||
|
||
Measured properly, through `llm-security`'s own entry point `scanForInjection()` — the
|
||
whole engine, with normalisation, homoglyph folding, the rot13 variant and all four pattern
|
||
arrays, at `b0de0ca`:
|
||
|
||
| Witness | `scanForInjection()` result |
|
||
| --- | --- |
|
||
| `<script>alert(1)` (unclosed) | `found: false` — no finding at all |
|
||
| `<script src=x.js>` | `found: false` — no finding at all |
|
||
| `<a aria-label="` + 300 × `a` + `ignore">` | `found: false` — no finding at all |
|
||
|
||
Controls in the same run behave as expected: `<script>alert(1)</script>` returns `high`
|
||
(hybrid-xss), and the short aria-label variant returns `critical`. So the `{0,256}` window
|
||
is a real evasion window and the `<script>` pattern really does require a closing tag.
|
||
`llm-security` reached the same three results independently and attributes the cause to
|
||
their own v7.8.3 #24 ReDoS hardening, which traded recall for boundedness without seeing
|
||
the window. Three confirmed recall holes, logged there as a v8.x task.
|
||
|
||
What survives from the original bullet is only this: the divergence is *also* in the
|
||
finding set, which is what a `conformance/expected.json` encodes.
|
||
- **Not that the guard misses an attack.** The guard's side of the 13 was re-checked the same
|
||
way, and the one row that looked like a miss (`hybrid-xss: iframe with executable src`) is
|
||
covered by a sibling pattern at the same severity and anchor. Its remaining divergences are
|
||
label-set and span differences.
|
||
- **Not dump-to-module fidelity.** *Superseded 2026-08-09.* Every check above proves the two
|
||
*ports* agree or disagree. Commons' side is now settled separately: the lexicon is verified
|
||
byte-identical to `injection-patterns.mjs` at `b0de0ca`, 83/83, which is recorded in the
|
||
data file rather than here. The guard's fidelity to the module remains its own to
|
||
establish.
|
||
- **Not exhaustive.** The corpus is targeted per pattern family, 401 comparisons. Absence of a
|
||
witness proves nothing except for the 6 escaping-only pairs, which are settled by string
|
||
identity rather than by the corpus.
|
||
|
||
## Consequence for `conformance/`
|
||
|
||
An `expected.json` names findings. Naming a finding needs a stable id, and the two runtimes do
|
||
not have one: the same pattern is `override:ignore-previous` in the guard and
|
||
`override: ignore previous instructions` in the Node table. The guard's JSON happens to carry
|
||
both — `id` and `desc` — which is evidence that a commons-owned id is achievable rather than
|
||
speculative.
|
||
|
||
So the id question is a **prerequisite** for the corpus, not a parallel task: until commons
|
||
owns a pattern id both ports map to, no fixture can be written, including for the 64 patterns
|
||
that are byte-identical. And for the 13 divergent patterns a fixture cannot be authored at all
|
||
without first deciding whose recall cost is the contract.
|
||
|
||
**Resolved for the first half, 2026-08-09 (operator decision).** `lexicon/injection-lexicon.json`
|
||
0.2.0 now carries a commons-owned `id` per pattern, plus an `aliases` object naming what each
|
||
seeding runtime calls it. The id was **adopted verbatim from the guard's port**, not invented
|
||
here — that port already carried both names, so the mapping came from source data. The
|
||
detection data is provably unmoved: stripping `id`, `aliases` and the new `pattern_id_space`
|
||
block reproduces the previous committed file **byte for byte** (23 566 bytes, identical).
|
||
|
||
**Ratified by both runtimes, 2026-08-09.** `llm-security` accepted the id space as-is,
|
||
including the 0.2.0 proposal, and treats an id change as breaking on the same terms; the
|
||
guard confirmed the space its own port supplied. `lexicon/injection-lexicon.json` 0.5.0
|
||
records both. The id is a cross-runtime contract now, not a proposal.
|
||
|
||
~~**The second half of the blocker stands, and it did not get smaller.**~~ **Dissolved
|
||
2026-08-10 by measurement, not by a decision.** This paragraph said the 13 divergent
|
||
patterns had no agreed expected behaviour, so their fixtures could not be authored, and that
|
||
someone would have to pick whose recall cost was the contract.
|
||
|
||
Nobody had to. The question was never asked of the right inputs. Every divergence in the
|
||
table above was found on a **witness** input — an attribute run padded past 256 characters,
|
||
an interior `<`, an unclosed `<script>`. The corpus is built from the seed suite's payloads,
|
||
which are short, unpadded and contain none of those shapes. Run through both runtimes'
|
||
public entry points, all 83 patterns produce **identical lexicon finding sets**, and that
|
||
includes 13 of 13 of the divergent ones on their own case input. Method, commits and counts:
|
||
[`conformance/manifest.json`](../conformance/manifest.json).
|
||
|
||
So the 13 carry no marker in the corpus and no caveat. Marking them would assert a doubt the
|
||
measurement disproves for these inputs, which is a different defect from the one it would
|
||
appear to prevent.
|
||
|
||
**What still stands is everything above this heading.** The divergence is real, it is
|
||
unresolved, and it will reappear the moment a fixture is written over a witness input.
|
||
`llm-security` has decided **not** to adopt the guard's regex strategy at this point: v0.1.0
|
||
is a behaviour-preservation release on their side too, and swapping strategies mid-vendoring
|
||
would void their own golden gate. Both behaviours stay registered as known divergence per
|
||
pattern. Their three confirmed recall holes are logged as a v8.x task; when it lands they
|
||
will say so, and those rows can close then. Until then the correct description of each is
|
||
*"known divergence, `llm-security` side has an open recall hole, measured 2026-08-09; not
|
||
reachable from any input in the v0.1.0 corpus"* — not *"undecided"*, and not *"resolved"*.
|