open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

chore(llm-security): pull commons subtree to v0.4.3

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-08-13 20:08:20 +02:00
commit 464825f9bf
8 changed files with 844 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

158
scanners/commons/docs/lexicon-port-divergence.md
View file

@ -5,6 +5,11 @@ records a measured disagreement between two ports of one source table, so the de
taken where each table is tested. Under this repository's behaviour-preservation invariant,
a divergence found here is **reported, not fixed**.
That invariant held through the closure recorded below. Two rows this document measured did change
in `lexicon/injection-lexicon.json` `0.8.0` — and not because this document found them wrong. The
finding was routed to the runtime that owns the value, that runtime re-measured it, decided, and
asked commons to carry the result. The measurement travelled; the authority did not.
Produced 2026-08-09. Every number below came from a command; the scripts live in the session
scratchpad rather than in this repository, because executable code here would breach the
charter. They are reproducible from the method column.
@ -166,18 +171,59 @@ The direction matters for what it cannot break: the new form matches a strict su
one, so relative to `0.6.0` it can add matches and cannot add a false negative. The reverse change
would not have been adoptable on the same reasoning.
**What is still open is the one-byte span difference**, and convergence did not touch it. The
guard excludes `<` from its negated class and the other two do not, so:
**The one-byte span difference is CLOSED as of commons `0.8.0`** — all three runtimes now carry
`[^><]*`. What follows is the account of the divergence while it was open, kept because the table
below is the measurement that made the closure decidable, and because a consumer diffing against
`v0.3.0` will find the two changed rows and needs the coordinate.
| Witness | commons `0.7.0` / `llm-security` | guard |
| --- | --- | --- |
| `<script>steal()</script>` | match | match |
| `<script src=x.js>` | match | match |
| `<script>alert(1)` | match | match |
| `<script <x>` | **match** | **no match** |
While it was open, the guard excluded `<` from its negated class and the other two did not. The
three forms turn out to be **totally ordered by what they match** — each a strict superset of the
next:
Measured by compiling all three forms, not reasoned from the character classes. The last row is
the whole residual divergence and neither side has claimed it.
| Witness | commons `0.7.0` / `llm-security` `[^>]` | guard `[^><]`, and commons from `0.8.0` | commons ≤ `0.6.0` |
| --- | --- | --- | --- |
| `<script>steal()</script>` | match | match | match |
| `<script src=x.js>` | match | match | no match |
| `<script>alert(1)` | match | match | no match |
| `<script\n src=x>` | match | match | no match |
| `<script a="<" >x` | **match** | **no match** | no match |
| `<script<div>` | **match** | **no match** | no match |
Measured by compiling all three forms, not reasoned from the character classes. The last two rows
are the whole residual divergence.
The witness set and the ordering observation are `llm-security`'s, from their coord message of
2026-08-11T11:01:59Z; both were **reproduced here independently** before being recorded, including
the strict-superset property in both directions. Their reading of it: `[^><]` refuses to cross a
`<`, and since anything that reaches `[^>]`-and-not-`[^><]` already contains a literal `<script`
opening tag, the narrower class buys an empty false-positive surface in exchange for two dropped
input classes — a recall loss with no precision gain to trade for.
**That argument is recorded, not endorsed, and it did not decide commons' form.** Commons carries
`[^>]` because the lexicon's declared provenance is `llm-security`'s table (see above), not
because their regex won on merit — the same reasoning would have taken commons to `[^><]` had the
source been the guard's. `llm-security` asked to hear the guard's reason for `[^><]` — a ReDoS
bound, or a span-extraction consumer needing the run to stop at `<` — before commons shipped, and
commons shipped first. That order is deliberate rather than an oversight: a reason that beats this
argument is a reason for the *guard's own port*, and commons would still track its declared
source. Put to the guard as an open question 2026-08-11; unanswered at time of writing.
**How it resolved, and why the resolution used the same rule rather than overturning it.** Both
halves of that open question came back the same day. The guard answered that `[^><]` is a
*measured* ReDoS fix on their side (their commit `cff0437`), not an aesthetic choice — so the
"reason that beats this argument" the paragraph above asked for turned out to exist. And commons
then measured its own copy and found the same defect in it: `<script\b[^>]*>` and
`<iframe\b[^>]*src…` are both quadratic in scan length, the second one having been named by nobody
until it was counted. `llm-security` re-measured rather than taking the report, reached `[^><]*`
independently, and asked commons to carry it — because their table is now *built from this file*,
so there is no local literal left for them to change without forking.
The paragraph above said commons "would still track its declared source." That is exactly what
`0.8.0` does: the declared source chose `[^><]*` and commons followed. The rule did not bend to
accommodate a better argument — the argument won *at the source*, which is the only place this
repository lets an argument decide a `pattern` value. The recall cost stated in the two bold rows
of the table is now paid deliberately: measured across all 90 conformance cases, zero lost a match
and zero gained one, so the dropped classes remain real and remain unwitnessed by the corpus.
## Why they diverge: two different ReDoS mitigations of one table
@ -223,6 +269,98 @@ disagreement — not an oversight on either side.
Neither strategy is free, and neither is obviously right. That is the decision the two owning
repositories have to take, and it is not commons' to take for them.
### The cost of commons' strategy, measured here — 2026-08-11
The section above says neither strategy is free and neither is obviously right. That was
written without a number for **this** side's cost. There is one now, and it is not symmetric
with the guard's.
The prompt came from the guard (coord, 2026-08-11): their `[^><]` is not a preference but a
measured ReDoS fix (`cff0437`, 2026-07-31), and they asked commons to measure its own form
rather than take their word. Measured here — Node v25.8.2 and Python 3.14.0, Intel i9-9880H,
patterns read out of `lexicon/injection-lexicon.json` and run over `'<script '` repeated,
the unit that denies the pattern the `>` it needs:
| chars | `<script\b[^>]*>` (commons, Node) | `<script\b[^><]*>` (guard form, Node) |
|---|---|---|
| 16 000 | 25.5 ms | 0.08 ms |
| 32 000 | 101.3 ms | 0.06 ms |
| 64 000 | 413.5 ms | 0.12 ms |
| 128 000 | 1 676.5 ms | 0.24 ms |
| 256 000 | 6 681.6 ms | 0.41 ms |
×4.0 per doubling against ×2 — quadratic against linear, reproduced in Python `re` on the same
inputs (0.021 s → 5.44 s over the same range, guard form 0.0001 s → 0.0015 s). It is
scan-position quadratic, not exponential backtracking: `<script` matches at ~n positions and
`[^>]*` scans to end of input at each one. Nothing here is catastrophic backtracking, and
nothing here needs a nested quantifier to happen.
> **Correction, 2026-08-11 (commons `v0.4.1`) — two of the Python figures above read low, and
> the `iframe-src` figure below read low by ~3×.** The guard flagged it (coord,
> 2026-08-11T19:51:55Z) after measuring the `iframe-src` row themselves and getting roughly 4×
> our published number. They attributed the gap to measurement surface — their composed
> `scan_lexicon()` against our standalone regex. **That is not the explanation.** Re-measured
> here standalone, the same surface the numbers above claim, Python 3.14.0, median-stable across
> repeated runs (the two 256 000-char `iframe-src` runs landed 3 ms apart in 51 s):
>
> | chars | `script-tag` `[^>]*` | `script-tag` `[^><]*` | `iframe-src` `[^>]*` | `iframe-src` `[^><]*` |
> |---|---|---|---|---|
> | 32 000 | 87.4 ms | 0.17 ms | 822.7 ms | 0.25 ms |
> | 100 000 | — | — | 7 859.5 ms | — |
> | 256 000 | 5 222.6 ms | 1.25 ms | 51 477.4 ms | 2.08 ms |
>
> So: the Python `script-tag` figure at 256 000 chars **reproduces** (5.44 s published against
> 5.22 s measured), and the one at 32 000 chars does **not** (0.021 s against 0.087 s). The
> `iframe-src` pair below reproduces at neither point. The error ratios are not constant, so a
> single mis-sized input does not explain it, and the original harness lived in a previous
> session's scratchpad and is gone — **the cause is not diagnosable and is not guessed at here.**
> Our standalone 100 000-char figure (7.86 s) sits close to the guard's composed 8.95 s, which
> is why their surface hypothesis does not survive: the two surfaces differ by much less than
> the error did.
>
> **What does not change:** the shape (quadratic, ×4 per doubling), the direction, and the
> decision. Every corrected figure is *larger*, so the case against the old forms is stronger
> than the one originally published, and the new forms are flat under both engines. The guard
> flagged this because a low number could matter if either side later pins a bound — they were
> right to, and the number they should pin against is the one in this box.
**Two of the 83 patterns have this shape, not one.** Counted over the whole file, 8 patterns
carry a bare `[^>]` and none carries `[^><]`. Six of the 8 are the html-obfuscation family and
are **bounded** (`[^>]{1,256}`) — measured linear (51 → 164 ms across a 8× input growth), so
bounding does work as a mitigation, at the recall cost the section above already records. The
other two are unbounded `[^>]*`:
- `hybrid-xss:script-tag` — the row this file already tracks.
- `hybrid-xss:iframe-src` — **not previously named by anyone**, found here by counting rather
than by being told. Same shape, same result: 119.6 ms → 16 857 ms over 32 000 → 256 000
chars of `'<iframe '`. **Both figures are superseded — see the correction box above:
822.7 ms → 51 477.4 ms on re-measurement, standalone, same unit.**
What this does and does not license:
- It does **not** license changing either pattern here. Both are byte-identical to
`llm-security`'s source, and the behaviour-preservation invariant is not suspended because
the measurement is unwelcome. Commons carries `[^>]` on provenance and would carry `[^><]`
had the source been the guard's — that was true before this measurement and stays true after.
- It does mean the finding is **owed to the owning runtime**, which is `llm-security` for both
patterns. Sent 2026-08-11, with these numbers and the `iframe-src` row they did not have.
**Outcome, same day:** the owning runtime reproduced both rows on its own vendored copy,
measured the exposure at its real entry-point caps (which the numbers above deliberately do not
cover — a pattern cost is not an exposure), judged it unacceptable, chose `[^><]*` over
`[^>]{0,256}` on the grounds that a bound is paddable, and asked commons to carry the change.
Commons `0.8.0` carries it. Note what did **and did not** license that: the measurement above
still licenses nothing on its own, and the two bullets stand as written. What licensed the
change is the *owner's decision*, arriving through the one channel this repository accepts for
moving a `pattern` value. Had they declined, the rows would still read `[^>]*` today and this
document would record a permanent divergence instead.
- The guard's own honest disclosure belongs on the record too: their suite passes under both
forms (717 green), because the DoS row that ought to discriminate uses `'<script>'` as its
repeating unit, which supplies the `>` and never exercises the run. The invariant is
documented there and not test-enforced. A conformance case could hold it — but a fixture that
pins a *timing* property is a different kind of case from anything in this corpus today, and
minting one is not commons' call to take alone.
## Severity: the 8 hybrid patterns
**Resolved 2026-08-09. The two sides never disagreed; only the evidence did.**