open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

fix(divergence): our own iframe number read 3x low, and the reported cause was not the cause

Browse source 
No data file changed and no pattern moved. A published figure was wrong.

llm-ingestion-pipeline-security flagged it (coord, 2026-08-11T19:51:55Z) by
measuring the hybrid-xss:iframe-src row themselves instead of citing ours, and
got roughly 4x our number. They attributed the gap to measurement surface -
their composed scan_lexicon() against our standalone regex - and said no
re-measure was needed.

Re-measured anyway, because the claim was about OUR number. Their diagnosis does
not hold: our standalone 100 000-char figure is 7.86 s against their composed
8.95 s, so the two surfaces differ by far less than the error did. Standalone,
Python 3.14.0, same unit the document claims:

  iframe-src [^>]*    822.7 ms @ 32k     51 477.4 ms @ 256k
  published           119.6 ms @ 32k     16 857    ms @ 256k
  script-tag [^>]*     87.4 ms @ 32k      5 222.6 ms @ 256k
  published            21    ms @ 32k      5 440    ms @ 256k

So the Python script-tag figure at 256k reproduces and the one at 32k does not,
and the iframe-src pair reproduces at neither point. Error ratios are not
constant, so a single mis-sized input does not explain it, and the original
harness lived in a previous session's scratchpad and is gone. Recorded as NOT
DIAGNOSABLE rather than given a plausible cause - a guessed cause would read
like a finding.

Superseded figures are struck in place rather than overwritten. Anyone who cited
the old number needs to be able to discover that they did.

The 0.4.0 decision does not depend on this. Every corrected figure is larger, the
shape is unchanged (quadratic, x4 per doubling), and both [^><]* forms stay flat.
The 0.4.0 CHANGELOG section still quotes the old figure and is left alone: that
section is the record of what was released, not a live claim.

Verified: JSON well-formed, specs normative, charter clean, README vendoring
examples and prose moved to v0.4.1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLEZ4XCSnSrQUFA8SzkQB4
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 22:02:15 +02:00
commit 2d86151e3b
3 changed files with 67 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

8
README.md
View file

@ -24,7 +24,7 @@ As a `git subtree` (recommended: history is preserved and upgrades are a single
```bash
git subtree add --prefix vendor/commons \
https://git.fromaitochitta.com/open/llm-security-commons.git v0.4.0 --squash
https://git.fromaitochitta.com/open/llm-security-commons.git v0.4.1 --squash
# later, to move to a newer tag
git subtree pull --prefix vendor/commons \
@ -34,7 +34,7 @@ git subtree pull --prefix vendor/commons \
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
```bash
git clone --depth 1 --branch v0.4.0 \
git clone --depth 1 --branch v0.4.1 \
https://git.fromaitochitta.com/open/llm-security-commons.git
```
@ -61,14 +61,14 @@ and keeping it that small is the point.
| [`schema/conformance-declaration.schema.json`](schema/conformance-declaration.schema.json) | **Normative.** The shape a runtime publishes alongside a conformance result: which commons tables it implements, the commons commit it measured, and the four verdict counts. Required by the corpus spec §1.1; not validated by anything here, because nothing here runs. |
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, what `exact-within-scope` requires of a runtime, and how a runtime declares its table set so a case scoped outside it reads as `not-applicable` rather than as a failure. |
| [`conformance/`](conformance/) | 90 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 84 cover the injection lexicon — 83 one per pattern, both seeding runtimes measured producing the same verdict on all 83, plus one variant case gating a pattern form against its predecessor. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.4.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.4.1.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions. Most of it is still open, and the two rows that closed in v0.4.0 closed because the runtime that owns the value decided, not because this document found them wrong. |
Every JSON file carries a top-level `version`. Every normative specification carries a
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
part of the contract, but the file does not exist yet — they are not links, and nothing in
v0.4.0 depends on them.
v0.4.1 depends on them.
Each data file records its own provenance and, in `verified`, how strongly it is backed.
`calibration/calibration.json` is currently the one file that says `false`: it was