feat(conformance): ship the six active-content cases; the id space already existed
The corpus goes 83 -> 89 and scope_covered gains signatures/active-content.json.
The blocker in STATE dissolved under measurement, the same way last session's
13-pattern one did. "An id space for carriers/active-content/secret-egress" was one
question in name only; the three tables have three unrelated problems:
- active-content needed NO id space invented. label_format ("active:{class}") and the
constructs keys were already extracted verbatim from the seed runtime, and their
concatenation IS what it emits - verified by comparing the six keys to the six class
strings at its _flag call sites. What blocked these cases was never naming; it was
spec section 1, fixed in the parent commit.
- carriers has no adoptable id space AND an entry-point dependence underneath it.
- secret-egress is not an id question at all: the two runtimes carry DIFFERENT tables.
18 entries here against the guard's 25, cut at different granularities - this file's
single `GitHub Token` is four ids there, `Private Key PEM Block` three, `Database
connection string` four - with membership diverging both ways. `aws-access-key-id`
is the one clean 1:1, which is why exactly one egress case was ever offered. That
number was a symptom, not modesty.
Both blockers are now recorded under scope_planned.blockers, measured, replacing a
blanket "no runtime has agreed to an id space" that was wrong for both.
Generated from measurement, not written. Payloads were extracted from the seed
runtime's coverage.py by AST - evaluating each _scan_case argument in that module's
own namespace rather than retyping detection data - then run through its public
output gate, the same entry point the 83 lexicon cases used. The fixtures were then
re-read from disk by a separate checker that re-computed every digest, re-scanned the
bytes and applied exact-within-scope independently of the generator: 6 cases, 0
failed checks.
Six built from seven offered. The runtime's matrix drives two payloads at
`active:markdown-image`; measured, their in-scope finding sets are identical, and the
second's only distinguishing signal (entropy:base64-blob) falls outside every table
this repository publishes. Dropped rather than given a discriminated case id, and
named under omitted_payloads so the count reads as a decision.
These six prove LESS than the 83, and the manifest says so: their payloads come from
the only runtime implementing the table, so no second implementation's agreement
could be measured. They pin one runtime's behaviour as a contract a future
implementer can be held to - less than cross-runtime agreement, more than nothing.
llm-security's absence of the table is measured at b0de0ca, not assumed: a tree-wide
search finds no implementation, and `git log -S` over --all returns zero commits,
closing the "it was there once" reading. Absent table is not absent capability -
their entropy scanner reaches markdown-image URLs by another route - and the manifest
says that too.
Provenance and measurement for the active-content half are kept in their own blocks:
a different source structure at a different commit, and one pin must not stand for
two measurements. The guard's HEAD moved twice during the work (3c56d50 -> de09711 ->
398eb74); measurement ran at de09711 and the drift is recorded, including that
active_content.py is NOT identical to the 0bf0729 the data file pins - the change
adds a scan-cap self-safety finding and touches no construct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVouC9nsfrfV5jRSejxbvQ
This commit is contained in:
parent
807c0d45d8
commit
bdcb1f1080
16 changed files with 367 additions and 20 deletions
86
CHANGELOG.md
86
CHANGELOG.md
|
|
@ -41,11 +41,97 @@ case ids, disposition semantics). Each JSON file additionally carries its own
|
|||
incorrect half past the check — which is precisely the defect
|
||||
`severity_provenance.not_from` was written to warn about, one level up.
|
||||
|
||||
### Added
|
||||
|
||||
- **`not-applicable`, a third conformance verdict** (`spec/conformance-corpus.md` §1.1). A
|
||||
runtime now declares the set of commons data files it implements; a case whose `scope`
|
||||
names a file outside that set is `not-applicable` rather than failed. §1 alone would have
|
||||
reported an architectural difference as a defect — one seeding runtime has no
|
||||
active-content table and never will, and 7 permanent failures say nothing a reader can use.
|
||||
The verdict is fenced so it cannot become an exit: it attaches to a **table**, never to a
|
||||
case (per-case opt-out is the silent skip §1 forbids), a declared set MUST NOT be narrowed
|
||||
to convert failures into `not-applicable`, and such cases MUST still be enumerated rather
|
||||
than dropped from the denominator. §8 now states the consequence: a pass count is
|
||||
unreadable without the declared set beside it.
|
||||
|
||||
- **Six active-content conformance cases** — `active__markdown-image`, `active__markdown-link`,
|
||||
`active__reference-link`, `active__autolink`, `active__raw-html`, `active__data-uri`. The
|
||||
corpus goes 83 → 89, and `scope_covered` gains `signatures/active-content.json`.
|
||||
|
||||
Generated from measurement, not written: payload strings were extracted from the seed
|
||||
runtime's `coverage.py` by AST — evaluating each `_scan_case` argument in that module's own
|
||||
namespace rather than retyping detection data — then run through its public output gate.
|
||||
The fixtures were then re-read from disk by a separate checker that re-computed every
|
||||
digest, re-scanned the bytes and applied `exact-within-scope` independently of the
|
||||
generator, because a generator agreeing with itself proves nothing: 6 cases, 0 failed
|
||||
checks.
|
||||
|
||||
**These six prove less than the 83, and the manifest says so.** Their payloads come from
|
||||
the only runtime implementing the table, so no second implementation's agreement could be
|
||||
measured. They pin one runtime's behaviour as a contract a future implementer can be held
|
||||
to — which is less than cross-runtime agreement and more than nothing.
|
||||
|
||||
- `signatures/active-content.json` **0.1.0 → 0.2.0** — a `pattern_id_space` block. Unlike the
|
||||
lexicon's, nothing was constructed: `label_format` and the `constructs` keys were already
|
||||
extracted verbatim, and their concatenation *is* what the seed runtime emits. The block
|
||||
states an id space the file already had implicitly, and records that it is ratified by
|
||||
**one** runtime rather than two.
|
||||
|
||||
### Changed
|
||||
|
||||
- `lexicon/injection-lexicon.json` **0.5.0 → 0.5.1** — provenance metadata only; no pattern,
|
||||
id, alias, family or severity value changes.
|
||||
|
||||
- `conformance/manifest.json` **0.1.1 → 0.2.0** — the six cases, `scope_covered`,
|
||||
`count_by_scope`, separate provenance and measurement blocks for the active-content half
|
||||
(a different source structure at a different commit; one pin must not stand for two
|
||||
measurements), and `scope_planned.blockers`.
|
||||
|
||||
- **`spec/conformance-corpus.md` §4 no longer claims scoping "asks a question both can
|
||||
answer."** That held only while every case was scoped to the one table both runtimes
|
||||
implement, and stopped being true the moment a case was scoped to a single-runtime table.
|
||||
Scope narrows *what* is compared; it does not make every runtime a valid addressee. The
|
||||
superseded sentence is named in place rather than edited away.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **§4 now states that "belongs to a data file" means published there, never "shares its
|
||||
prefix."** The distinction has a live witness: the seed runtime emits `active:oversize-input`,
|
||||
a self-safety flag about its own scan cap, which carries the `active:` prefix but is no
|
||||
construct in `signatures/active-content.json`. A prefix-matching runtime would pull it into
|
||||
the comparison and fail a case over a finding the corpus makes no claim about. Recorded in
|
||||
that file under `pattern_id_space.not_every_active_label` as well.
|
||||
|
||||
- **§6 now states the derivation's cost.** `case_id` derives from `pattern_id` alone, so a
|
||||
single-finding scope holds at most one case per pattern id — there is nowhere in the name
|
||||
for a second. The seed runtime's matrix drives *two* payloads at `active:markdown-image`;
|
||||
measured, their in-scope finding sets are identical, and the second's only distinguishing
|
||||
signal (`entropy:base64-blob`) falls outside every table this repository publishes. It was
|
||||
dropped rather than given a discriminated id, which would have broken the reverse
|
||||
transform, and it is named in `conformance/manifest.json` under `omitted_payloads` so that
|
||||
6 built from 7 offered reads as a decision rather than a miscount.
|
||||
|
||||
### Measured, not shipped
|
||||
|
||||
- **The remaining four cases are blocked on two distinct unresolved questions**, now recorded
|
||||
under `scope_planned.blockers` instead of the earlier blanket "no runtime has agreed to an
|
||||
id space". That framing was wrong for both:
|
||||
|
||||
- **Carriers (3).** No adoptable id space, and a second problem underneath. The guard emits
|
||||
two *stage-coupled* labels for one carrier — `sanitize:zero-width` on input,
|
||||
`output:zero-width-present` on output, same split for bidi and unicode-tag — while
|
||||
llm-security emits prose titles. A commons id must be invented stage-neutral, which no
|
||||
other id space here required. And since `exact-within-scope` compares a finding *set*, an
|
||||
id aliasing both labels makes the verdict depend on which entry point the runtime was
|
||||
measured through — an entry-point dependence the lexicon cases do not have.
|
||||
- **Secret egress (1).** Not an id-naming question at all. The two runtimes carry
|
||||
**different tables**: 18 entries here against the guard's 25, cut at different
|
||||
granularities (this file's single `GitHub Token` is four ids there, `Private Key PEM
|
||||
Block` three, `Database connection string` four), with membership diverging both ways.
|
||||
`aws-access-key-id` is the one clean 1:1 — which is why exactly one egress case was ever
|
||||
offered. That number was a symptom, not modesty. A shared id space presupposes a table
|
||||
reconciliation nobody has done.
|
||||
|
||||
## [0.1.0] — 2026-08-10
|
||||
|
||||
Initial extraction. Runtime-neutral detection data, the finding contract, and a conformance
|
||||
|
|
|
|||
40
README.md
40
README.md
|
|
@ -59,7 +59,7 @@ and keeping it that small is the point.
|
|||
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
|
||||
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
|
||||
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, and what `exact-within-scope` requires of a runtime. |
|
||||
| [`conformance/`](conformance/) | 83 cases, one per injection-lexicon pattern. One directory per case: `input.txt` in, `expected.json` out. Ground truth. Both seeding runtimes were measured producing the same verdict on all 83 — see [`conformance/manifest.json`](conformance/manifest.json). |
|
||||
| [`conformance/`](conformance/) | 89 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 83 cover the injection lexicon, one per pattern, and both seeding runtimes were measured producing the same verdict on all 83. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
|
||||
| `spec/decode-pipeline.md` | **Planned, not in v0.1.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
|
||||
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
|
||||
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
|
||||
|
|
@ -88,12 +88,24 @@ in its own commit, with the reason written down.
|
|||
There is **no CI in this organisation** and nothing runs that comparison automatically. It
|
||||
runs in each consumer's own test suite, against a pinned tag.
|
||||
|
||||
The v0.1.0 corpus covers `lexicon/injection-lexicon.json`, the one table whose finding ids
|
||||
are ratified by both seeding runtimes. Eleven further cases exist in a seed suite for the
|
||||
carrier, active-content and secret-egress tables and are **not** shipped: naming a finding
|
||||
in those tables would mean minting a cross-runtime id space no runtime has agreed to.
|
||||
`conformance/manifest.json` names them under `scope_planned` so the gap is visible rather
|
||||
than inferred.
|
||||
The corpus covers two tables, and they do not carry equal weight — treating them as one
|
||||
number would misreport both:
|
||||
|
||||
- `lexicon/injection-lexicon.json` — 83 cases. Both seeding runtimes implement it and both
|
||||
ratified its id space.
|
||||
- `signatures/active-content.json` — 6 cases. One runtime implements it. For a runtime that
|
||||
does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
|
||||
runtime declares which commons data files it implements, and a case scoped outside that
|
||||
set was never addressed to it. See [§1.1](spec/conformance-corpus.md) — and note that
|
||||
`not-applicable` says the corpus did not ask, never that the runtime is blind.
|
||||
|
||||
Four cases remain unshipped, for the carrier and secret-egress tables, and neither is
|
||||
blocked on effort. Carriers has no adoptable id space: one runtime labels the same carrier
|
||||
differently depending on pipeline stage, which would make a case's verdict depend on the
|
||||
entry point it was measured through. Secret egress is not an id question at all — the two
|
||||
runtimes carry *different tables*, 18 entries against 25, cut at different granularities.
|
||||
`conformance/manifest.json` records both blockers under `scope_planned.blockers`, measured,
|
||||
so the gap is visible rather than inferred.
|
||||
|
||||
## Non-goals
|
||||
|
||||
|
|
@ -114,10 +126,18 @@ than inferred.
|
|||
|
||||
- **Coverage is the union of what the seed implementations detected**, not of what exists.
|
||||
A class absent from the tables above has not been shown to work anywhere.
|
||||
- **The corpus is narrower than the data.** `conformance/` constrains one of the seven data
|
||||
files. The other six are published, provenance-checked and unfixtured: a runtime can
|
||||
- **The corpus is narrower than the data.** `conformance/` constrains two of the seven data
|
||||
files. The other five are published, provenance-checked and unfixtured: a runtime can
|
||||
pass every case and still read `calibration.json` wrongly. Passing the corpus is evidence
|
||||
about the injection lexicon and about nothing else.
|
||||
about the injection lexicon and about active content, and about nothing else.
|
||||
- **A pass count is unreadable without the declared table set.** A runtime implementing one
|
||||
table and a runtime implementing four can print the same number. `not-applicable` cases
|
||||
must be reported, not dropped from the denominator — `76/83` and `76 passed, 6
|
||||
not-applicable` describe different runtimes.
|
||||
- **The six active-content cases prove less than the 83.** Their payloads come from the only
|
||||
runtime that implements the table, so no second implementation's agreement could be
|
||||
measured. They pin one runtime's behaviour as a contract a future implementer can be held
|
||||
to; they are not cross-runtime agreement.
|
||||
- **Regex portability is a real risk.** Pattern data is written for a common subset, but
|
||||
engines differ (lookbehind, named groups, Unicode property escapes). A consumer whose
|
||||
engine rejects a pattern must report it rather than silently skip it — a skipped pattern
|
||||
|
|
|
|||
19
conformance/active__autolink/expected.json
Normal file
19
conformance/active__autolink/expected.json
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
{
|
||||
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
|
||||
"case_id": "active__autolink",
|
||||
"input": {
|
||||
"file": "input.txt",
|
||||
"bytes": 44,
|
||||
"sha256": "effb96b0d00c49dc20622dc4c3c76b969486383bda554ed4c166db1ad506fae5"
|
||||
},
|
||||
"scope": [
|
||||
"signatures/active-content.json"
|
||||
],
|
||||
"match": "exact-within-scope",
|
||||
"findings": [
|
||||
{
|
||||
"pattern_id": "active:autolink"
|
||||
}
|
||||
],
|
||||
"observed_out_of_scope": {}
|
||||
}
|
||||
1
conformance/active__autolink/input.txt
Normal file
1
conformance/active__autolink/input.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||
contact <https://evil.test/collect?d=stolen>
|
||||
23
conformance/active__data-uri/expected.json
Normal file
23
conformance/active__data-uri/expected.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
|
||||
"case_id": "active__data-uri",
|
||||
"input": {
|
||||
"file": "input.txt",
|
||||
"bytes": 42,
|
||||
"sha256": "e19699e84d06921aae81097ad4976867ea2fd1d19e8c75a37689e7eb89295440"
|
||||
},
|
||||
"scope": [
|
||||
"signatures/active-content.json"
|
||||
],
|
||||
"match": "exact-within-scope",
|
||||
"findings": [
|
||||
{
|
||||
"pattern_id": "active:data-uri"
|
||||
}
|
||||
],
|
||||
"observed_out_of_scope": {
|
||||
"llm_ingestion_guard": [
|
||||
"data-uri:executable"
|
||||
]
|
||||
}
|
||||
}
|
||||
1
conformance/active__data-uri/input.txt
Normal file
1
conformance/active__data-uri/input.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||
payload data:text/html;base64,PHN2Zz4= end
|
||||
19
conformance/active__markdown-image/expected.json
Normal file
19
conformance/active__markdown-image/expected.json
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
{
|
||||
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
|
||||
"case_id": "active__markdown-image",
|
||||
"input": {
|
||||
"file": "input.txt",
|
||||
"bytes": 42,
|
||||
"sha256": "0c2e81798f37b962c1b75bb3b3d0862d29f2d7728718ebe1a5e314e93b603180"
|
||||
},
|
||||
"scope": [
|
||||
"signatures/active-content.json"
|
||||
],
|
||||
"match": "exact-within-scope",
|
||||
"findings": [
|
||||
{
|
||||
"pattern_id": "active:markdown-image"
|
||||
}
|
||||
],
|
||||
"observed_out_of_scope": {}
|
||||
}
|
||||
1
conformance/active__markdown-image/input.txt
Normal file
1
conformance/active__markdown-image/input.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||

|
||||
19
conformance/active__markdown-link/expected.json
Normal file
19
conformance/active__markdown-link/expected.json
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
{
|
||||
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
|
||||
"case_id": "active__markdown-link",
|
||||
"input": {
|
||||
"file": "input.txt",
|
||||
"bytes": 46,
|
||||
"sha256": "46fa07ed3032e9eb3f58935146e6e154e3284876e75347795bb2d2cf61068efe"
|
||||
},
|
||||
"scope": [
|
||||
"signatures/active-content.json"
|
||||
],
|
||||
"match": "exact-within-scope",
|
||||
"findings": [
|
||||
{
|
||||
"pattern_id": "active:markdown-link"
|
||||
}
|
||||
],
|
||||
"observed_out_of_scope": {}
|
||||
}
|
||||
1
conformance/active__markdown-link/input.txt
Normal file
1
conformance/active__markdown-link/input.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||
see [here](https://evil.test/collect?d=stolen)
|
||||
23
conformance/active__raw-html/expected.json
Normal file
23
conformance/active__raw-html/expected.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
|
||||
"case_id": "active__raw-html",
|
||||
"input": {
|
||||
"file": "input.txt",
|
||||
"bytes": 24,
|
||||
"sha256": "569e51d8bb3248bc3ececd96d95362d177fd71c2aa0eaefc4345b25fe6c6cec3"
|
||||
},
|
||||
"scope": [
|
||||
"signatures/active-content.json"
|
||||
],
|
||||
"match": "exact-within-scope",
|
||||
"findings": [
|
||||
{
|
||||
"pattern_id": "active:raw-html"
|
||||
}
|
||||
],
|
||||
"observed_out_of_scope": {
|
||||
"llm_ingestion_guard": [
|
||||
"hybrid-xss:script-tag"
|
||||
]
|
||||
}
|
||||
}
|
||||
1
conformance/active__raw-html/input.txt
Normal file
1
conformance/active__raw-html/input.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||
<script>steal()</script>
|
||||
19
conformance/active__reference-link/expected.json
Normal file
19
conformance/active__reference-link/expected.json
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
{
|
||||
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
|
||||
"case_id": "active__reference-link",
|
||||
"input": {
|
||||
"file": "input.txt",
|
||||
"bytes": 41,
|
||||
"sha256": "2f80e304dab5436a078642d2465f29772cef3a201fd6ff1f72b6bff0005d1e10"
|
||||
},
|
||||
"scope": [
|
||||
"signatures/active-content.json"
|
||||
],
|
||||
"match": "exact-within-scope",
|
||||
"findings": [
|
||||
{
|
||||
"pattern_id": "active:reference-link"
|
||||
}
|
||||
],
|
||||
"observed_out_of_scope": {}
|
||||
}
|
||||
1
conformance/active__reference-link/input.txt
Normal file
1
conformance/active__reference-link/input.txt
Normal file
|
|
@ -0,0 +1 @@
|
|||
[ref]: https://evil.test/collect?d=stolen
|
||||
|
|
@ -1,25 +1,41 @@
|
|||
{
|
||||
"version": "0.1.1",
|
||||
"version": "0.2.0",
|
||||
"id": "llm-security-commons/conformance",
|
||||
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
|
||||
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
|
||||
"case_id_derivation": {
|
||||
"rule": "case_id = pattern_id with ':' replaced by '__'",
|
||||
"reverse": "pattern_id = case_id with '__' replaced by ':'",
|
||||
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 83.",
|
||||
"stability": "A case id is a stable identifier. Changing one is a BREAKING change."
|
||||
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 89.",
|
||||
"stability": "A case id is a stable identifier. Changing one is a BREAKING change.",
|
||||
"one_case_per_pattern_id": "The derivation takes a pattern id and nothing else, so a single-finding scope holds at most one case per pattern id — there is nowhere in the name to put a second. See spec/conformance-corpus.md section 6, and `omitted_payloads` below for the one payload this actually cost."
|
||||
},
|
||||
"match_semantics": "exact-within-scope",
|
||||
"scope_covered": [
|
||||
"lexicon/injection-lexicon.json"
|
||||
"lexicon/injection-lexicon.json",
|
||||
"signatures/active-content.json"
|
||||
],
|
||||
"scope_covered_note": "The two entries do NOT have equal standing, and averaging them would misreport both. `lexicon/injection-lexicon.json` is implemented by both seeding runtimes and its id space is ratified by both. `signatures/active-content.json` is implemented by one; its cases are `not-applicable` for the other under spec/conformance-corpus.md section 1.1, not failures. See that file's `pattern_id_space.single_runtime`.",
|
||||
"scope_planned": {
|
||||
"$comment": "Named rather than faked. The guard's coverage matrix holds 11 further convertible cases (3 carrier, 7 active-content, 1 secret-egress), but the lexicon is the only table whose finding id space is ratified by both seeding runtimes. Fixtures for the others would mint a cross-runtime contract this repository has no agreement for, so they wait for one.",
|
||||
"$comment": "Named rather than faked. Four convertible cases remain in the guard's coverage matrix (3 carrier, 1 secret-egress), and neither table is blocked on effort — each is blocked on a distinct unresolved question, measured 2026-08-10 and recorded below rather than left as 'no agreement yet'.",
|
||||
"codepoints/carriers.json": 3,
|
||||
"signatures/active-content.json": 7,
|
||||
"signatures/secret-egress.json": 1
|
||||
"signatures/secret-egress.json": 1,
|
||||
"blockers": {
|
||||
"codepoints/carriers.json": "No adoptable id space, and a second problem underneath it. The guard emits TWO stage-coupled labels for the same carrier depending on pipeline position — `sanitize:zero-width` (input) versus `output:zero-width-present` (output), and the same split for bidi and unicode-tag; llm-security emits prose titles (unicode-scanner.mjs:191,236). A commons id would therefore have to be invented stage-neutral, which no other id space here required. And because `exact-within-scope` compares a finding SET, a commons id aliasing both guard labels would make the verdict depend on which entry point the runtime was measured through — an entry-point dependence the lexicon cases do not have, since this manifest pins entry point as a measurement fact rather than as contract.",
|
||||
"signatures/secret-egress.json": "Not an id-naming question at all. The two runtimes carry DIFFERENT TABLES, not two namings of one: this file holds 18 entries from llm-security, the guard's `_SECRET_PATTERNS` (output.py) holds 25 at different cut points — this file's single `GitHub Token` is four ids there, `Private Key PEM Block` is three, `Database connection string` is four — and membership diverges both ways (the guard has `gcp-service-account-json` and `openai-api-key-legacy`, which are absent here; this file has `Slack/Discord Webhook URL` and `Azure AI Services Key`, which are absent there). `aws-access-key-id` is the one clean one-to-one, which is why exactly one egress case was ever offered. A shared id space presupposes a table reconciliation that has not happened."
|
||||
}
|
||||
},
|
||||
"omitted_payloads": [
|
||||
{
|
||||
"source": "llm-ingestion-pipeline-security src/llm_ingestion_guard/coverage.py:484",
|
||||
"described_as": "opaque (base64) path segment",
|
||||
"expected_label": "active:markdown-image",
|
||||
"reason": "Its in-scope finding set is `[active:markdown-image]` — identical, measured, to the case built from coverage.py:476. The only thing that distinguishes it is `entropy:base64-blob`, and this repository publishes no entropy table, so the difference falls outside every declared scope. A second case could not have failed in any way the first does not, and the case-id derivation has no room for it (see `case_id_derivation.one_case_per_pattern_id`).",
|
||||
"$comment": "Recorded so that 6 built from 7 offered reads as a decision rather than as a miscount."
|
||||
}
|
||||
],
|
||||
"payload_provenance": {
|
||||
"$scope": "The 83 lexicon cases. The 6 active-content cases have their own provenance in `active_content_provenance` below — they come from a different structure in the same file, at a different commit, and folding them in here would let one pin stand for two measurements.",
|
||||
"source_repo": "llm-ingestion-pipeline-security",
|
||||
"source_file": "src/llm_ingestion_guard/coverage.py",
|
||||
"source_export": "_LEX_PAYLOADS",
|
||||
|
|
@ -30,6 +46,7 @@
|
|||
"stability_note": "llm-ingestion-pipeline-security states (coord message 2026-08-10T12:42:31Z) that _LEX_PAYLOADS and the 83 pattern ids are an internal surface on their side, with no README/CHANGELOG/docs statement promising id or payload stability - their own test suite enforces id coverage as their gate, not as a promise to this repository. Their stated position: if a payload changes upstream, this manifest's pin diverges and should be re-pinned; divergence is a re-pin signal, not a breach of a contract they never granted."
|
||||
},
|
||||
"measurement": {
|
||||
"$scope": "The 83 lexicon cases. The 6 active-content cases are measured in `active_content_measurement`.",
|
||||
"date": "2026-08-10",
|
||||
"method": "Each payload was run through both runtimes' PUBLIC entry point — not through a rebuilt regex table — and the resulting finding labels were mapped to commons pattern ids through the lexicon's own aliases block. Comparing at the entry point is deliberate: a table-level comparison produces a number that describes neither runtime.",
|
||||
"runtimes": [
|
||||
|
|
@ -66,7 +83,55 @@
|
|||
"see": "docs/lexicon-port-divergence.md"
|
||||
}
|
||||
},
|
||||
"count": 83,
|
||||
"active_content_provenance": {
|
||||
"source_repo": "llm-ingestion-pipeline-security",
|
||||
"source_file": "src/llm_ingestion_guard/coverage.py",
|
||||
"source_structure": "the `active` group of `_build_cases()` — seven `_scan_case(...)` call sites, coverage.py:476-488",
|
||||
"source_commit": "de09711",
|
||||
"source_version": "0.4.0",
|
||||
"extraction_method": "The payload strings were located by parsing coverage.py's AST for `_scan_case` calls whose first argument is `\"active\"`, and each payload expression was evaluated against that module's own namespace (which supplies the base64 constant one of them embeds). Retyping them here would have been the from-memory failure this repository forbids for detection data.",
|
||||
"$comment": "A different structure and a different commit from `payload_provenance` above, which covers the 83 lexicon cases. Six of the seven extracted payloads became cases; the seventh is in `omitted_payloads` with its measured reason.",
|
||||
"asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing."
|
||||
},
|
||||
"active_content_measurement": {
|
||||
"date": "2026-08-10",
|
||||
"method": "Each payload was run through the guard's PUBLIC output gate — the same entry point the 83 lexicon cases were measured through, so the two halves of the corpus are comparable. Findings were split into in-scope and residue by MEMBERSHIP in signatures/active-content.json's published `pattern_id_space.ids`, not by the `active:` prefix (see spec/conformance-corpus.md section 4).",
|
||||
"verification": "The generated fixtures were then re-read from disk by a separate checker that re-computed each digest, re-scanned the bytes, and applied `exact-within-scope` independently of the generator — a generator agreeing with itself proves nothing. 6 cases, 0 failed checks.",
|
||||
"runtimes": [
|
||||
{
|
||||
"name": "llm_ingestion_guard",
|
||||
"repo": "llm-ingestion-pipeline-security",
|
||||
"commit": "de09711",
|
||||
"version": "0.4.0",
|
||||
"entry_point": "scan_output(text, source=Source.OUTPUT)",
|
||||
"engine": "CPython 3.14.0",
|
||||
"measurement_limit": "Findings outside signatures/active-content.json that this gate raised are recorded per case in observed_out_of_scope as informative evidence, never as expectation. Two of the six carry such residue, and in both it is a lexicon id."
|
||||
},
|
||||
{
|
||||
"name": "llm_security",
|
||||
"repo": "ssh://git@git.fromaitochitta.com/open/llm-security.git",
|
||||
"commit": "b0de0ca6d86ce697f39669d177c2c2654c280128",
|
||||
"status": "not-applicable",
|
||||
"status_basis": "This runtime does not implement signatures/active-content.json, so under spec/conformance-corpus.md section 1.1 these six cases are `not-applicable` there — neither passes nor failures. It was NOT run against them; a measurement whose verdict the spec defines as inapplicable would produce a number that means nothing.",
|
||||
"absence_evidence": "Measured at b0de0ca, not assumed. A case-insensitive tree-wide search for an active-content detector, class table or emitted `active:` label returns no implementation — only prose in `knowledge/` and an entropy-scanner test mentioning the markdown-image construct. `git log -S 'active_content'` and `git log -S 'active:markdown'` over `--all` return zero commits, closing the 'it was there once' reading.",
|
||||
"not_a_coverage_claim": "Absent table is not absent capability: this runtime's entropy scanner reaches markdown-image URLs by a different route (host- and query-aware). What is measured absent is the table, which is all `not-applicable` asserts."
|
||||
}
|
||||
],
|
||||
"results": {
|
||||
"payloads_extracted": 7,
|
||||
"cases_built": 6,
|
||||
"payloads_omitted": 1,
|
||||
"cases_verified_independently": 6,
|
||||
"failed_checks": 0,
|
||||
"cases_with_out_of_scope_residue_in_the_guard": 2
|
||||
},
|
||||
"guard_drift_note": "The guard's HEAD moved twice during this measurement (3c56d50 -> de09711 -> 398eb74). Measurement ran at de09711. Checked rather than hoped: 398eb74 touches only docs/PLAN-v1.md and tests/test_disposition.py, so no measured file changed under the measurement. active_content.py DID change between 0bf0729 — the commit signatures/active-content.json pins — and de09711; the change adds a scan-cap self-safety finding and touches no construct. See that file's `pattern_id_space.verified.at_commit_note`."
|
||||
},
|
||||
"count": 89,
|
||||
"count_by_scope": {
|
||||
"lexicon/injection-lexicon.json": 83,
|
||||
"signatures/active-content.json": 6
|
||||
},
|
||||
"cases": [
|
||||
"override__ignore-previous",
|
||||
"override__forget-instructions",
|
||||
|
|
@ -150,6 +215,12 @@
|
|||
"hybrid-xss__script-tag",
|
||||
"hybrid-xss__javascript-uri",
|
||||
"hybrid-xss__event-handler",
|
||||
"hybrid-xss__iframe-src"
|
||||
"hybrid-xss__iframe-src",
|
||||
"active__markdown-image",
|
||||
"active__markdown-link",
|
||||
"active__reference-link",
|
||||
"active__autolink",
|
||||
"active__raw-html",
|
||||
"active__data-uri"
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,11 +1,53 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"id": "active-content",
|
||||
"description": "Active content in persisted text: constructs that fetch, render or execute on their own when a renderer touches them. The EchoLeak class (CVE-2025-32711). Report-only data: no construct here is defined as something to remove.",
|
||||
"owasp": "LLM05",
|
||||
"detector": "active_content",
|
||||
"label_format": "active:{class}",
|
||||
|
||||
"pattern_id_space": {
|
||||
"$comment": "Stable, commons-owned identifier for each construct, so that a conformance fixture can name an active-content finding. Unlike the injection lexicon's id space, nothing was constructed here: `label_format` and the `constructs` keys were already extracted verbatim from the seed runtime, and their concatenation IS what that runtime emits. This block states an id space that the file already had implicitly rather than introducing one.",
|
||||
"field": "\"active:\" + the key of the entry in `constructs`",
|
||||
"ids": [
|
||||
"active:autolink",
|
||||
"active:data-uri",
|
||||
"active:markdown-image",
|
||||
"active:markdown-link",
|
||||
"active:raw-html",
|
||||
"active:reference-link"
|
||||
],
|
||||
"verified": {
|
||||
"method": "The six `constructs` keys were compared to the six class strings the seed runtime passes to its finding emitter (`_flag(cls, …)` call sites, active_content.py:337-369, which build `label=f\"active:{cls}\"` at line 309). Set-equal, exact strings, no normalisation applied.",
|
||||
"date": "2026-08-10",
|
||||
"at_commit": "de09711",
|
||||
"at_commit_note": "Named because that is where the comparison was actually run, which is NOT the commit this file's provenance pins (0bf0729). active_content.py did change over that range: `git diff 0bf0729..de09711` adds 23 lines. Measured, not assumed - the change is a self-safety length cap (a new `max_scan_chars` parameter, and an `active:oversize-input` finding when the input exceeds it). It adds no construct, removes none, and leaves every `_flag` call site and every construct pattern untouched, so the six-way comparison above holds at both commits. The drift is recorded rather than smoothed over: a reader checking this file against 0bf0729 will see a different active_content.py and is entitled to know why that does not invalidate the check."
|
||||
},
|
||||
"not_every_active_label": {
|
||||
"$comment": "The seed runtime emits at least one `active:`-prefixed label that is NOT in this id space, and a consumer that matches by prefix rather than by membership will mis-scope it.",
|
||||
"known_example": "active:oversize-input",
|
||||
"what_it_is": "A self-safety flag (OWASP LLM10) raised when the input exceeds the runtime's scan cap, announcing that the tail went unread. It reports the detector's own coverage, not a construct found in the text, which is why it is not a member of this table.",
|
||||
"rule": "Membership in this id space is by presence in `ids`, never by the `active:` prefix. A finding carrying that prefix but absent from `ids` does not belong to this data file, and is therefore OUT of scope for a conformance case scoped here - see spec/conformance-corpus.md section 4.",
|
||||
"introduced": "Between 0bf0729 and de09711; absent at the commit this file's provenance pins."
|
||||
},
|
||||
"stability": "An id is a stable identifier. Changing one is a BREAKING change for any consumer matching on it, and requires a major bump of this file.",
|
||||
"single_runtime": {
|
||||
"$comment": "This id space is ratified by ONE runtime, not two, and that is a weaker status than the injection lexicon's - stated here rather than left to be discovered by a consumer who assumes the two files carry equal weight.",
|
||||
"implemented_by": [
|
||||
"llm_ingestion_guard"
|
||||
],
|
||||
"not_implemented_by": [
|
||||
"llm_security"
|
||||
],
|
||||
"absence_evidence": "Measured in llm-security at b0de0ca, not assumed: a case-insensitive tree-wide search for an active-content detector, class table or emitted `active:` label returns no implementation - only prose in `knowledge/` and an entropy-scanner test that mentions the markdown-image construct. `git log -S 'active_content'` and `git log -S 'active:markdown'` over `--all` return zero commits, so the table was never present and later removed either.",
|
||||
"not_a_coverage_claim": "That runtime having no active-content TABLE is not the same as it being blind to these constructs. Its entropy scanner reaches markdown-image URLs from a different angle (rule E18, host- and query-aware). What is absent is this table, which is all `not_implemented_by` asserts.",
|
||||
"conformance_consequence": "Under spec/conformance-corpus.md section 1.1, a runtime that does not declare this file reports cases scoped to it as `not-applicable` - neither pass nor error."
|
||||
},
|
||||
"aliases": {
|
||||
"$comment": "No `aliases` object is carried per construct, and the omission is deliberate rather than pending. The lexicon needs aliases because two runtimes name the same pattern differently; here the sole implementing runtime's emitted label IS the id, so an alias would be a copy of the key. A second runtime adopting this table registers its alias then - see the lexicon's aliases block for the shape."
|
||||
}
|
||||
},
|
||||
|
||||
"$comment": "Extracted verbatim, without behaviour change, from llm-ingestion-pipeline-security v0.3.4 (commit 0bf0729, 2026-08-03): src/llm_ingestion_guard/active_content.py (pattern table, pass order, URL-shape analysis, defang) and src/llm_ingestion_guard/calibration.py (per-construct severities and opacity floors). That module documents itself as the canonical home of this table with the Node port sharing its severities, which is why the guard is the seed source here rather than llm-security.",
|
||||
|
||||
"provenance": {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue