feat(conformance): ship the six active-content cases; the id space already existed
The corpus goes 83 -> 89 and scope_covered gains signatures/active-content.json.
The blocker in STATE dissolved under measurement, the same way last session's
13-pattern one did. "An id space for carriers/active-content/secret-egress" was one
question in name only; the three tables have three unrelated problems:
- active-content needed NO id space invented. label_format ("active:{class}") and the
constructs keys were already extracted verbatim from the seed runtime, and their
concatenation IS what it emits - verified by comparing the six keys to the six class
strings at its _flag call sites. What blocked these cases was never naming; it was
spec section 1, fixed in the parent commit.
- carriers has no adoptable id space AND an entry-point dependence underneath it.
- secret-egress is not an id question at all: the two runtimes carry DIFFERENT tables.
18 entries here against the guard's 25, cut at different granularities - this file's
single `GitHub Token` is four ids there, `Private Key PEM Block` three, `Database
connection string` four - with membership diverging both ways. `aws-access-key-id`
is the one clean 1:1, which is why exactly one egress case was ever offered. That
number was a symptom, not modesty.
Both blockers are now recorded under scope_planned.blockers, measured, replacing a
blanket "no runtime has agreed to an id space" that was wrong for both.
Generated from measurement, not written. Payloads were extracted from the seed
runtime's coverage.py by AST - evaluating each _scan_case argument in that module's
own namespace rather than retyping detection data - then run through its public
output gate, the same entry point the 83 lexicon cases used. The fixtures were then
re-read from disk by a separate checker that re-computed every digest, re-scanned the
bytes and applied exact-within-scope independently of the generator: 6 cases, 0
failed checks.
Six built from seven offered. The runtime's matrix drives two payloads at
`active:markdown-image`; measured, their in-scope finding sets are identical, and the
second's only distinguishing signal (entropy:base64-blob) falls outside every table
this repository publishes. Dropped rather than given a discriminated case id, and
named under omitted_payloads so the count reads as a decision.
These six prove LESS than the 83, and the manifest says so: their payloads come from
the only runtime implementing the table, so no second implementation's agreement
could be measured. They pin one runtime's behaviour as a contract a future
implementer can be held to - less than cross-runtime agreement, more than nothing.
llm-security's absence of the table is measured at b0de0ca, not assumed: a tree-wide
search finds no implementation, and `git log -S` over --all returns zero commits,
closing the "it was there once" reading. Absent table is not absent capability -
their entropy scanner reaches markdown-image URLs by another route - and the manifest
says that too.
Provenance and measurement for the active-content half are kept in their own blocks:
a different source structure at a different commit, and one pin must not stand for
two measurements. The guard's HEAD moved twice during the work (3c56d50 -> de09711 ->
398eb74); measurement ran at de09711 and the drift is recorded, including that
active_content.py is NOT identical to the 0bf0729 the data file pins - the change
adds a scan-cap self-safety finding and touches no construct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVouC9nsfrfV5jRSejxbvQ
This commit is contained in:
parent
807c0d45d8
commit
bdcb1f1080
16 changed files with 367 additions and 20 deletions
40
README.md
40
README.md
|
|
@ -59,7 +59,7 @@ and keeping it that small is the point.
|
|||
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
|
||||
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
|
||||
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, and what `exact-within-scope` requires of a runtime. |
|
||||
| [`conformance/`](conformance/) | 83 cases, one per injection-lexicon pattern. One directory per case: `input.txt` in, `expected.json` out. Ground truth. Both seeding runtimes were measured producing the same verdict on all 83 — see [`conformance/manifest.json`](conformance/manifest.json). |
|
||||
| [`conformance/`](conformance/) | 89 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 83 cover the injection lexicon, one per pattern, and both seeding runtimes were measured producing the same verdict on all 83. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
|
||||
| `spec/decode-pipeline.md` | **Planned, not in v0.1.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
|
||||
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
|
||||
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
|
||||
|
|
@ -88,12 +88,24 @@ in its own commit, with the reason written down.
|
|||
There is **no CI in this organisation** and nothing runs that comparison automatically. It
|
||||
runs in each consumer's own test suite, against a pinned tag.
|
||||
|
||||
The v0.1.0 corpus covers `lexicon/injection-lexicon.json`, the one table whose finding ids
|
||||
are ratified by both seeding runtimes. Eleven further cases exist in a seed suite for the
|
||||
carrier, active-content and secret-egress tables and are **not** shipped: naming a finding
|
||||
in those tables would mean minting a cross-runtime id space no runtime has agreed to.
|
||||
`conformance/manifest.json` names them under `scope_planned` so the gap is visible rather
|
||||
than inferred.
|
||||
The corpus covers two tables, and they do not carry equal weight — treating them as one
|
||||
number would misreport both:
|
||||
|
||||
- `lexicon/injection-lexicon.json` — 83 cases. Both seeding runtimes implement it and both
|
||||
ratified its id space.
|
||||
- `signatures/active-content.json` — 6 cases. One runtime implements it. For a runtime that
|
||||
does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
|
||||
runtime declares which commons data files it implements, and a case scoped outside that
|
||||
set was never addressed to it. See [§1.1](spec/conformance-corpus.md) — and note that
|
||||
`not-applicable` says the corpus did not ask, never that the runtime is blind.
|
||||
|
||||
Four cases remain unshipped, for the carrier and secret-egress tables, and neither is
|
||||
blocked on effort. Carriers has no adoptable id space: one runtime labels the same carrier
|
||||
differently depending on pipeline stage, which would make a case's verdict depend on the
|
||||
entry point it was measured through. Secret egress is not an id question at all — the two
|
||||
runtimes carry *different tables*, 18 entries against 25, cut at different granularities.
|
||||
`conformance/manifest.json` records both blockers under `scope_planned.blockers`, measured,
|
||||
so the gap is visible rather than inferred.
|
||||
|
||||
## Non-goals
|
||||
|
||||
|
|
@ -114,10 +126,18 @@ than inferred.
|
|||
|
||||
- **Coverage is the union of what the seed implementations detected**, not of what exists.
|
||||
A class absent from the tables above has not been shown to work anywhere.
|
||||
- **The corpus is narrower than the data.** `conformance/` constrains one of the seven data
|
||||
files. The other six are published, provenance-checked and unfixtured: a runtime can
|
||||
- **The corpus is narrower than the data.** `conformance/` constrains two of the seven data
|
||||
files. The other five are published, provenance-checked and unfixtured: a runtime can
|
||||
pass every case and still read `calibration.json` wrongly. Passing the corpus is evidence
|
||||
about the injection lexicon and about nothing else.
|
||||
about the injection lexicon and about active content, and about nothing else.
|
||||
- **A pass count is unreadable without the declared table set.** A runtime implementing one
|
||||
table and a runtime implementing four can print the same number. `not-applicable` cases
|
||||
must be reported, not dropped from the denominator — `76/83` and `76 passed, 6
|
||||
not-applicable` describe different runtimes.
|
||||
- **The six active-content cases prove less than the 83.** Their payloads come from the only
|
||||
runtime that implements the table, so no second implementation's agreement could be
|
||||
measured. They pin one runtime's behaviour as a contract a future implementer can be held
|
||||
to; they are not cross-runtime agreement.
|
||||
- **Regex portability is a real risk.** Pattern data is written for a common subset, but
|
||||
engines differ (lookbehind, named groups, Unicode property escapes). A consumer whose
|
||||
engine rejects a pattern must report it rather than silently skip it — a skipped pattern
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue