# Conformance corpus **Status: normative.** This document defines how the files under [`conformance/`](../conformance/) are to be read. Where it disagrees with any informative document in `docs/`, this wins. RFC 2119 language is used deliberately: MUST, MUST NOT, SHOULD, MAY. ## 1. What a case is A case is a directory under `conformance/` holding exactly two files: ``` conformance//input.txt the exact bytes to scan conformance//expected.json the findings a conforming runtime produces ``` A runtime claiming conformance MUST run every case. It MUST NOT skip a case because its engine rejects the pattern behind it — a silently skipped case is an invisible false negative, which is the failure mode this corpus exists to catch. A case that cannot be run MUST be reported as an error, not as a pass. ### 1.1 Declared tables and `not-applicable` The runtimes that share this core do not implement the same set of data files, and some never will: a table can be meaningful in an output gate and meaningless in the surface next to it. §1 as stated would make every such case a permanent failure, which reports an architectural difference as a defect and tells a reader nothing. A runtime claiming conformance therefore MUST declare **the set of commons data files it implements**, and MUST publish that set alongside its conformance result. A case whose `scope` (§4) names a data file outside the declared set MUST be reported as **`not-applicable`** — a third verdict beside pass and fail, and a different thing from the error of §1. §1's error means *this runtime tried and could not*; `not-applicable` means *this question was never addressed to this runtime*. A `not-applicable` case MUST still be enumerated in the result. It MUST NOT be counted as a pass, and MUST NOT be silently dropped from the denominator: `76/83 passed` and `76 passed, 7 not-applicable` describe different runtimes, and only the second one can be checked. Two constraints keep the verdict from becoming an exit: - A runtime MUST NOT withdraw a table from its declared set in order to convert failing cases into `not-applicable` ones. The declaration describes what the runtime implements, not what it would prefer to be asked. Narrowing it is a visible change to a published claim, and it MUST be reported as such rather than made quietly between runs. - `not-applicable` attaches to a **table**, never to a case. A runtime that declares a table MUST run every case scoped to it, including the ones it fails. Per-case opt-out is exactly the silent skip §1 forbids. **The shape of the declaration.** A runtime publishing a declaration MUST use the shape in [`schema/conformance-declaration.schema.json`](../schema/conformance-declaration.schema.json): runtime, the commons commit measured, the declared table set, and the four verdict counts with their total. The requirement is the shape, not a validation step — nothing in this repository runs, so no runtime is gated on it. It exists because a free-form declaration makes `83 passed, 6 not-applicable` unparseable by anyone but its author, and a result only its author can read is not a cross-runtime measurement. That schema also carries the two arithmetic invariants this section implies but cannot state in prose without ambiguity: the counts MUST sum to the total, and the total MUST equal the corpus case count at the commit measured. A runtime SHOULD derive its declared set from the same constant its runner uses to accept or reject a `scope`, rather than maintain a second list beside it, and SHOULD record which of the two it did. The anti-narrowing constraint above is then structural rather than promised: narrowing the declaration is narrowing what the runtime accepts, which is a change to code with tests behind it instead of an edit to a published file nobody re-reads. **What the verdict does not say.** `not-applicable` records that a runtime does not implement a named commons table. It is not a statement that the runtime is blind to the phenomenon behind it — a runtime may well detect the same construct through some other mechanism that this corpus does not constrain, and reading the verdict as a coverage gap would assert something nobody measured. See §8. ## 2. `input.txt` is bytes, not text `input.txt` holds the input **verbatim**, UTF-8 encoded. A runtime MUST read it as bytes and MUST NOT strip, trim, normalise or re-encode anything before scanning. There is **no trailing newline**. A trailing newline would be part of the input, and no case wants one. Two of the current cases carry characters that are invisible on screen — a zero-width space and a Cyrillic homoglyph — so an editor that "cleans up" a fixture can change what it tests without changing how it looks. For that reason every `expected.json` records the input's byte length and SHA-256, and a runtime SHOULD verify both before scanning. A digest mismatch means the fixture has been altered; the result of scanning it means nothing. ## 3. `expected.json` ```json { "case_id": "hybrid-xss__script-tag", "input": { "file": "input.txt", "bytes": 24, "sha256": "…" }, "scope": ["lexicon/injection-lexicon.json"], "match": "exact-within-scope", "findings": [{ "pattern_id": "hybrid-xss:script-tag" }], "observed_out_of_scope": { "llm_ingestion_guard": ["active:raw-html"] } } ``` | Field | Normative? | Meaning | | --- | --- | --- | | `$comment` | no | Provenance prose. Carries no contract. | | `case_id` | yes | Equals the directory name. | | `input` | yes | Integrity of the input file. See §2. | | `scope` | yes | Which commons data files this case constrains. See §4. | | `match` | yes | How `findings` is to be compared. See §4. | | `findings` | yes | The expected findings, named by commons `pattern_id`. See §3.1. | | `observed_out_of_scope` | **no — informative** | What a named runtime was measured emitting outside `scope`. Evidence, never expectation. See §5. | A finding carries a `pattern_id` and nothing else. Severity, OWASP anchor and human label are properties of the pattern, published once in [`lexicon/injection-lexicon.json`](../lexicon/injection-lexicon.json) and looked up there by that id. They MUST NOT be restated in a fixture: a value written in two places is a value that will eventually disagree with itself. ### 3.1 `pattern_id` is not the finding `id` The field is called `pattern_id` and not `id` because this repository already publishes a different `id`, and confusing the two produces a runtime that fails every case for a reason unrelated to detection. | | Identifies | Stable? | | --- | --- | --- | | `pattern_id` — here; `families[].patterns[].id` in the lexicon | the **detection rule** | Yes. A stable cross-runtime contract, ratified by both seeding runtimes. | | `id` — in [`schema/finding.schema.json`](../schema/finding.schema.json) | one **emitted finding** in one run | No. `DS--`, from a process-global counter: not stable across runs, not unique across processes. The schema says so itself. | A runtime therefore MUST NOT compare a serialized finding's `id` to a fixture's `pattern_id`. They answer different questions. **Getting from a runtime's own label to a `pattern_id`.** Nothing requires a runtime to name its patterns the way this repository does, and the two seeding runtimes do not: the same rule is `override: ignore previous instructions` in one and `override:ignore-previous` in the other. The mapping is published, per pattern, in the lexicon's `aliases` object: ```json { "id": "override:ignore-previous", "aliases": { "llm_security": "override: ignore previous instructions", "llm_ingestion_guard": "override:ignore-previous" } } ``` (The lexicon's own field is spelled `id`, because that name is ratified and renaming it would break both consumers. `pattern_id` here refers to exactly that value.) A runtime SHOULD carry that id directly on its own pattern table, which makes the comparison a lookup rather than a translation. A runtime that instead maps through `aliases` MUST register its own alias there — a runtime absent from that object has no published way to be compared, and a mapping maintained privately in a consumer is the drift this repository exists to prevent. Fixture files carry **no individual `version` field**, which is the one place this repository's "every JSON file has a top-level `version`" convention does not apply. The corpus is versioned as a whole in [`conformance/manifest.json`](../conformance/manifest.json). The exception is deliberate and stated here rather than left to be discovered. ## 4. `exact-within-scope` This is the only `match` value defined in this version, and it means: > Restricted to findings whose id belongs to a data file named in `scope`, the runtime's > finding set MUST equal `findings` exactly. Findings outside that scope MUST be ignored > by the comparison — neither required nor forbidden. So a runtime MUST NOT raise a lexicon finding this case does not list, and MUST raise every one it does. Whether the same runtime also flags the input as active content, as a carrier, or as anything else is **not** this case's business. **"Belongs to" means published in that file, never "shares its prefix."** A runtime MUST decide membership by looking the id up in the scoped data file, and MUST NOT infer it from the id's leading segment. The two are not the same set, and the difference is not hypothetical: one seeding runtime emits `active:oversize-input`, a self-safety flag about its own scan cap, which carries the `active:` prefix but is no construct in [`signatures/active-content.json`](../signatures/active-content.json) and is therefore out of scope for a case scoped there. A prefix-matching implementation would pull it into the comparison and fail a case over a finding the corpus makes no claim about. The scoping is what makes the exactness safe. The two seeding runtimes do not implement the same set of tables — one has no active-content table at all — so a whole-report comparison would fail for reasons that have nothing to do with the pattern under test. Scoping the comparison to one published table asks a question about that table alone. Through version 0.1.1 this paragraph ended "…asks a question both can answer", which held only while every case was scoped to the one table both runtimes implement. It stopped being true the moment a case was scoped to `signatures/active-content.json`, which only one of them has. Scope narrows *what* is compared; it does not guarantee that every runtime is a valid addressee of the comparison. That second question is answered by the declared table set in §1.1, and a case scoped outside a runtime's declared set is `not-applicable` there rather than failed. The exactness is what makes the corpus worth running. A corpus that only checks that the expected finding is *present* is passed by a runtime that flags everything. A future case MAY declare a different `scope` or a different `match`. A runtime MUST reject a `match` value it does not implement rather than degrade to a weaker comparison. ## 5. `observed_out_of_scope` is evidence, not expectation This block records what a **named** runtime, at a **named** commit, was measured emitting outside the case's scope. It exists so that a reader can see what was left out of the contract instead of having to trust that nothing was. A runtime MUST NOT be failed for disagreeing with it, and MUST NOT be required to reproduce it. **Absence of a runtime's key means unmeasured, not measured-empty.** The current corpus carries no `llm_security` key anywhere, because that runtime was measured through its injection-scanner entry point, which cannot emit an out-of-scope finding by construction. Reading that absence as "this runtime emits nothing else" would be a claim nobody made. ## 6. Case ids ``` case_id = pattern_id with ":" replaced by "__", optionally followed by "--" and a variant slug of [a-z0-9-] pattern_id = case_id truncated at the first "--" if present, then "__" replaced by ":" ``` `:` is not a legal filename character on Windows, and fork-and-own is a supported use of this repository, so the id space cannot reach the filesystem unchanged. `__` does not occur anywhere in the ratified id space, so the transform is one-to-one — verified collision-free across all cases rather than assumed. `--` does not occur there either: the ratified ids use single hyphens throughout, measured across every id space this repository publishes and every case id in the corpus. That measurement is what keeps the reverse transform **lexical**. A runtime recovers a `pattern_id` by splitting the string, and MUST NOT need a lookup against the published id list to find where the id ends and the variant begins — a reverse transform that has to ask which of two readings is real is a different rule from the one written above, and it fails on the first id space that is vendored without its lookup table. **A `pattern_id` may carry more than one case.** Through corpus version 0.2.0 this section said the opposite: that a case id derives from a pattern id alone, so a single-finding scope holds at most one case per pattern id, with nowhere in the name to put a second. Corpus version 0.3.0 extended the derivation with the optional suffix above, and the corpus has shipped a case under it since (`hybrid-xss__script-tag--src-no-close`, whose own `$comment` carries the rationale for that one). The superseded sentence is named here rather than edited away, because it was the stated ground on which a real payload was dropped — `omitted_payloads` in [`conformance/manifest.json`](../conformance/manifest.json) records that ground as withdrawn and a second, independent ground as still standing. The bar for minting a second case is neither that the two inputs differ, nor that their in-scope finding sets differ: > A variant case MAY be minted when the second input can fail, **within the case's scope**, > under a change to a scoped data file that the first input would pass. Where no edit to a > published table separates the two inputs, the second case cannot fail in any way the first > does not, and it MUST NOT be minted. Equal in-scope finding sets do not settle that question, and reading them as if they did is the error this paragraph replaces. The corpus's first variant case expects exactly the finding set its base case expects — one `pattern_id`, one scope, one `match` — and still gates something the base cannot: the base input matches the scoped pattern both in its published form and in the stricter form that preceded it, so reinstating the stricter form leaves it passing, while the variant input matches only the published form and fails. The distinguishing signal is *inside* the scope, in the form of the scoped rule itself, which is exactly what a finding-set comparison cannot see. The payload that stays out is the mirror image. A source runtime's own test matrix may drive two payloads at the same pattern, as one of the seeding runtimes does for `active:markdown-image`, whose only difference is a signal from a table this repository does not publish. No edit to any scoped file separates them, so the second case could not fail where the first passes. Such a payload SHOULD be recorded as a named omission in `conformance/manifest.json`, so the drop is visible rather than inferred from a count. A variant case MUST be scoped and matched exactly like its base case, and MUST expect the same `pattern_id`. **The suffix distinguishes inputs, never findings.** It is not a licence to record a second, different verdict for one rule: two cases at one pattern id expecting different findings within the same scope are not a variant pair, they are the corpus contradicting itself. **A case id is a stable identifier. Changing one is a BREAKING change** and requires a major bump of the corpus version, exactly like changing a pattern id. Consumers name cases in their own test suites; a renamed case is a silently disappeared test. ## 7. When a runtime and a fixture disagree The fixture is ground truth. A runtime that disagrees is wrong — **unless the fixture itself is proven wrong**, in which case the fixture changes in its own commit, with the reason written down, and the corpus version is bumped. This ordering is the whole point of the repository. Two implementations that return different verdicts on the same input are not holding different opinions; one of them has a bug. ### 7.1 Where the second paragraph does not hold **Through corpus version 0.8.1 this section stated the rule above with no scope at all**, and the scope was load-bearing: the justification names *two* implementations. Where a case's scope is a table only one runtime implements, and that runtime authored the payload the case was extracted from, there is no second implementation whose disagreement the paragraph could adjudicate. Which cases those are is recorded in the corpus, not asserted per run — see `active_content_provenance.asymmetry` in [`conformance/manifest.json`](../conformance/manifest.json). For such a case, a disagreement by the **seed runtime itself** is a third thing, and it is neither of the two the paragraph offers: - The fixture is not proven wrong. It recorded that runtime's behaviour correctly at the commit and version its own measurement block pins, and a later classification does not reach back and falsify an earlier measurement. - The runtime does not necessarily have a bug. Where the seed runtime has stated that the classification behind such a table is calibration it does not freeze, a release that classifies the payload differently is a change it reserved, not a defect. So: the fixture MUST NOT be rewritten on the strength of the divergence alone; the divergence SHOULD be recorded against the version pinned; and re-pinning the case to a later version of the seed runtime is a separate decision, taken deliberately and released on its own. This is the disposition §5 already applies to a stale `observed_out_of_scope` entry, extended to the one place where it can reach a verdict — and a divergence recorded here is the signal that the re-pinning decision is due, not a reason to leave it open. Three things this does **not** do. - **It creates no fourth verdict.** The counts of §1.1 and [`schema/conformance-declaration.schema.json`](../schema/conformance-declaration.schema.json) are unchanged: a case whose expected findings are not produced still **fails**, and is still named in `failed_cases`. What changes is what the failure licenses concluding, not what is reported. - **It does not reach a third-party implementer** of the same table. Against them the fixture is the contract, exactly as §7 states — which is what these cases were minted to provide, and the only thing they can prove while one runtime is all there is. - **It is not a licence to self-declare.** The exemption is carried by the corpus's own provenance record for the scope. A runtime MUST NOT claim it for a case by asserting that its own divergence is calibration. ## 8. What conformance does and does not prove Passing this corpus proves that a runtime agrees with the other runtimes that pass it, on these inputs. It does not prove the fixtures are right. A wrong `expected.json` makes every passing runtime wrong in the same way, which is a real failure mode and not a rhetorical one — it is why each case records how it was produced. Nor is the corpus a coverage claim. It covers the pattern tables named in `conformance/manifest.json` under `scope_covered`, and the tables under `scope_planned` are named there precisely so that their absence is visible rather than inferred. And a result is only readable next to the declared table set of §1.1. A pass count on its own does not say which questions were asked: a runtime that implements one table and one that implements four can print the same number. The declared set is what turns the number back into a claim — which is also why a `not-applicable` verdict proves nothing in either direction about the runtime's actual detection. It says the corpus did not ask.