open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

Squashed 'scanners/commons/' changes from 0ffee85..532d70d

Browse source 
532d70d release(0.2.0): the contract gained a MUST, so the tag has to move
946f51d fix(active-content,conformance): cite line numbers per commit — they do not resolve at the pin
bdcb1f1 feat(conformance): ship the six active-content cases; the id space already existed
807c0d4 feat(spec): add not-applicable, so a single-runtime table stops reading as 7 defects
a1578e6 fix(conformance): record the guard's internal-surface position on _LEX_PAYLOADS
4d351d2 fix(mapping): state that three of four OWASP maps have no production consumer
f082a91 fix(lexicon,docs): retract the claim that the guard's port cites severity.mjs

git-subtree-dir: scanners/commons
git-subtree-split: 532d70d5ed2f9b23a8efad760ef490356cf52ada
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:02:33 +02:00
commit 9b95fc607e
20 changed files with 533 additions and 37 deletions
Download patch file Download diff file Expand all files Collapse all files

92
conformance/manifest.json
View file

@ -1,34 +1,52 @@
{
"version": "0.1.0",
"version": "0.2.0",
"id": "llm-security-commons/conformance",
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
"case_id_derivation": {
"rule": "case_id = pattern_id with ':' replaced by '__'",
"reverse": "pattern_id = case_id with '__' replaced by ':'",
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 83.",
"stability": "A case id is a stable identifier. Changing one is a BREAKING change."
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 89.",
"stability": "A case id is a stable identifier. Changing one is a BREAKING change.",
"one_case_per_pattern_id": "The derivation takes a pattern id and nothing else, so a single-finding scope holds at most one case per pattern id — there is nowhere in the name to put a second. See spec/conformance-corpus.md section 6, and `omitted_payloads` below for the one payload this actually cost."
},
"match_semantics": "exact-within-scope",
"scope_covered": [
"lexicon/injection-lexicon.json"
"lexicon/injection-lexicon.json",
"signatures/active-content.json"
],
"scope_covered_note": "The two entries do NOT have equal standing, and averaging them would misreport both. `lexicon/injection-lexicon.json` is implemented by both seeding runtimes and its id space is ratified by both. `signatures/active-content.json` is implemented by one; its cases are `not-applicable` for the other under spec/conformance-corpus.md section 1.1, not failures. See that file's `pattern_id_space.single_runtime`.",
"scope_planned": {
"$comment": "Named rather than faked. The guard's coverage matrix holds 11 further convertible cases (3 carrier, 7 active-content, 1 secret-egress), but the lexicon is the only table whose finding id space is ratified by both seeding runtimes. Fixtures for the others would mint a cross-runtime contract this repository has no agreement for, so they wait for one.",
"$comment": "Named rather than faked. Four convertible cases remain in the guard's coverage matrix (3 carrier, 1 secret-egress), and neither table is blocked on effort — each is blocked on a distinct unresolved question, measured 2026-08-10 and recorded below rather than left as 'no agreement yet'.",
"codepoints/carriers.json": 3,
"signatures/active-content.json": 7,
"signatures/secret-egress.json": 1
"signatures/secret-egress.json": 1,
"blockers": {
"codepoints/carriers.json": "No adoptable id space, and a second problem underneath it. The guard emits TWO stage-coupled labels for the same carrier depending on pipeline position — `sanitize:zero-width` (input) versus `output:zero-width-present` (output), and the same split for bidi and unicode-tag; llm-security emits prose titles (unicode-scanner.mjs:191,236). A commons id would therefore have to be invented stage-neutral, which no other id space here required. And because `exact-within-scope` compares a finding SET, a commons id aliasing both guard labels would make the verdict depend on which entry point the runtime was measured through — an entry-point dependence the lexicon cases do not have, since this manifest pins entry point as a measurement fact rather than as contract.",
"signatures/secret-egress.json": "Not an id-naming question at all. The two runtimes carry DIFFERENT TABLES, not two namings of one: this file holds 18 entries from llm-security, the guard's `_SECRET_PATTERNS` (output.py) holds 25 at different cut points — this file's single `GitHub Token` is four ids there, `Private Key PEM Block` is three, `Database connection string` is four — and membership diverges both ways (the guard has `gcp-service-account-json` and `openai-api-key-legacy`, which are absent here; this file has `Slack/Discord Webhook URL` and `Azure AI Services Key`, which are absent there). `aws-access-key-id` is the one clean one-to-one, which is why exactly one egress case was ever offered. A shared id space presupposes a table reconciliation that has not happened."
}
},
"omitted_payloads": [
{
"source": "llm-ingestion-pipeline-security src/llm_ingestion_guard/coverage.py:484, at commit de09711 (line numbers are commit-relative; the structure is the fifth `_scan_case` of `_build_cases()`'s `active` group)",
"described_as": "opaque (base64) path segment",
"expected_label": "active:markdown-image",
"reason": "Its in-scope finding set is `[active:markdown-image]` — identical, measured, to the case built from coverage.py:476. The only thing that distinguishes it is `entropy:base64-blob`, and this repository publishes no entropy table, so the difference falls outside every declared scope. A second case could not have failed in any way the first does not, and the case-id derivation has no room for it (see `case_id_derivation.one_case_per_pattern_id`).",
"$comment": "Recorded so that 6 built from 7 offered reads as a decision rather than as a miscount."
}
],
"payload_provenance": {
"$scope": "The 83 lexicon cases. The 6 active-content cases have their own provenance in `active_content_provenance` below — they come from a different structure in the same file, at a different commit, and folding them in here would let one pin stand for two measurements.",
"source_repo": "llm-ingestion-pipeline-security",
"source_file": "src/llm_ingestion_guard/coverage.py",
"source_export": "_LEX_PAYLOADS",
"source_commit": "0bf07295c2191d5061537834abf22929f7d50826",
"source_version": "0.3.4",
"$comment": "The inputs were authored by one of the two runtimes, as one payload per pattern id, and are reproduced verbatim. That asymmetry is stated rather than averaged away: what makes them usable as a cross-runtime corpus is not their origin but the measurement below, which ran them through the other runtime as well and found the same lexicon verdict on every one.",
"id_set_check": "The 83 payload keys and the 83 commons pattern ids are the same set — compared, not assumed."
"id_set_check": "The 83 payload keys and the 83 commons pattern ids are the same set — compared, not assumed.",
"stability_note": "llm-ingestion-pipeline-security states (coord message 2026-08-10T12:42:31Z) that _LEX_PAYLOADS and the 83 pattern ids are an internal surface on their side, with no README/CHANGELOG/docs statement promising id or payload stability - their own test suite enforces id coverage as their gate, not as a promise to this repository. Their stated position: if a payload changes upstream, this manifest's pin diverges and should be re-pinned; divergence is a re-pin signal, not a breach of a contract they never granted."
},
"measurement": {
"$scope": "The 83 lexicon cases. The 6 active-content cases are measured in `active_content_measurement`.",
"date": "2026-08-10",
"method": "Each payload was run through both runtimes' PUBLIC entry point — not through a rebuilt regex table — and the resulting finding labels were mapped to commons pattern ids through the lexicon's own aliases block. Comparing at the entry point is deliberate: a table-level comparison produces a number that describes neither runtime.",
"runtimes": [
@ -65,7 +83,55 @@
"see": "docs/lexicon-port-divergence.md"
}
},
"count": 83,
"active_content_provenance": {
"source_repo": "llm-ingestion-pipeline-security",
"source_file": "src/llm_ingestion_guard/coverage.py",
"source_structure": "the `active` group of `_build_cases()` — seven `_scan_case(...)` call sites, coverage.py:476-488",
"source_commit": "de09711",
"source_version": "0.4.0",
"extraction_method": "The payload strings were located by parsing coverage.py's AST for `_scan_case` calls whose first argument is `\"active\"`, and each payload expression was evaluated against that module's own namespace (which supplies the base64 constant one of them embeds). Retyping them here would have been the from-memory failure this repository forbids for detection data.",
"$comment": "A different structure and a different commit from `payload_provenance` above, which covers the 83 lexicon cases. Six of the seven extracted payloads became cases; the seventh is in `omitted_payloads` with its measured reason.",
"asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing."
},
"active_content_measurement": {
"date": "2026-08-10",
"method": "Each payload was run through the guard's PUBLIC output gate — the same entry point the 83 lexicon cases were measured through, so the two halves of the corpus are comparable. Findings were split into in-scope and residue by MEMBERSHIP in signatures/active-content.json's published `pattern_id_space.ids`, not by the `active:` prefix (see spec/conformance-corpus.md section 4).",
"verification": "The generated fixtures were then re-read from disk by a separate checker that re-computed each digest, re-scanned the bytes, and applied `exact-within-scope` independently of the generator — a generator agreeing with itself proves nothing. 6 cases, 0 failed checks.",
"runtimes": [
{
"name": "llm_ingestion_guard",
"repo": "llm-ingestion-pipeline-security",
"commit": "de09711",
"version": "0.4.0",
"entry_point": "scan_output(text, source=Source.OUTPUT)",
"engine": "CPython 3.14.0",
"measurement_limit": "Findings outside signatures/active-content.json that this gate raised are recorded per case in observed_out_of_scope as informative evidence, never as expectation. Two of the six carry such residue, and in both it is a lexicon id."
},
{
"name": "llm_security",
"repo": "ssh://git@git.fromaitochitta.com/open/llm-security.git",
"commit": "b0de0ca6d86ce697f39669d177c2c2654c280128",
"status": "not-applicable",
"status_basis": "This runtime does not implement signatures/active-content.json, so under spec/conformance-corpus.md section 1.1 these six cases are `not-applicable` there — neither passes nor failures. It was NOT run against them; a measurement whose verdict the spec defines as inapplicable would produce a number that means nothing.",
"absence_evidence": "Measured at b0de0ca, not assumed. A case-insensitive tree-wide search for an active-content detector, class table or emitted `active:` label returns no implementation — only prose in `knowledge/` and an entropy-scanner test mentioning the markdown-image construct. `git log -S 'active_content'` and `git log -S 'active:markdown'` over `--all` return zero commits, closing the 'it was there once' reading.",
"not_a_coverage_claim": "Absent table is not absent capability: this runtime's entropy scanner reaches markdown-image URLs by a different route (host- and query-aware). What is measured absent is the table, which is all `not-applicable` asserts."
}
],
"results": {
"payloads_extracted": 7,
"cases_built": 6,
"payloads_omitted": 1,
"cases_verified_independently": 6,
"failed_checks": 0,
"cases_with_out_of_scope_residue_in_the_guard": 2
},
"guard_drift_note": "The guard's HEAD moved twice during this measurement (3c56d50 -> de09711 -> 398eb74). Measurement ran at de09711. Checked rather than hoped: 398eb74 touches only docs/PLAN-v1.md and tests/test_disposition.py, so no measured file changed under the measurement. active_content.py DID change between 0bf0729 — the commit signatures/active-content.json pins — and de09711; the change adds a scan-cap self-safety finding and touches no construct. See that file's `pattern_id_space.verified.at_commit_note`."
},
"count": 89,
"count_by_scope": {
"lexicon/injection-lexicon.json": 83,
"signatures/active-content.json": 6
},
"cases": [
"override__ignore-previous",
"override__forget-instructions",
@ -149,6 +215,12 @@
"hybrid-xss__script-tag",
"hybrid-xss__javascript-uri",
"hybrid-xss__event-handler",
"hybrid-xss__iframe-src"
"hybrid-xss__iframe-src",
"active__markdown-image",
"active__markdown-link",
"active__reference-link",
"active__autolink",
"active__raw-html",
"active__data-uri"
]
}