open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

feat(schema): close the finding contract against its producer

Browse source 
The schema was seeded from sarif-formatter.mjs, which consumes findings.
That could only ever establish a lower bound on the property set, so
additionalProperties had to stay open. The producer is now readable:
finding() in scanners/lib/output.mjs line 32, at b0de0ca. It returns an
object literal with exactly ten keys and no spread, so the set is complete
and the schema closes.

Added: id and evidence, the two keys a consumer-side reading could not see.
id gets its own definition, DS-<prefix>-<counter>, with pattern
^DS-[A-Za-z]+-[0-9]{3,} - the {3,} because padStart(3) is a minimum, so a
run past 999 findings produces four digits. It comes from a process-global
counter and is stable neither across runs nor across processes; the
definition says so before someone keys on it.

Nullability is evidence now, not convention. Five keys are emitted as null
rather than omitted, so a serialised finding always carries all ten. The
four assigned straight from opts are the exception: omit description and
the key is undefined and disappears from the JSON. Reproduced against the
real producer - ten keys in memory, nine serialised.

owasp is a string, not an array, and not one code. Multiple codes are
joined with ", ". Measured across the seed runtime: 31 distinct values over
157 sites, 13 multi-code, and four that mix taxonomies inside a single
value with no discriminator. That has a consequence nobody had written
down: sarif-formatter builds tags: [f.owasp], so "LLM06, ASI02" becomes ONE
tag with a comma in it and nothing filtering on LLM06 matches. Reproduced
end to end through the real finding() and toSARIF(), logged as
known_lossiness.owasp-tag-not-split. It is consumer behaviour, not data, so
it is reported rather than fixed here.

The JSONL profile is set to "not applicable" rather than "unspecified".
The distinction carries weight: unspecified would assert a profile exists
and has merely not been written down. There is no finding-JSONL - findings
are emitted only inside one JSON envelope. The single module that does
write JSONL, audit-trail.mjs, writes audit events under a different schema
where owasp is an ARRAY. Same field name, different type, same repository.
The profile records that trap instead of leaving a TODO.

Verified: valid Draft 2020-12, every finding built by the real producer
validates, and four negative controls are rejected.

One new open question left unpatched: the producer's JSDoc lists seventeen
scanner prefixes including IDE, while all four maps in owasp-map.json are
keyed on sixteen without it. An IDE finding has no taxonomy mapping
anywhere. Adding the key would be inventing detection data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SNMcqrfNyoLRQ7qXUFZnb9
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 22:46:13 +02:00
commit 7e2e92eecf
2 changed files with 116 additions and 32 deletions
Download patch file Download diff file Expand all files Collapse all files

99
schema/finding.schema.json
View file

@ -1,23 +1,27 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.fromaitochitta.com/open/llm-security-commons/raw/branch/main/schema/finding.schema.json",
"version": "0.1.0",
"version": "0.2.0",
"status": "normative",
"title": "Finding",
"description": "The finding contract: what a detector emits, and how that shape maps onto the SARIF and JSONL output profiles. Normative for the field names, types and severity vocabulary. A consumer that emits a differently-named field is not producing a finding under this contract, however similar the meaning.",
"description": "The finding contract: what a detector emits, and how that shape maps onto the SARIF output profile. Normative for the field names, types and severity vocabulary. A consumer that emits a differently-named field is not producing a finding under this contract, however similar the meaning. A JSONL profile is carried as `not applicable` rather than dropped, because the reason it does not exist is itself part of the contract - see profiles.jsonl.",
"$comment": "Seeded from llm-security/scanners/lib/sarif-formatter.mjs (operator dump `dump-01-sarif-formatter.mjs`, 2026-08-09). READ THE EVIDENCE NOTE in `provenance.evidence_limits` before tightening anything here: that file CONSUMES findings, it does not produce them. It therefore proves the shape of the fields SARIF reads, and says nothing at all about fields it ignores. `additionalProperties` is deliberately left open for exactly that reason.",
"$comment": "Seeded from llm-security/scanners/lib/sarif-formatter.mjs, which CONSUMES findings; closed in 0.2.0 against the PRODUCER, `finding()` in scanners/lib/output.mjs, read at commit b0de0ca. The producer settles what the consumer could not: the property list is now the complete set, not a lower bound, and `additionalProperties` is closed.",
"provenance": {
"source_repo": "llm-security",
"source_files": ["scanners/lib/sarif-formatter.mjs"],
"source_delivery": "operator dump, 2026-08-09",
"source_commit": "unknown — not supplied with the dump",
"verified": "partially",
"source_files": [
"scanners/lib/output.mjs",
"scanners/lib/sarif-formatter.mjs"
],
"source_delivery": "operator dump, 2026-08-09, for the formatter; the producer was read directly from the public remote at the commit below",
"source_commit": "b0de0ca6d86ce697f39669d177c2c2654c280128",
"source_remote": "ssh://git@git.fromaitochitta.com/open/llm-security.git",
"verified": "the property set, its order and the id format are verified against output.mjs:finding() (line 32); the SARIF profile remains verified against sarif-formatter.mjs",
"evidence_limits": [
"The formatter is a consumer. Any finding field it does not read is invisible in this evidence, so the property list here is a LOWER BOUND on what a finding carries, never the full set.",
"`title` is the only property the source PROVES to be required: `f.title.replace(...)` throws if it is absent. `scanner` and `severity` are required here by design, not by evidence — the formatter tolerates their absence (producing the rule id `undefined/...`, and defaulting the level to `note` respectively). See `open_questions`.",
"The finding PRODUCER (the scan orchestrator that builds the envelope) has not been supplied. Until it is, this schema cannot be called closed."
"`finding()` returns an object literal with ten keys and no spread, so the property set is complete for the SEED runtime. It is not proof that a second runtime cannot carry more; it is proof that this contract's producer does not.",
"Required-field enforcement is still a design decision, not an observation. `finding()` validates nothing: called without `severity` it emits a finding whose severity is undefined. See `open_questions`.",
"The scanner prefixes are documented in the producer's JSDoc, not enforced by it. `scanner` is typed as a plain non-empty string here for that reason. See `open_questions` for the IDE prefix, which the JSDoc lists and mapping/owasp-map.json does not."
]
},
@ -32,17 +36,31 @@
},
"owasp": {
"description": "An OWASP taxonomy anchor, e.g. `LLM05`. Resolved against mapping/owasp-map.json. Optional: a finding may predate its taxonomy entry.",
"description": "Taxonomy anchor(s) for the finding, as ONE string. Not an array, and not a single code: when a finding belongs to several categories the codes are joined with a comma and a space, e.g. `MCP03, MCP06`. Resolved against mapping/owasp-map.json. `null` when the finding carries no anchor.",
"type": ["string", "null"],
"minLength": 1,
"$comment": "Measured across the seed runtime's scanners at b0de0ca: 31 distinct values over 157 emission sites, of which 13 values (21 sites) carry more than one code. The separator is `, ` at every one of them - no bare comma, no other spelling. Four values mix TAXONOMIES inside a single string (`LLM01, ASI01`, `LLM01, ASI02`, `LLM03, ASI04`, `LLM06, ASI02`), so the field carries no discriminator saying which taxonomy a code belongs to. A consumer cannot resolve `LLM06, ASI02` against mapping/owasp-map.json without splitting on `, ` first and then dispatching per code prefix. See open_questions and known_lossiness.owasp-tag-not-split.",
"multi_value": {
"separator": ", ",
"split_rule": "Split on `, ` to obtain individual codes. Each code independently matches ^(?:LLM|ASI|AST|MCP)[0-9]{1,2}$ in the seed runtime; the code prefix, not the field, identifies the taxonomy.",
"taxonomy_discriminator": "absent by design in the seed runtime"
}
},
"findingId": {
"description": "Identifier assigned by the producer. Format `DS-<scanner prefix>-<counter>`, where the counter is zero-padded to at least three digits.",
"type": "string",
"minLength": 1
"pattern": "^DS-[A-Za-z]+-[0-9]{3,}$",
"$comment": "Built at output.mjs:34 as `DS-${opts.scanner}-${String(n).padStart(3, '0')}` from a process-global counter. Two consequences a consumer must not design around: the id is NOT stable across runs (it depends on emission order within a process), and it is NOT unique across processes. `padStart(3)` sets a minimum, not a maximum - a run emitting more than 999 findings produces four-digit counters, which is why the pattern is `{3,}` and not `{3}`."
},
"finding": {
"type": "object",
"required": ["scanner", "title", "severity"],
"additionalProperties": true,
"$comment": "`additionalProperties: true` is a statement about the evidence, not laxness. Closing it would assert that no finding carries a field the SARIF formatter ignores, which this seed source cannot support.",
"required": ["id", "scanner", "title", "severity"],
"additionalProperties": false,
"$comment": "Closed in 0.2.0. `finding()` in output.mjs returns an object literal with exactly these ten keys and no spread, so a finding under this contract carries nothing else. The five optional-in-practice keys are emitted as `null` rather than omitted (`opts.x || null`), so a serialised seed finding always has all ten keys present. `description`, `severity`, `title` and `scanner` are the exception: they are assigned straight from `opts` with no fallback, so if a caller omits one the key is `undefined` and disappears from the JSON entirely. That is why they are typed non-nullable and required rather than nullable. Reproduced against the real producer: `finding({scanner, severity, title})` with no `description` returns ten keys in memory and serialises to nine.",
"properties": {
"id": { "$ref": "#/$defs/findingId" },
"scanner": {
"description": "Identifier of the detector that produced the finding. Forms the first segment of the SARIF rule id.",
"type": "string",
@ -59,18 +77,22 @@
"type": "string"
},
"recommendation": {
"description": "What to do about it. Carried through to SARIF as a result property, not as part of the message.",
"type": "string"
"description": "What to do about it. Carried through to SARIF as a result property, not as part of the message. Emitted as `null` when the producer has none.",
"type": ["string", "null"]
},
"owasp": { "$ref": "#/$defs/owasp" },
"file": {
"description": "Path the finding is anchored to. A finding without `file` has no SARIF location at all.",
"type": "string"
"description": "Path the finding is anchored to. A finding whose `file` is `null` has no SARIF location at all.",
"type": ["string", "null"]
},
"line": {
"description": "1-indexed line within `file`. Silently dropped if `file` is absent — see `known_lossiness`.",
"type": "integer",
"description": "1-indexed line within `file`. Silently dropped if `file` is absent — see `known_lossiness`. Emitted as `null` when the producer has none; note that `opts.line || null` also turns a literal 0 into `null`, which is harmless here only because line numbers are 1-indexed.",
"type": ["integer", "null"],
"minimum": 1
},
"evidence": {
"description": "Redacted excerpt supporting the finding. Emitted as `null` when the producer has none.",
"type": ["string", "null"]
}
}
},
@ -174,13 +196,20 @@
},
"jsonl": {
"status": "unspecified",
"$comment": "TODO — NOT extracted, and deliberately not invented. The plan calls for a JSONL profile, but no JSONL emitter was supplied with the dump, and the obvious reading (one finding object per line) is inference, not evidence. Guessing here would publish a normative contract nobody implements. Needed to close it: whichever module in llm-security writes JSONL output.",
"needed_source": "llm-security — the JSONL/NDJSON output writer",
"must_not_assume": [
"that a line holds a bare finding rather than a wrapper carrying scanner or run context",
"that severity is emitted as the finding vocabulary rather than the SARIF level",
"that field order, or a trailing newline on the final line, is unconstrained"
"status": "not applicable",
"$comment": "Closed 2026-08-09 as NOT APPLICABLE rather than unspecified, and the distinction is the point: `unspecified` would say a profile exists and has not been written down, which would be false. No finding-JSONL exists in the seed runtime. Findings are emitted only inside a single JSON envelope (output.mjs:140, `envelope()`), never line by line, so there is nothing for a JSONL finding profile to describe. A consumer wanting newline-delimited findings is defining a new format, not conforming to an existing one, and MUST NOT cite this schema as the authority for its line shape.",
"resolved_from": "llm-security, coord reply 2026-08-09, confirmed against the module at b0de0ca",
"not_to_be_confused_with": [
{
"what": "scanners/lib/audit-trail.mjs",
"is": "The one module in llm-security that does write JSONL (`appendFileSync(path, JSON.stringify(entry) + '\\n')`). It writes AUDIT EVENTS, not findings, under a different schema entirely: timestamp, session_id, event_type, severity, source, details, owasp, action_taken.",
"why_it_matters": "Its `owasp` field is an ARRAY of strings. The finding contract's `owasp` is a single comma-joined STRING. Two shapes, two types, one field name, in the same repository. A consumer that reads audit JSONL and finding JSON through one code path will be wrong about one of them.",
"activation": "Driven by the policy key `audit.log_path`; a no-op without it."
},
{
"what": "/tmp/llm-security-session-*.jsonl",
"is": "Hook session state. A third format again, and not a findings stream."
}
]
}
},
@ -201,12 +230,20 @@
"id": "severity-not-invertible",
"claim": "critical and high both map to `error`; low and info both map to `note`. A finding round-tripped through SARIF cannot recover its original severity.",
"verified": "[critical, high, medium, low, info] -> [error, error, warning, note, note]."
},
{
"id": "owasp-tag-not-split",
"claim": "A multi-code `owasp` value becomes ONE SARIF tag, not one per code. The formatter writes `tags: [f.owasp]` at sarif-formatter.mjs:49 and :90, so a finding anchored to `LLM06, ASI02` is tagged with the literal string `\"LLM06, ASI02\"`. No SARIF consumer filtering on the tag `LLM06` will match it, and the finding is effectively untagged for every taxonomy it claims.",
"scope": "Affects the 13 multi-code values emitted at 21 sites in the seed runtime at b0de0ca, four of which also mix taxonomies.",
"verified": "Reproduced, not only read. A finding built by the real `finding()` with `owasp: 'LLM06, ASI02'` and run through the real `toSARIF()` at b0de0ca yields `properties.tags` of `[\"LLM06, ASI02\"]` on BOTH the rule and the result - one tag, not two.",
"not_fixed_here": "This is consumer behaviour in llm-security, not data. v0.1.0 preserves behaviour; the defect is recorded and reported rather than corrected in this repository."
}
],
"open_questions": [
"Required-field set. Only `title` is enforced by the seed implementation; `scanner` and `severity` are required here by design. Adopting this schema as a validator in llm-security could therefore reject findings it currently emits. That is a behaviour change and belongs in llm-security's own phase, decided against its own tests — not here.",
"Cross-runtime field names. The Python guard's Finding carries `label`, `detector`, `count`, `evidence` and `source`; only `severity` and `owasp` are common with the Node shape. A single contract two runtimes conform to needs either a rename in one of them or a declared alias table. Deferred to v0.2.0 with the decision stated, rather than settled silently here.",
"The finding producer (scan orchestrator) has not been supplied, so the property list is a lower bound."
"Required-field set. The producer enforces nothing - `finding()` validates no argument, so a caller omitting `severity` emits a finding with `severity: undefined`. This schema requires `id`, `scanner`, `title` and `severity` by design. Adopting it as a validator inside llm-security could therefore reject findings that runtime currently emits. That is a behaviour change and belongs in llm-security's own phase, decided against its own tests - not here.",
"Cross-runtime field names. The Python guard's Finding (report.py:53) carries `label`, `severity`, `source`, `detector`, `count`, `offset`, `evidence` and `owasp`; only `severity`, `evidence` and `owasp` are common with the Node shape, and `owasp` is common in name only - the guard has not been shown to join multiple codes into one string. A single contract two runtimes conform to needs either a rename in one of them or a declared alias table. Deferred to v0.2.0 of the REPOSITORY contract with the decision stated, rather than settled silently here.",
"Taxonomy discriminator on `owasp`. The field carries codes from four taxonomies in one comma-joined string with nothing saying which is which, and mapping/owasp-map.json now records that one of those four (AST) is not an OWASP taxonomy at all. Splitting on the code prefix works today only because the four prefixes happen to be distinct. This is the seam most likely to break a second runtime.",
"The `IDE` scanner prefix. The producer's JSDoc lists seventeen prefixes - UNI, ENT, PRM, DEP, TNT, GIT, NET, MEM, SCR, WFL, TRG, SIG, AST, TFA, IDE, MCI, PST - while all four maps in mapping/owasp-map.json are keyed on sixteen, without IDE. A finding from the IDE scanner therefore has an id and a scanner prefix but no taxonomy mapping in any of the four maps. Reported rather than patched: adding a key to those maps would be inventing detection data."
]
}