open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

feat(schema): add finding.schema.json with normative SARIF profile

Browse source 
The finding contract plus the SARIF 2.1.0 output profile, modelled on
llm-security's sarif-formatter.mjs (operator dump). Draft 2020-12, valid
against the metaschema.

The SARIF profile is proven, not transcribed: the mapping was re-implemented
from the commons JSON alone and diffed against the real toSARIF over 10
envelope shapes (all severities incl. unknown/undefined, slug edge cases,
rule-id collision, every optional-field combination, multi-scanner, explicit
version) — 0 differences.

Three things left honest rather than closed:
- additionalProperties stays open. The formatter CONSUMES findings, so fields
  it ignores are invisible in this evidence; the property list is a lower
  bound until the producer is supplied.
- The JSONL profile is status=unspecified with a TODO. "One finding per line"
  is inference, and a guessed normative contract is worse than a missing one.
- Node and the Python guard share only severity+owasp; the rest of the field
  names diverge (scanner/title vs detector/label). Recorded as an open
  question for v0.2.0, not settled silently here.

One published claim was wrong and is corrected: rule-id collision does NOT
cover punctuation, only case and whitespace runs. Caught by executing the
claim instead of asserting it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0191AKc2qW6tmXDFSx1xn53q
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 14:52:53 +02:00
commit 0433240b14
2 changed files with 230 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

18
docs/extraction-plan.md
View file

@ -93,6 +93,24 @@ Not verified, and not claimed: that the Node consumer's active-content behaviour
this table. The source module states the Node port shares its severities; that is the this table. The source module states the Node port shares its severities; that is the
module's claim, and confirming it needs the Node file. module's claim, and confirming it needs the Node file.
### `schema/finding.schema.json` — extracted 2026-08-09
Source: `llm-security/scanners/lib/sarif-formatter.mjs`, supplied as an operator dump. No
commit hash accompanied it, so provenance is recorded as `unknown` rather than guessed.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed | `python3 -m json.tool` | pass |
| Valid JSON Schema | `jsonschema` `check_schema` against draft 2020-12 | pass |
| Accepts/rejects findings correctly | 2 valid + 3 invalid findings (missing `scanner`, unknown severity, `line: 0`) | 5/5 as intended |
| SARIF profile reproduces the source | re-implemented the mapping **from the commons JSON alone** and diffed `JSON.stringify` against the real `toSARIF` over 10 envelope shapes: empty, missing `scanners`, empty `scanners`, scanner with no findings, all five severities plus an unknown and an `undefined` one, five slug edge cases (double space, tab, newline, leading/trailing space, mixed case), a rule-id collision, all seven optional-field combinations, two scanners, and an explicit `version` argument | 10/10 identical, 0 differences |
| The three `known_lossiness` claims are true | executed each against the real formatter | all three confirmed, **and one earlier claim corrected**: punctuation does *not* collapse — the slug lowercases and collapses whitespace only, so `Zero-width carrier` and `Zero-width carrier!` remain distinct ids. The wrong claim was published in the first draft of this file and fixed before commit. |
Not verified, and recorded in the file as open: the finding **producer** was not supplied, so
the property list is a lower bound; `scanner` and `severity` are required by design rather
than by evidence; and the JSONL profile is left explicitly `unspecified` rather than
invented, because "one finding per line" is inference.
## Definition of done for v0.1.0 ## Definition of done for v0.1.0
1. Repository initialized, Forgejo remote `open/llm-security-commons`, MIT, `STATE.md` 1. Repository initialized, Forgejo remote `open/llm-security-commons`, MIT, `STATE.md`

212
schema/finding.schema.json Normal file
View file

@ -0,0 +1,212 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.fromaitochitta.com/open/llm-security-commons/raw/branch/main/schema/finding.schema.json",
"version": "0.1.0",
"status": "normative",
"title": "Finding",
"description": "The finding contract: what a detector emits, and how that shape maps onto the SARIF and JSONL output profiles. Normative for the field names, types and severity vocabulary. A consumer that emits a differently-named field is not producing a finding under this contract, however similar the meaning.",
"$comment": "Seeded from llm-security/scanners/lib/sarif-formatter.mjs (operator dump `dump-01-sarif-formatter.mjs`, 2026-08-09). READ THE EVIDENCE NOTE in `provenance.evidence_limits` before tightening anything here: that file CONSUMES findings, it does not produce them. It therefore proves the shape of the fields SARIF reads, and says nothing at all about fields it ignores. `additionalProperties` is deliberately left open for exactly that reason.",
"provenance": {
"source_repo": "llm-security",
"source_files": ["scanners/lib/sarif-formatter.mjs"],
"source_delivery": "operator dump, 2026-08-09",
"source_commit": "unknown — not supplied with the dump",
"verified": "partially",
"evidence_limits": [
"The formatter is a consumer. Any finding field it does not read is invisible in this evidence, so the property list here is a LOWER BOUND on what a finding carries, never the full set.",
"`title` is the only property the source PROVES to be required: `f.title.replace(...)` throws if it is absent. `scanner` and `severity` are required here by design, not by evidence — the formatter tolerates their absence (producing the rule id `undefined/...`, and defaulting the level to `note` respectively). See `open_questions`.",
"The finding PRODUCER (the scan orchestrator that builds the envelope) has not been supplied. Until it is, this schema cannot be called closed."
]
},
"type": "object",
"$ref": "#/$defs/finding",
"$defs": {
"severity": {
"description": "The finding severity vocabulary. Ordered most to least severe; both seed implementations use these five values.",
"type": "string",
"enum": ["critical", "high", "medium", "low", "info"]
},
"owasp": {
"description": "An OWASP taxonomy anchor, e.g. `LLM05`. Resolved against mapping/owasp-map.json. Optional: a finding may predate its taxonomy entry.",
"type": "string",
"minLength": 1
},
"finding": {
"type": "object",
"required": ["scanner", "title", "severity"],
"additionalProperties": true,
"$comment": "`additionalProperties: true` is a statement about the evidence, not laxness. Closing it would assert that no finding carries a field the SARIF formatter ignores, which this seed source cannot support.",
"properties": {
"scanner": {
"description": "Identifier of the detector that produced the finding. Forms the first segment of the SARIF rule id.",
"type": "string",
"minLength": 1
},
"title": {
"description": "Short human-readable name of the detected class. Forms the second segment of the SARIF rule id, and is the fallback for both `description` and the SARIF result message.",
"type": "string",
"minLength": 1
},
"severity": { "$ref": "#/$defs/severity" },
"description": {
"description": "Longer explanation. Falls back to `title` wherever it is absent.",
"type": "string"
},
"recommendation": {
"description": "What to do about it. Carried through to SARIF as a result property, not as part of the message.",
"type": "string"
},
"owasp": { "$ref": "#/$defs/owasp" },
"file": {
"description": "Path the finding is anchored to. A finding without `file` has no SARIF location at all.",
"type": "string"
},
"line": {
"description": "1-indexed line within `file`. Silently dropped if `file` is absent — see `known_lossiness`.",
"type": "integer",
"minimum": 1
}
}
},
"scannerResult": {
"description": "One scanner's contribution to an envelope.",
"type": "object",
"additionalProperties": true,
"properties": {
"findings": {
"type": "array",
"items": { "$ref": "#/$defs/finding" }
}
}
},
"envelope": {
"description": "The multi-scanner container the SARIF profile is generated from. A missing or empty `scanners` map is legal and yields an empty SARIF run rather than an error.",
"type": "object",
"additionalProperties": true,
"properties": {
"scanners": {
"type": "object",
"additionalProperties": { "$ref": "#/$defs/scannerResult" }
}
}
}
},
"profiles": {
"$comment": "Profiles are mapping rules, not JSON Schema keywords — a validator ignores this key, so the file stays a usable schema while carrying the mappings the plan requires alongside it.",
"sarif": {
"status": "normative",
"spec": "SARIF 2.1.0 (OASIS)",
"spec_url": "https://docs.oasis-open.org/sarif/sarif/v2.1.0/",
"schema_url": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json",
"sarif_version": "2.1.0",
"tool": {
"name": "llm-security",
"information_uri": "https://git.fromaitochitta.com/open/llm-security",
"version_default": "6.0.0",
"$comment": "`name` and `information_uri` identify the SEED implementation. A different consumer emitting SARIF under this profile MUST substitute its own tool identity — copying these would attribute its findings to llm-security."
},
"external_dependencies": "none",
"rule_id": {
"template": "{scanner}/{slug(title)}",
"slug": {
"steps": [
"replace every run of whitespace with a single `-`",
"lowercase"
],
"pattern_applied": "\\s+",
"replacement": "-"
}
},
"severity_to_level": {
"critical": "error",
"high": "error",
"medium": "warning",
"low": "note",
"info": "note"
},
"severity_to_level_default": "note",
"severity_to_level_note": "The mapping is many-to-one and NOT invertible: `critical` and `high` both become `error`, `low` and `info` both become `note`. A consumer that round-trips a finding through SARIF loses the distinction. Findings, not SARIF, are the interchange format here.",
"rules_array": {
"keyed_by": "rule_id",
"first_wins": true,
"first_wins_note": "The rule is built from the FIRST finding carrying a given rule id, so `fullDescription` reflects that finding's `description`. Each individual result keeps its own message, so nothing is lost at result level — but two findings sharing a title and differing in description produce one rule text.",
"fields": {
"id": "rule_id",
"name": "title",
"shortDescription.text": "title",
"fullDescription.text": "description, falling back to title",
"defaultConfiguration.level": "severity_to_level[severity]",
"properties.tags": "[owasp] when present, otherwise []"
}
},
"result": {
"fields": {
"ruleId": "rule_id",
"ruleIndex": "index into runs[0].tool.driver.rules",
"level": "severity_to_level[severity]",
"message.text": "description, falling back to title",
"properties.tags": "[owasp] — omitted entirely when owasp is absent",
"properties.recommendation": "recommendation — omitted when absent",
"locations": "single-element array, present only when `file` is present"
},
"location_shape": {
"physicalLocation.artifactLocation.uri": "file",
"physicalLocation.region.startLine": "line — added only when `line` is present"
}
},
"envelope_traversal": "Findings are collected across every value of `envelope.scanners`, in object-iteration order, and flattened into one run. Scanner identity survives only inside each finding's own `scanner` field.",
"runs": "exactly one"
},
"jsonl": {
"status": "unspecified",
"$comment": "TODO — NOT extracted, and deliberately not invented. The plan calls for a JSONL profile, but no JSONL emitter was supplied with the dump, and the obvious reading (one finding object per line) is inference, not evidence. Guessing here would publish a normative contract nobody implements. Needed to close it: whichever module in llm-security writes JSONL output.",
"needed_source": "llm-security — the JSONL/NDJSON output writer",
"must_not_assume": [
"that a line holds a bare finding rather than a wrapper carrying scanner or run context",
"that severity is emitted as the finding vocabulary rather than the SARIF level",
"that field order, or a trailing newline on the final line, is unconstrained"
]
}
},
"known_lossiness": [
{
"id": "rule-id-collision",
"claim": "Two titles that differ only in letter case, or only in which whitespace runs separate their words, produce ONE rule id — silently merging two classes into one entry in the SARIF rules array. The merged rule carries the FIRST finding's description; each result keeps its own message, so nothing is lost at result level.",
"does_not_apply_to": "Punctuation. The slug lowercases and collapses whitespace and nothing else, so `Zero-width carrier` and `Zero-width carrier!` stay distinct ids.",
"verified": "`A B` + `a b` -> 1 rule / 2 results, rule fullDescription `first`, result messages [`first`,`second`]. `Zero-width carrier` + `Zero-width carrier!` -> ids [`sc/zero-width-carrier`, `sc/zero-width-carrier!`]."
},
{
"id": "line-without-file-dropped",
"claim": "A finding carrying `line` but no `file` produces no SARIF location at all; the line number is discarded without warning.",
"verified": "`{line: 9}` with no `file` -> `result.locations` is absent."
},
{
"id": "severity-not-invertible",
"claim": "critical and high both map to `error`; low and info both map to `note`. A finding round-tripped through SARIF cannot recover its original severity.",
"verified": "[critical, high, medium, low, info] -> [error, error, warning, note, note]."
}
],
"open_questions": [
"Required-field set. Only `title` is enforced by the seed implementation; `scanner` and `severity` are required here by design. Adopting this schema as a validator in llm-security could therefore reject findings it currently emits. That is a behaviour change and belongs in llm-security's own phase, decided against its own tests — not here.",
"Cross-runtime field names. The Python guard's Finding carries `label`, `detector`, `count`, `evidence` and `source`; only `severity` and `owasp` are common with the Node shape. A single contract two runtimes conform to needs either a rename in one of them or a declared alias table. Deferred to v0.2.0 with the decision stated, rather than settled silently here.",
"The finding producer (scan orchestrator) has not been supplied, so the property list is a lower bound."
]
}