249 lines
18 KiB
JSON
249 lines
18 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"$id": "https://git.fromaitochitta.com/open/llm-security-commons/raw/branch/main/schema/finding.schema.json",
|
|
"version": "0.2.0",
|
|
"status": "normative",
|
|
"title": "Finding",
|
|
"description": "The finding contract: what a detector emits, and how that shape maps onto the SARIF output profile. Normative for the field names, types and severity vocabulary. A consumer that emits a differently-named field is not producing a finding under this contract, however similar the meaning. A JSONL profile is carried as `not applicable` rather than dropped, because the reason it does not exist is itself part of the contract - see profiles.jsonl.",
|
|
|
|
"$comment": "Seeded from llm-security/scanners/lib/sarif-formatter.mjs, which CONSUMES findings; closed in 0.2.0 against the PRODUCER, `finding()` in scanners/lib/output.mjs, read at commit b0de0ca. The producer settles what the consumer could not: the property list is now the complete set, not a lower bound, and `additionalProperties` is closed.",
|
|
|
|
"provenance": {
|
|
"source_repo": "llm-security",
|
|
"source_files": [
|
|
"scanners/lib/output.mjs",
|
|
"scanners/lib/sarif-formatter.mjs"
|
|
],
|
|
"source_delivery": "operator dump, 2026-08-09, for the formatter; the producer was read directly from the public remote at the commit below",
|
|
"source_commit": "b0de0ca6d86ce697f39669d177c2c2654c280128",
|
|
"source_remote": "ssh://git@git.fromaitochitta.com/open/llm-security.git",
|
|
"verified": "the property set, its order and the id format are verified against output.mjs:finding() (line 32); the SARIF profile remains verified against sarif-formatter.mjs",
|
|
"evidence_limits": [
|
|
"`finding()` returns an object literal with ten keys and no spread, so the property set is complete for the SEED runtime. It is not proof that a second runtime cannot carry more; it is proof that this contract's producer does not.",
|
|
"Required-field enforcement is still a design decision, not an observation. `finding()` validates nothing: called without `severity` it emits a finding whose severity is undefined. See `open_questions`.",
|
|
"The scanner prefixes are documented in the producer's JSDoc, not enforced by it. `scanner` is typed as a plain non-empty string here for that reason. See `open_questions` for the IDE prefix, which the JSDoc lists and mapping/owasp-map.json does not."
|
|
]
|
|
},
|
|
|
|
"type": "object",
|
|
"$ref": "#/$defs/finding",
|
|
|
|
"$defs": {
|
|
"severity": {
|
|
"description": "The finding severity vocabulary. Ordered most to least severe; both seed implementations use these five values.",
|
|
"type": "string",
|
|
"enum": ["critical", "high", "medium", "low", "info"]
|
|
},
|
|
|
|
"owasp": {
|
|
"description": "Taxonomy anchor(s) for the finding, as ONE string. Not an array, and not a single code: when a finding belongs to several categories the codes are joined with a comma and a space, e.g. `MCP03, MCP06`. Resolved against mapping/owasp-map.json. `null` when the finding carries no anchor.",
|
|
"type": ["string", "null"],
|
|
"minLength": 1,
|
|
"$comment": "Measured across the seed runtime's scanners at b0de0ca: 31 distinct values over 157 emission sites, of which 13 values (21 sites) carry more than one code. The separator is `, ` at every one of them - no bare comma, no other spelling. Four values mix TAXONOMIES inside a single string (`LLM01, ASI01`, `LLM01, ASI02`, `LLM03, ASI04`, `LLM06, ASI02`), so the field carries no discriminator saying which taxonomy a code belongs to. A consumer cannot resolve `LLM06, ASI02` against mapping/owasp-map.json without splitting on `, ` first and then dispatching per code prefix. See open_questions and known_lossiness.owasp-tag-not-split.",
|
|
"multi_value": {
|
|
"separator": ", ",
|
|
"split_rule": "Split on `, ` to obtain individual codes. Each code independently matches ^(?:LLM|ASI|AST|MCP)[0-9]{1,2}$ in the seed runtime; the code prefix, not the field, identifies the taxonomy.",
|
|
"taxonomy_discriminator": "absent by design in the seed runtime"
|
|
}
|
|
},
|
|
|
|
"findingId": {
|
|
"description": "Identifier assigned by the producer. Format `DS-<scanner prefix>-<counter>`, where the counter is zero-padded to at least three digits.",
|
|
"type": "string",
|
|
"pattern": "^DS-[A-Za-z]+-[0-9]{3,}$",
|
|
"$comment": "Built at output.mjs:34 as `DS-${opts.scanner}-${String(n).padStart(3, '0')}` from a process-global counter. Two consequences a consumer must not design around: the id is NOT stable across runs (it depends on emission order within a process), and it is NOT unique across processes. `padStart(3)` sets a minimum, not a maximum - a run emitting more than 999 findings produces four-digit counters, which is why the pattern is `{3,}` and not `{3}`."
|
|
},
|
|
|
|
"finding": {
|
|
"type": "object",
|
|
"required": ["id", "scanner", "title", "severity"],
|
|
"additionalProperties": false,
|
|
"$comment": "Closed in 0.2.0. `finding()` in output.mjs returns an object literal with exactly these ten keys and no spread, so a finding under this contract carries nothing else. The five optional-in-practice keys are emitted as `null` rather than omitted (`opts.x || null`), so a serialised seed finding always has all ten keys present. `description`, `severity`, `title` and `scanner` are the exception: they are assigned straight from `opts` with no fallback, so if a caller omits one the key is `undefined` and disappears from the JSON entirely. That is why they are typed non-nullable and required rather than nullable. Reproduced against the real producer: `finding({scanner, severity, title})` with no `description` returns ten keys in memory and serialises to nine.",
|
|
"properties": {
|
|
"id": { "$ref": "#/$defs/findingId" },
|
|
"scanner": {
|
|
"description": "Identifier of the detector that produced the finding. Forms the first segment of the SARIF rule id.",
|
|
"type": "string",
|
|
"minLength": 1
|
|
},
|
|
"title": {
|
|
"description": "Short human-readable name of the detected class. Forms the second segment of the SARIF rule id, and is the fallback for both `description` and the SARIF result message.",
|
|
"type": "string",
|
|
"minLength": 1
|
|
},
|
|
"severity": { "$ref": "#/$defs/severity" },
|
|
"description": {
|
|
"description": "Longer explanation. Falls back to `title` wherever it is absent.",
|
|
"type": "string"
|
|
},
|
|
"recommendation": {
|
|
"description": "What to do about it. Carried through to SARIF as a result property, not as part of the message. Emitted as `null` when the producer has none.",
|
|
"type": ["string", "null"]
|
|
},
|
|
"owasp": { "$ref": "#/$defs/owasp" },
|
|
"file": {
|
|
"description": "Path the finding is anchored to. A finding whose `file` is `null` has no SARIF location at all.",
|
|
"type": ["string", "null"]
|
|
},
|
|
"line": {
|
|
"description": "1-indexed line within `file`. Silently dropped if `file` is absent — see `known_lossiness`. Emitted as `null` when the producer has none; note that `opts.line || null` also turns a literal 0 into `null`, which is harmless here only because line numbers are 1-indexed.",
|
|
"type": ["integer", "null"],
|
|
"minimum": 1
|
|
},
|
|
"evidence": {
|
|
"description": "Redacted excerpt supporting the finding. Emitted as `null` when the producer has none.",
|
|
"type": ["string", "null"]
|
|
}
|
|
}
|
|
},
|
|
|
|
"scannerResult": {
|
|
"description": "One scanner's contribution to an envelope.",
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": {
|
|
"findings": {
|
|
"type": "array",
|
|
"items": { "$ref": "#/$defs/finding" }
|
|
}
|
|
}
|
|
},
|
|
|
|
"envelope": {
|
|
"description": "The multi-scanner container the SARIF profile is generated from. A missing or empty `scanners` map is legal and yields an empty SARIF run rather than an error.",
|
|
"type": "object",
|
|
"additionalProperties": true,
|
|
"properties": {
|
|
"scanners": {
|
|
"type": "object",
|
|
"additionalProperties": { "$ref": "#/$defs/scannerResult" }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
|
|
"profiles": {
|
|
"$comment": "Profiles are mapping rules, not JSON Schema keywords — a validator ignores this key, so the file stays a usable schema while carrying the mappings the plan requires alongside it.",
|
|
|
|
"sarif": {
|
|
"status": "normative",
|
|
"spec": "SARIF 2.1.0 (OASIS)",
|
|
"spec_url": "https://docs.oasis-open.org/sarif/sarif/v2.1.0/",
|
|
"schema_url": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json",
|
|
"sarif_version": "2.1.0",
|
|
"tool": {
|
|
"name": "llm-security",
|
|
"information_uri": "https://git.fromaitochitta.com/open/llm-security",
|
|
"version_default": "6.0.0",
|
|
"$comment": "`name` and `information_uri` identify the SEED implementation. A different consumer emitting SARIF under this profile MUST substitute its own tool identity — copying these would attribute its findings to llm-security."
|
|
},
|
|
"external_dependencies": "none",
|
|
|
|
"rule_id": {
|
|
"template": "{scanner}/{slug(title)}",
|
|
"slug": {
|
|
"steps": [
|
|
"replace every run of whitespace with a single `-`",
|
|
"lowercase"
|
|
],
|
|
"pattern_applied": "\\s+",
|
|
"replacement": "-"
|
|
}
|
|
},
|
|
|
|
"severity_to_level": {
|
|
"critical": "error",
|
|
"high": "error",
|
|
"medium": "warning",
|
|
"low": "note",
|
|
"info": "note"
|
|
},
|
|
"severity_to_level_default": "note",
|
|
"severity_to_level_note": "The mapping is many-to-one and NOT invertible: `critical` and `high` both become `error`, `low` and `info` both become `note`. A consumer that round-trips a finding through SARIF loses the distinction. Findings, not SARIF, are the interchange format here.",
|
|
|
|
"rules_array": {
|
|
"keyed_by": "rule_id",
|
|
"first_wins": true,
|
|
"first_wins_note": "The rule is built from the FIRST finding carrying a given rule id, so `fullDescription` reflects that finding's `description`. Each individual result keeps its own message, so nothing is lost at result level — but two findings sharing a title and differing in description produce one rule text.",
|
|
"fields": {
|
|
"id": "rule_id",
|
|
"name": "title",
|
|
"shortDescription.text": "title",
|
|
"fullDescription.text": "description, falling back to title",
|
|
"defaultConfiguration.level": "severity_to_level[severity]",
|
|
"properties.tags": "[owasp] when present, otherwise []"
|
|
}
|
|
},
|
|
|
|
"result": {
|
|
"fields": {
|
|
"ruleId": "rule_id",
|
|
"ruleIndex": "index into runs[0].tool.driver.rules",
|
|
"level": "severity_to_level[severity]",
|
|
"message.text": "description, falling back to title",
|
|
"properties.tags": "[owasp] — omitted entirely when owasp is absent",
|
|
"properties.recommendation": "recommendation — omitted when absent",
|
|
"locations": "single-element array, present only when `file` is present"
|
|
},
|
|
"location_shape": {
|
|
"physicalLocation.artifactLocation.uri": "file",
|
|
"physicalLocation.region.startLine": "line — added only when `line` is present"
|
|
}
|
|
},
|
|
|
|
"envelope_traversal": "Findings are collected across every value of `envelope.scanners`, in object-iteration order, and flattened into one run. Scanner identity survives only inside each finding's own `scanner` field.",
|
|
"runs": "exactly one"
|
|
},
|
|
|
|
"jsonl": {
|
|
"status": "not applicable",
|
|
"$comment": "Closed 2026-08-09 as NOT APPLICABLE rather than unspecified, and the distinction is the point: `unspecified` would say a profile exists and has not been written down, which would be false. No finding-JSONL exists in the seed runtime. Findings are emitted only inside a single JSON envelope (output.mjs:140, `envelope()`), never line by line, so there is nothing for a JSONL finding profile to describe. A consumer wanting newline-delimited findings is defining a new format, not conforming to an existing one, and MUST NOT cite this schema as the authority for its line shape.",
|
|
"resolved_from": "llm-security, coord reply 2026-08-09, confirmed against the module at b0de0ca",
|
|
"not_to_be_confused_with": [
|
|
{
|
|
"what": "scanners/lib/audit-trail.mjs",
|
|
"is": "The one module in llm-security that does write JSONL (`appendFileSync(path, JSON.stringify(entry) + '\\n')`). It writes AUDIT EVENTS, not findings, under a different schema entirely: timestamp, session_id, event_type, severity, source, details, owasp, action_taken.",
|
|
"why_it_matters": "Its `owasp` field is an ARRAY of strings. The finding contract's `owasp` is a single comma-joined STRING. Two shapes, two types, one field name, in the same repository. A consumer that reads audit JSONL and finding JSON through one code path will be wrong about one of them.",
|
|
"activation": "Driven by the policy key `audit.log_path`; a no-op without it."
|
|
},
|
|
{
|
|
"what": "/tmp/llm-security-session-*.jsonl",
|
|
"is": "Hook session state. A third format again, and not a findings stream."
|
|
}
|
|
]
|
|
}
|
|
},
|
|
|
|
"known_lossiness": [
|
|
{
|
|
"id": "rule-id-collision",
|
|
"claim": "Two titles that differ only in letter case, or only in which whitespace runs separate their words, produce ONE rule id — silently merging two classes into one entry in the SARIF rules array. The merged rule carries the FIRST finding's description; each result keeps its own message, so nothing is lost at result level.",
|
|
"does_not_apply_to": "Punctuation. The slug lowercases and collapses whitespace and nothing else, so `Zero-width carrier` and `Zero-width carrier!` stay distinct ids.",
|
|
"verified": "`A B` + `a b` -> 1 rule / 2 results, rule fullDescription `first`, result messages [`first`,`second`]. `Zero-width carrier` + `Zero-width carrier!` -> ids [`sc/zero-width-carrier`, `sc/zero-width-carrier!`]."
|
|
},
|
|
{
|
|
"id": "line-without-file-dropped",
|
|
"claim": "A finding carrying `line` but no `file` produces no SARIF location at all; the line number is discarded without warning.",
|
|
"verified": "`{line: 9}` with no `file` -> `result.locations` is absent."
|
|
},
|
|
{
|
|
"id": "severity-not-invertible",
|
|
"claim": "critical and high both map to `error`; low and info both map to `note`. A finding round-tripped through SARIF cannot recover its original severity.",
|
|
"verified": "[critical, high, medium, low, info] -> [error, error, warning, note, note]."
|
|
},
|
|
{
|
|
"id": "owasp-tag-not-split",
|
|
"claim": "A multi-code `owasp` value becomes ONE SARIF tag, not one per code. The formatter writes `tags: [f.owasp]` at sarif-formatter.mjs:49 and :90, so a finding anchored to `LLM06, ASI02` is tagged with the literal string `\"LLM06, ASI02\"`. No SARIF consumer filtering on the tag `LLM06` will match it, and the finding is effectively untagged for every taxonomy it claims.",
|
|
"scope": "Affects the 13 multi-code values emitted at 21 sites in the seed runtime at b0de0ca, four of which also mix taxonomies.",
|
|
"verified": "Reproduced, not only read. A finding built by the real `finding()` with `owasp: 'LLM06, ASI02'` and run through the real `toSARIF()` at b0de0ca yields `properties.tags` of `[\"LLM06, ASI02\"]` on BOTH the rule and the result - one tag, not two.",
|
|
"not_fixed_here": "This is consumer behaviour in llm-security, not data. v0.1.0 preserves behaviour; the defect is recorded and reported rather than corrected in this repository."
|
|
}
|
|
],
|
|
|
|
"open_questions": [
|
|
"Required-field set. The producer enforces nothing - `finding()` validates no argument, so a caller omitting `severity` emits a finding with `severity: undefined`. This schema requires `id`, `scanner`, `title` and `severity` by design. Adopting it as a validator inside llm-security could therefore reject findings that runtime currently emits. That is a behaviour change and belongs in llm-security's own phase, decided against its own tests - not here.",
|
|
"Cross-runtime field names. The Python guard's Finding (report.py:53) carries `label`, `severity`, `source`, `detector`, `count`, `offset`, `evidence` and `owasp`; only `severity`, `evidence` and `owasp` are common with the Node shape, and `owasp` is common in name only - the guard has not been shown to join multiple codes into one string. A single contract two runtimes conform to needs either a rename in one of them or a declared alias table. Deferred to v0.2.0 of the REPOSITORY contract with the decision stated, rather than settled silently here.",
|
|
"Taxonomy discriminator on `owasp`. The field carries codes from four taxonomies in one comma-joined string with nothing saying which is which, and mapping/owasp-map.json now records that one of those four (AST) is not an OWASP taxonomy at all. Splitting on the code prefix works today only because the four prefixes happen to be distinct. This is the seam most likely to break a second runtime.",
|
|
"The `IDE` scanner prefix. The producer's JSDoc lists seventeen prefixes - UNI, ENT, PRM, DEP, TNT, GIT, NET, MEM, SCR, WFL, TRG, SIG, AST, TFA, IDE, MCI, PST - while all four maps in mapping/owasp-map.json are keyed on sixteen, without IDE. A finding from the IDE scanner therefore has an id and a scanner prefix but no taxonomy mapping in any of the four maps. Reported rather than patched: adding a key to those maps would be inventing detection data."
|
|
]
|
|
}
|