open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

feat(calibration): add calibration.json, marked transcribed-only

Browse source 
The risk-score tier constants, verdict thresholds, risk-band cutoffs and
posture grade thresholds. Constants only — the log scaling and the if/else
chains that consume them are engine and stay in the consumer.

This file is the exception in this repository and is marked as such in its
own verification block. Every other data file here was rebuilt from its
commons JSON and diffed against an imported module. This one arrived as a
human-written PROSE SUMMARY of severity.mjs, so there was nothing to import
and nothing to diff. verified: false, with the two skipped checks named
rather than a caveat attached to a pass.

The consequence inverts this repository's central rule and the file says so:
for every other file a disagreeing consumer is wrong; for this one, until the
module arrives in executable form, a disagreement is not automatically the
consumer's bug.

What COULD be checked was: bands are contiguous and non-overlapping across
0-100, the BLOCK trigger (65) is exactly the Critical band floor, the WARNING
trigger (15) is exactly the Medium band floor, and the per-tier reachable
minima recompute exactly (80/48/20/4, exact because log2(2) is 1).

Corrects README and the extraction plan in the same commit: both promised
entropy floors, scan caps and disposition ranks. None arrived — 0 occurrences
each across the whole dump. Named in the file under not_supplied so the
absence is visible rather than inferred, and both rows now describe what is
actually present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 21:14:13 +02:00
commit 8ee297c585
3 changed files with 210 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -55,7 +55,7 @@ and keeping it that small is the point.
| [`signatures/secret-egress.json`](signatures/secret-egress.json) | Credential and token shapes that must never leave a machine, in a portable regex dialect. |
| [`signatures/malware-signatures.json`](signatures/malware-signatures.json) | Signature set for the malicious-code class (`SIG`). |
| [`signatures/active-content.json`](signatures/active-content.json) | Active content that renders or fetches on its own — Markdown images, links, reference definitions and autolinks, `data:` URIs, active HTML. The EchoLeak class. |
| [`calibration/calibration.json`](calibration/calibration.json) | The numbers a detector must not invent: entropy floors, scan caps, disposition ranks. |
| [`calibration/calibration.json`](calibration/calibration.json) | The numbers a detector must not invent: risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. Transcribed from a prose summary, not differentially verified — the file says so itself. |
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract, plus the SARIF and JSONL output profiles. |
| [`spec/decode-pipeline.md`](spec/decode-pipeline.md) | **Normative.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input; this is the file that stops that. |

176
calibration/calibration.json Normal file
View file

@ -0,0 +1,176 @@
{
"version": "0.1.0",
"id": "calibration",
"description": "The numbers a detector must not invent: the risk-score tier constants, the verdict thresholds, the risk-band cutoffs and the posture grade thresholds. Constants only. The formulas that consume them - the log scaling, the if/else chains - are engine code and stay in the consumer.",
"$comment": "READ verification.status BEFORE RELYING ON THIS FILE. Unlike every other data file in this repository, this one was NOT delivered as source code. It arrived as an operator prose summary of llm-security/scanners/lib/severity.mjs inside coord dump 2/2 on 2026-08-09, so there was nothing to import and nothing to diff against. Every other file here carries a differential result; this one carries a transcription and says so. It is the weakest evidence in the repository.",
"provenance": {
"source_repo": "llm-security",
"source_files": [
"scanners/lib/severity.mjs"
],
"source_exports": [
"riskScore",
"verdict",
"riskBand",
"gradeFromPassRate"
],
"source_delivery": "operator dump 2/2, coord message from llm-security, 2026-08-09 - PROSE SUMMARY, not source code",
"source_commit": "unknown - not supplied with the dump",
"verified": false
},
"verification": {
"status": "transcribed-only",
"$comment": "No differential check was possible. The producing module was not supplied in any executable form, so the constants below could not be rebuilt from this file and compared against a running implementation, which is the check every other file in this repository passed. What HAS been checked is internal: the JSON is well-formed, the band cutoffs are contiguous and non-overlapping, and the band boundary agrees with the BLOCK threshold.",
"checks_not_run": [
"rebuild-from-commons-and-diff-against-source (no importable source)",
"differential scoring over a corpus (the formulas are engine and were not supplied in runnable form)"
],
"consequence": "A consumer calibrating against this file matches numbers that a human transcribed. Until the module is supplied, a disagreement between a consumer and this file is NOT automatically the consumer's bug - which is the opposite of the rule that holds for the rest of this repository."
},
"not_supplied": {
"$comment": "This repository's README and extraction plan originally described this file as holding entropy floors, scan caps and disposition ranks. None of those arrived. Searched across the whole delivered dump: 'entropy' 0 occurrences, 'disposition' 0, 'rank' 0, 'floor' 0. They are named here so that their absence is visible in the file itself rather than inferred from what is missing. The README row has been corrected to describe what is present.",
"items": [
"entropy floors",
"scan caps",
"disposition ranks"
]
},
"risk_score": {
"$comment": "Severity-dominated: the highest severity present picks the tier, and the count only moves the score within that tier. The engine formula is `base + min(increment_cap, log2(count + 1) * log2_multiplier)`. That formula is NOT data and does not move here; it is written out so each constant below has a meaning. `stated_range` is the dump's own wording and describes base..base+increment_cap. `reachable_minimum` is the score at count = 1, which is exact for every tier because log2(2) = 1 - it is arithmetic on the supplied constants, not a new claim. Whether intermediate values are rounded, floored or kept fractional was not supplied.",
"formula": "base + min(increment_cap, log2(count + 1) * log2_multiplier)",
"formula_is_engine": true,
"no_findings_score": 0,
"info_is_scoring_inert": true,
"tiers": [
{
"id": "critical-present",
"severity": "critical",
"base": 70,
"increment_cap": 25,
"log2_multiplier": 10,
"stated_range": "~70-95",
"reachable_minimum": 80
},
{
"id": "high-only",
"severity": "high",
"base": 40,
"increment_cap": 25,
"log2_multiplier": 8,
"stated_range": "~40-65",
"reachable_minimum": 48
},
{
"id": "medium-only",
"severity": "medium",
"base": 15,
"increment_cap": 20,
"log2_multiplier": 5,
"stated_range": "~15-35",
"reachable_minimum": 20
},
{
"id": "low-only",
"severity": "low",
"base": 1,
"increment_cap": 10,
"log2_multiplier": 3,
"stated_range": "~1-11",
"reachable_minimum": 4
}
]
},
"verdict": {
"$comment": "Evaluated in order; the first rule that holds wins. Two independent triggers per verdict: a severity count, or the numeric score.",
"rules": [
{
"verdict": "BLOCK",
"if_critical_count_at_least": 1,
"or_score_at_least": 65
},
{
"verdict": "WARNING",
"if_high_count_at_least": 1,
"or_score_at_least": 15
},
{
"verdict": "ALLOW",
"otherwise": true
}
]
},
"risk_band": {
"$comment": "Inclusive integer cutoffs over the 0-100 score.",
"bands": [
{
"name": "Low",
"min": 0,
"max": 14
},
{
"name": "Medium",
"min": 15,
"max": 39
},
{
"name": "High",
"min": 40,
"max": 64
},
{
"name": "Critical",
"min": 65,
"max": 84
},
{
"name": "Extreme",
"min": 85,
"max": 100
}
]
},
"grade": {
"$comment": "Posture grade from a pass rate, evaluated in this order - F is tested FIRST, so a run with three or more critical findings grades F regardless of its pass rate. `critCount` and `failsInCritCats` are the source's own names; what counts as a critical category was not supplied and is not invented here.",
"source_function": "gradeFromPassRate",
"unresolved_terms": [
"critCount",
"failsInCritCats"
],
"rules": [
{
"grade": "F",
"pass_rate_below": 0.33,
"or_crit_count_at_least": 3
},
{
"grade": "A",
"pass_rate_at_least": 0.89,
"and_fails_in_crit_cats": 0,
"and_crit_count": 0
},
{
"grade": "B",
"pass_rate_at_least": 0.72,
"and_crit_count": 0
},
{
"grade": "C",
"pass_rate_at_least": 0.56
},
{
"grade": "D",
"pass_rate_at_least": 0.33
},
{
"grade": "F",
"otherwise": true
}
]
},
"consistency_notes": [
"The risk bands are contiguous and non-overlapping across 0-100, with no gap and no shared value.",
"The BLOCK score threshold (65) is exactly the lower bound of the Critical band, so the numeric BLOCK trigger and the Critical band begin at the same score.",
"The WARNING score threshold (15) is exactly the lower bound of the Medium band.",
"A low-only result cannot reach WARNING by score: that tier's ceiling is 11, below the threshold of 15. A medium-only result cannot reach BLOCK by score: that tier's ceiling is 35, below 65. Both follow from the supplied constants by arithmetic."
]
}

34
docs/extraction-plan.md
View file

@ -52,7 +52,7 @@ data, never from recollection or inference.
| `signatures/secret-egress.json` | `hooks/scripts/pre-edit-secrets.mjs` — `SECRET_PATTERNS`, the **18-entry hook table**, NOT the PCRE-flavored agent-consumed variant in `knowledge/secrets-patterns.md`. *(Corrected 2026-08-09: this row originally named `knowledge/secrets-patterns.md` as the source file. The delivered dump named `pre-edit-secrets.mjs` and stated explicitly that the two are different files. The row now names the file that was actually delivered.)* |
| `signatures/malware-signatures.json` | `knowledge/signatures.json` (the SIG scanner) |
| `signatures/active-content.json` | currently only in a guard repo's `active_content.py`. If unavailable: stub with a version field and a TODO naming the source |
| `calibration/calibration.json` | `scanners/lib/severity.mjs` — thresholds + scanner caps |
| `calibration/calibration.json` | `scanners/lib/severity.mjs` — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. *(Corrected 2026-08-09: this row said "thresholds + scanner caps" and the README said "entropy floors, scan caps, disposition ranks". No scan cap, entropy floor or disposition rank was delivered — searched across the whole dump: 0 occurrences each. Both rows now describe what arrived.)* |
| `mapping/owasp-map.json` | `scanners/lib/severity.mjs` — `OWASP_MAP` (+ 3 sibling maps in the same file) |
| `schema/finding.schema.json` | modelled on `scanners/lib/sarif-formatter.mjs`'s SARIF shape |
| `conformance/` | union of the guard repo's `coverage.py` matrix (126 classes + 4 gaps-must-hold) and `llm-security/examples/` |
@ -222,6 +222,38 @@ and the rest are reproduced as opaque keys. Deliberately not inherited: the dump
fallback map inside one of `llm-security`'s own agent definitions covers only part of the
prefix set — that is drift in the consumer, and the complete 16-prefix set is what moves here.
### `calibration/calibration.json` — transcribed 2026-08-09
Source: `llm-security/scanners/lib/severity.mjs`, delivered as a **prose summary** inside
operator dump 2/2 — not as source code.
**This file is the exception in this repository, and it is marked as such in its own
`verification` block.** Every other data file here was rebuilt from its commons JSON and
diffed against an imported module. There was nothing to import here: the constants arrived
as human-written prose describing the module. The differential check was therefore not run
— not "passed with caveats", not run — and the file records `verified: false` with the
specific checks that were skipped.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present | `python3 -m json.tool` | pass |
| Risk bands are contiguous and non-overlapping | compared each band's `max + 1` to the next band's `min` across 0–100 | contiguous, no gap, no shared value |
| Band boundaries agree with the verdict thresholds | compared the BLOCK and WARNING score triggers to the band lower bounds | BLOCK 65 = Critical band min; WARNING 15 = Medium band min |
| `reachable_minimum` is arithmetic, not a new claim | recomputed `base + min(cap, log2(2) * mult)` independently for all four tiers | 80 / 48 / 20 / 4 — exact, because `log2(2)` is exactly 1 |
| Rebuild-from-commons and diff against source | **not run** — no importable source | — |
| Differential scoring over a corpus | **not run** — the formulas are engine and were not supplied in runnable form | — |
Recorded in the file as `not_supplied`, so the absence is visible rather than inferred: the
README and this plan both promised **entropy floors, scan caps and disposition ranks**. None
were delivered. Searched across the entire dump message: `entropy` 0 occurrences,
`disposition` 0, `rank` 0, `floor` 0. Both descriptions have been corrected to name what
arrived rather than what was expected.
The consequence is stated in the file and is worth repeating here, because it inverts this
repository's central rule: for every other file, a consumer that disagrees is wrong. For this
one, until the module is supplied in executable form, a disagreement is not automatically the
consumer's bug.
## Definition of done for v0.1.0
1. Repository initialized, Forgejo remote `open/llm-security-commons`, MIT, `STATE.md`