open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

feat(llm-security)!: v8 Phase 3 complete - riskScoreV1, posture heuristic, docs

Browse source 
Closes Phase 3 (B11) of the v8.0.0 plan. Three parts, all with the failing
test written first.

riskScoreV1 removed. scanners/lib/severity.mjs drops riskScoreV1() and its
SEVERITY_WEIGHTS_V1 table - @deprecated since v7.0.0, kept for diff/comparison,
zero callers in code or tests (re-verified, not taken from the plan). The v1
weights are recorded in CHANGELOG so an old score stays re-derivable. riskScore
(v2) is untouched; a test pins that one critical still lands in the 70-95 tier
and that 50 lows score below it, which is exactly the case v1 collapsed to 100.

Posture category 12 no longer keys off an identifier name. The check was
/TRIFECTA_MODE/i over the session-guard source, which measured what a constant
was CALLED rather than whether enforcement was configurable. With the env-var
gone, that regex would have dropped every correctly-migrated project from PASS
to PARTIAL - the gate punishing the migration it exists to encourage. It now
matches getPolicyValue('trifecta', 'mode', ...) and still accepts a pre-v8
vendored guard reading the old env-var, because a third-party project carries
its own hook copy and is equally configurable either way; the evidence line
says which of the two was found. The PARTIAL finding recommended setting an
env-var that v8 ignores; it now names the policy key. The grade-a fixture hook
moves to the policy-era form.

Two never-implemented env-vars deleted from the docs. LLM_SECURITY_SCR_OFFLINE
(ci-cd-guide) and LLM_SECURITY_OFFLINE (supply-chain-attack example) were
documented as OSV.dev / npm-audit kill-switches. No code has ever read either -
verified by grep across scanners, hooks and scripts, which finds them only in
markdown. A promised kill-switch that does nothing is worse than a documented
absence: it is trusted precisely when the run is meant to be air-gapped. The
docs now say there is none and that egress must be blocked at the network
layer. The LLM_SECURITY_AUDIT_* wildcard is narrowed to the one real key.

Docs. Migration section in README + CHANGELOG with the env-var -> policy-key
table, the detection commands (env + shell rc + .envrc + workflows), and the
explicit warning that a removed variable is now INERT rather than an error -
which is the failure mode that loses a project its configuration silently. The
hardening-guide env table splits into surviving vars and a removed-vars
migration table; its "promote to block" runbook named two variables that no
longer exist. Also swept: CLAUDE.md hook table, scanner-reference, ci-cd-guide,
both lethal-trifecta example docs, mitigation-matrix, injection-research.

Test counts in README/CLAUDE.md synced 2034 -> 2045.

Suite 2045 tests, 0 fail (2039 + 4 posture-trifecta + 2 riskScoreV1). The two
known parallel-load flakes did not recur this run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 10:25:03 +02:00
commit fdec4b36ad
16 changed files with 333 additions and 62 deletions
Download patch file Download diff file Expand all files Collapse all files

9
docs/ci-cd-guide.md
View file

@ -6,7 +6,7 @@ Integrate llm-security into your CI/CD pipeline for automated security scanning
**The standalone CLI makes zero network calls by default.** All 14 scanners operate locally on your source code using Shannon entropy analysis, regex pattern matching, AST traversal, and git log parsing. No data is transmitted to any external service.
**Exception: supply-chain-recheck** — When scanning lockfiles for known vulnerabilities, this scanner optionally queries the [OSV.dev](https://osv.dev/) batch API. This sends only package names and versions (not source code) over HTTPS. To disable: set `LLM_SECURITY_SCR_OFFLINE=1`.
**Exception: supply-chain-recheck** — When scanning lockfiles for known vulnerabilities, this scanner optionally queries the [OSV.dev](https://osv.dev/) batch API. This sends only package names and versions (not source code) over HTTPS. There is no kill-switch for this call today; block the host at the network layer if the run must be air-gapped.
**What about Claude Code integration?** The Claude Code plugin (hooks, agents, commands) uses AI models and sends data to Anthropic. These components are **not included** in the standalone CLI. When you run `npx llm-security scan`, only deterministic scanners execute.
@ -122,8 +122,11 @@ CLI flags always take precedence over policy file values.
| Variable | Description |
|----------|-------------|
| `LLM_SECURITY_SCR_OFFLINE=1` | Disable OSV.dev network calls in supply-chain-recheck |
| `LLM_SECURITY_AUDIT_LOG=<path>` | Write structured JSONL audit trail (SIEM-ready) |
| `LLM_SECURITY_PRECOMPACT_MODE` | `block\|warn\|off` for the PreCompact transcript scan (default `warn`) |
| `LLM_SECURITY_UPDATE_CHECK=off` | Disable the daily update-check HTTP call |
The audit trail moved to the policy file in v8.0.0: set `audit.log_path` in
`.llm-security/policy.json` instead of `LLM_SECURITY_AUDIT_LOG`.
## Exit Codes

2
docs/scanner-reference.md
View file

@ -27,7 +27,7 @@ AST-taint (AST) shells out to a PARSE-ONLY python3 helper (`scanners/lib/py-ast-
Utility: `node scanners/lib/fs-utils.mjs <backup|restore|cleanup|tmppath> [args]`.
Lib: `sarif-formatter.mjs` — converts scan output to OASIS SARIF 2.1.0 format. Used by `--format sarif` flag.
Lib: `audit-trail.mjs` — writes structured JSONL audit events (ISO 8601, OWASP tags, SIEM-ready). Env: `LLM_SECURITY_AUDIT_*`.
Lib: `audit-trail.mjs` — writes structured JSONL audit events (ISO 8601, OWASP tags, SIEM-ready). Enabled by the `audit.log_path` key in `.llm-security/policy.json` (v8.0.0; was `LLM_SECURITY_AUDIT_LOG`).
Lib: `policy-loader.mjs` — reads `.llm-security/policy.json` for distributable hook configuration. Includes `ci` section (`failOn`, `compact`) for CI/CD defaults. Defaults match hardcoded values.
**Standalone (8):** `posture-scanner.mjs` — deterministic posture assessment, 16 categories (incl. EU AI Act, NIST AI RMF, ISO 42001), <50ms. NOT in scan-orchestrator (meta-level, not code-level).

37
docs/security-hardening-guide.md
View file

@ -23,16 +23,33 @@ in production. Deviations are fine, but the defaults here are the tested path.
| Variable | Default | Modes |
|----------|---------|-------|
| `LLM_SECURITY_INJECTION_MODE` | `block` | `block` — exit 2 on critical/high injection patterns. `warn` — advisory via systemMessage. `off` — disables scan. |
| `LLM_SECURITY_TRIFECTA_MODE` | `warn` | `block` — exit 2 when lethal trifecta (untrusted input + sensitive data + exfiltration sink) detected. `warn` — advisory. `off` — disables. |
| `LLM_SECURITY_PRECOMPACT_MODE` | `warn` | `block` — exit 2 on findings during PreCompact. `warn` — advisory via systemMessage. `off` — disables scan. |
| `LLM_SECURITY_PRECOMPACT_MAX_BYTES` | `512000` | Tail size in bytes read from transcript for scanning. Higher values increase coverage at the cost of latency. |
| `LLM_SECURITY_UPDATE_CHECK` | `on` | `off` disables the daily update-check HTTP call. |
| `LLM_SECURITY_AUDIT_*` | unset | Audit trail configuration (destination, format, etc.) for SIEM-ready JSONL output. |
Apply env vars via shell profile, `.envrc`, or the host MDM. Do not write them
into the repository.
**Removed in v8.0.0.** Four modes moved to `.llm-security/policy.json`, which
travels with the repository instead of the operator's shell. Setting the old
variable is now inert — it neither warns nor configures.
| Removed variable | Policy key | Default |
|------------------|-----------|---------|
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` — exit 2 on critical injection patterns. `warn` — advisory via systemMessage. `off` — disables scan. |
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` — advisory. `block` — exit 2 when the lethal trifecta (untrusted input + sensitive data + exfiltration sink) is detected. `off` — disables. |
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` — calls between untrusted input and sub-agent delegation that still count as escalation-after-input. |
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset — set a path to write the SIEM-ready JSONL audit trail. |
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | Silenced the deprecation warning; removed with the warning. |
```json
{
"injection": { "mode": "block" },
"trifecta": { "mode": "warn", "escalation_window": 5 },
"audit": { "log_path": "/var/log/llm-security/audit.jsonl" }
}
```
---
## 2. Sandboxing
@ -73,14 +90,16 @@ shell; avoid root, which disables user-namespace confinement.
### 3.1 Start in warn mode
Every new integration of `llm-security` should begin with all modes set to
`warn`. This yields advisories without breaking workflow, and lets the team
calibrate false-positive rates against their actual repositories.
`warn` — `injection.mode` and `trifecta.mode` in `.llm-security/policy.json`,
`LLM_SECURITY_PRECOMPACT_MODE` in the environment. This yields advisories
without breaking workflow, and lets the team calibrate false-positive rates
against their actual repositories.
### 3.2 Promote to block after baselining
After a baseline period (typically 1-2 weeks), flip each mode to `block` in this
order: `LLM_SECURITY_INJECTION_MODE`, `LLM_SECURITY_TRIFECTA_MODE`,
`LLM_SECURITY_PRECOMPACT_MODE`. The injection hook is first because false
order: `injection.mode`, `trifecta.mode` (both in `.llm-security/policy.json`),
then `LLM_SECURITY_PRECOMPACT_MODE`. The injection hook is first because false
positives there are the most visible; blocking comes last because the others
build confidence.
@ -183,7 +202,9 @@ tier:
Verdict cutoffs (`BLOCK ≥65`, `WARNING ≥15`) are locked to the `riskBand()`
boundaries so you can't get a "BLOCK / Medium band" contradiction. The legacy
formula is kept as `riskScoreV1()` for reference only.
v1 formula was kept as `riskScoreV1()` for reference through the v7 line and
was removed in v8.0.0; see `CHANGELOG.md` for the v1 weights if you need them
to re-derive an old score.
**CI impact:** Pipelines with `--fail-on high` keep working (the severity
gate is unaffected). Pipelines with score-based thresholds need recalibration