feat(llm-security)!: v8 Phase 3 complete - riskScoreV1, posture heuristic, docs
Closes Phase 3 (B11) of the v8.0.0 plan. Three parts, all with the failing
test written first.
riskScoreV1 removed. scanners/lib/severity.mjs drops riskScoreV1() and its
SEVERITY_WEIGHTS_V1 table - @deprecated since v7.0.0, kept for diff/comparison,
zero callers in code or tests (re-verified, not taken from the plan). The v1
weights are recorded in CHANGELOG so an old score stays re-derivable. riskScore
(v2) is untouched; a test pins that one critical still lands in the 70-95 tier
and that 50 lows score below it, which is exactly the case v1 collapsed to 100.
Posture category 12 no longer keys off an identifier name. The check was
/TRIFECTA_MODE/i over the session-guard source, which measured what a constant
was CALLED rather than whether enforcement was configurable. With the env-var
gone, that regex would have dropped every correctly-migrated project from PASS
to PARTIAL - the gate punishing the migration it exists to encourage. It now
matches getPolicyValue('trifecta', 'mode', ...) and still accepts a pre-v8
vendored guard reading the old env-var, because a third-party project carries
its own hook copy and is equally configurable either way; the evidence line
says which of the two was found. The PARTIAL finding recommended setting an
env-var that v8 ignores; it now names the policy key. The grade-a fixture hook
moves to the policy-era form.
Two never-implemented env-vars deleted from the docs. LLM_SECURITY_SCR_OFFLINE
(ci-cd-guide) and LLM_SECURITY_OFFLINE (supply-chain-attack example) were
documented as OSV.dev / npm-audit kill-switches. No code has ever read either -
verified by grep across scanners, hooks and scripts, which finds them only in
markdown. A promised kill-switch that does nothing is worse than a documented
absence: it is trusted precisely when the run is meant to be air-gapped. The
docs now say there is none and that egress must be blocked at the network
layer. The LLM_SECURITY_AUDIT_* wildcard is narrowed to the one real key.
Docs. Migration section in README + CHANGELOG with the env-var -> policy-key
table, the detection commands (env + shell rc + .envrc + workflows), and the
explicit warning that a removed variable is now INERT rather than an error -
which is the failure mode that loses a project its configuration silently. The
hardening-guide env table splits into surviving vars and a removed-vars
migration table; its "promote to block" runbook named two variables that no
longer exist. Also swept: CLAUDE.md hook table, scanner-reference, ci-cd-guide,
both lethal-trifecta example docs, mitigation-matrix, injection-research.
Test counts in README/CLAUDE.md synced 2034 -> 2045.
Suite 2045 tests, 0 fail (2039 + 4 posture-trifecta + 2 riskScoreV1). The two
known parallel-load flakes did not recur this run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez
This commit is contained in:
parent
b6af9b46df
commit
fdec4b36ad
16 changed files with 333 additions and 62 deletions
59
README.md
59
README.md
|
|
@ -199,13 +199,13 @@ Hooks run on every operation — no commands needed. They activate the moment th
|
|||
|
||||
| Hook | Event | What it does |
|
||||
|------|-------|--------------|
|
||||
| **Prompt injection scan** | UserPromptSubmit | Blocks direct injection (override instructions, spoofed system headers, identity redefinition) and warns on subtle signals (leetspeak, homoglyphs, zero-width chars, multi-language). Decodes obfuscated payloads (Unicode Tag, hex, URL, base64, rot13) before matching. Mode: `LLM_SECURITY_INJECTION_MODE=block\|warn\|off` (default block) |
|
||||
| **Prompt injection scan** | UserPromptSubmit | Blocks direct injection (override instructions, spoofed system headers, identity redefinition) and warns on subtle signals (leetspeak, homoglyphs, zero-width chars, multi-language). Decodes obfuscated payloads (Unicode Tag, hex, URL, base64, rot13) before matching. Mode: policy key `injection.mode` = `block\|warn\|off` (default block) |
|
||||
| **Secret detection** | Edit, Write | Blocks AWS keys, Azure tokens, GitHub PATs, npm tokens, PEM keys, database URLs, Bearer tokens, and 30+ other secret patterns |
|
||||
| **Path guarding** | Write | Blocks writes to `.env*` (multi-segment-suffix-safe), `.ssh/`, `.aws/`, `.gnupg/`, credentials files, hook scripts, `/etc/`, `settings.json` |
|
||||
| **Destructive commands** | Bash | Blocks `rm -rf /`, `chmod 777`, pipe-to-shell, fork bombs, eval-with-substitution, T8 base64-pipe-shell loaders. Bash-normalize T1-T9 collapses obfuscation (empty quotes, `${IFS}`, ANSI-C hex, process substitution, eval-via-variable) before pattern matching |
|
||||
| **Supply-chain guardrail** | Bash | Blocks known-compromised npm/pip packages, Levenshtein typosquats, age-gated installs (<72 h), OSV.dev CVE checks. Covers npm, pip, brew, docker, go, cargo, gem. v7.3.0: npm scope-hop typosquat advisory (E13) — `@evil/lodash`-class catches scope-jumping when the unscoped name matches a popular package |
|
||||
| **Output verification** | All tools (post) | Advisory: scans ALL tool output for indirect injection (LLM01) and HITL traps (DeepMind kat. 6). Bash-specific: leaked secrets, unexpected URLs, oversized MCP responses. v7.3.0: per-update MCP description drift AND cumulative drift vs sticky baseline (E14) — slow-burn rug-pulls that stay under per-update thresholds but cumulatively diverge ≥25% emit `mcp-cumulative-drift` MEDIUM |
|
||||
| **Session guard** | All tools (post) | Advisory: monitors tool-call sequences for the lethal trifecta (untrusted input + sensitive read + exfiltration sink). 20-call sliding window + 100-call long-horizon window. Mode: `LLM_SECURITY_TRIFECTA_MODE=block\|warn\|off`. Sub-agent delegation tracking via Task/Agent tools surfaces escalation-after-input as a separate advisory |
|
||||
| **Session guard** | All tools (post) | Advisory: monitors tool-call sequences for the lethal trifecta (untrusted input + sensitive read + exfiltration sink). 20-call sliding window + 100-call long-horizon window. Mode: policy key `trifecta.mode` = `block\|warn\|off`. Sub-agent delegation tracking via Task/Agent tools surfaces escalation-after-input as a separate advisory |
|
||||
| **Pre-compact scan** | PreCompact | Scans transcript tail (max 512 KB, <500 ms) for injection patterns + credentials before context compaction. Prevents poisoned content from surviving in compact form. Mode: `LLM_SECURITY_PRECOMPACT_MODE=block\|warn\|off` (default warn) |
|
||||
| **Update check** | UserPromptSubmit | Checks for newer plugin versions max 1× / 24 h, cached. Disable: `LLM_SECURITY_UPDATE_CHECK=off` |
|
||||
|
||||
|
|
@ -329,9 +329,9 @@ Average ~69 %. Strongest at prompt injection (95 % with input + output scanning
|
|||
| **Compliance mapping** | EU AI Act (Art. 9, 15, 17), NIST AI RMF (Map / Measure / Manage / Govern), ISO 42001 (Annex A), MITRE ATLAS techniques. Posture categories 14-16 assess readiness |
|
||||
| **Norwegian context** | Datatilsynet DPIA-for-AI guidance, NSM basic security principles, Digitaliseringsdirektoratet — relevant for Norwegian public-sector deployments |
|
||||
| **SARIF 2.1.0 output** | `--format sarif` on scan / deep-scan produces OASIS SARIF for CI/CD ingestion (GitHub Advanced Security, Azure DevOps, SonarQube) |
|
||||
| **Structured audit trail** | JSONL events with ISO 8601 timestamps and OWASP category tags (`LLM_SECURITY_AUDIT_*` env-vars or `audit.log_path` policy key) — SIEM-ready |
|
||||
| **Structured audit trail** | JSONL events with ISO 8601 timestamps and OWASP category tags (`audit.log_path` policy key) — SIEM-ready |
|
||||
| **AI-BOM** | CycloneDX 1.6 BOM for AI components — models, MCP servers, plugins, knowledge files, hooks (`llm-security audit-bom <target>`) |
|
||||
| **Policy-as-code** | `.llm-security/policy.json` ships hook configuration with the team. v7.3.0 (D3) adds a one-time-per-process stderr deprecation line when both an env-var AND its `policy.json` equivalent are explicitly set; env still wins through the v7.x runway, env reads removed in v8.0.0. Suppress noise with `LLM_SECURITY_DEPRECATION_QUIET=1` |
|
||||
| **Policy-as-code** | `.llm-security/policy.json` ships hook configuration with the team — the only source for injection, trifecta, and audit configuration since v8.0.0 removed the overlapping env-vars |
|
||||
| **Standalone CLI** | `node bin/llm-security.mjs scan <target>` — runs scanners without Claude Code. Subcommands: `scan`, `deep-scan`, `posture`, `audit-bom`, `benchmark`. Schrems II compatible in default offline mode (optional OSV.dev enrichment is the only network call and is opt-in) |
|
||||
| **CI/CD integration** | `--fail-on <severity>` for threshold-based exit codes, `--compact` for one-liner output. Templates for GitHub Actions, Azure DevOps, GitLab CI in `ci/`. Guide: `docs/ci-cd-guide.md` |
|
||||
|
||||
|
|
@ -419,14 +419,14 @@ These gaps are surfaced advisorily through `/security threat-model` and `/securi
|
|||
|
||||
This is a **solo open-source project in stabilization mode** as of 2026-05-01.
|
||||
The current feature set (5 frameworks, 22 scanners, 9 hooks, 6 agents,
|
||||
20 commands, 23 knowledge files, 2034+ tests including a dedicated end-to-end suite) is the natural plateau for
|
||||
20 commands, 23 knowledge files, 2045+ tests including a dedicated end-to-end suite) is the natural plateau for
|
||||
what a deterministic + advisory plugin can defend against without crossing
|
||||
into commercial-grade territory. Going forward, work focuses on:
|
||||
|
||||
- **Bug fixes** and security patches
|
||||
- **Compatibility** with new Claude Code releases
|
||||
- **Knowledge-base refresh** (OWASP updates, new published research, new attack patterns)
|
||||
- **Deprecation cleanup** — v8.0.0 removes the `LLM_SECURITY_*` env vars and `riskScoreV1` constant deprecated in v7.3.0
|
||||
- **Deprecation cleanup** — v8.0.0 removed the four `LLM_SECURITY_*` mode env vars and `riskScoreV1`, both deprecated in v7.3.0 (see [Migration](#migrating-to-v800))
|
||||
- **Opportunistic small additions** that fit the existing deterministic architecture
|
||||
|
||||
## Non-goals
|
||||
|
|
@ -479,6 +479,53 @@ Prompt injection is **structurally unsolvable** with current architectures (join
|
|||
|
||||
---
|
||||
|
||||
## Migrating to v8.0.0
|
||||
|
||||
v8.0.0 removes the four `LLM_SECURITY_*` configuration env-vars deprecated in
|
||||
v7.3.0. Configuration moved to `.llm-security/policy.json`, which travels with
|
||||
the repository instead of living in whoever's shell happened to launch Claude
|
||||
Code.
|
||||
|
||||
**A removed variable is now inert.** It does not warn and it does not
|
||||
configure — a project that relied on `LLM_SECURITY_INJECTION_MODE=off` silently
|
||||
returns to the `block` default. Check for these before upgrading:
|
||||
|
||||
```bash
|
||||
env | grep '^LLM_SECURITY_\(INJECTION_MODE\|TRIFECTA_MODE\|ESCALATION_WINDOW\|AUDIT_LOG\|DEPRECATION_QUIET\)='
|
||||
grep -rn 'LLM_SECURITY_\(INJECTION_MODE\|TRIFECTA_MODE\|ESCALATION_WINDOW\|AUDIT_LOG\)' \
|
||||
~/.zshenv ~/.bashrc .envrc .github/workflows/ 2>/dev/null
|
||||
```
|
||||
|
||||
| Removed env-var | Policy key | Default |
|
||||
|-----------------|------------|---------|
|
||||
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` |
|
||||
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` |
|
||||
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` |
|
||||
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset (audit trail off) |
|
||||
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | removed with the warning it silenced |
|
||||
|
||||
Translate each one you find into `.llm-security/policy.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"injection": { "mode": "block" },
|
||||
"trifecta": { "mode": "warn", "escalation_window": 5 },
|
||||
"audit": { "log_path": "/var/log/llm-security/audit.jsonl" }
|
||||
}
|
||||
```
|
||||
|
||||
**Unaffected.** Env-vars with no policy equivalent keep working and are not
|
||||
part of this change: `LLM_SECURITY_PRECOMPACT_MODE`,
|
||||
`LLM_SECURITY_PRECOMPACT_MAX_BYTES`, `LLM_SECURITY_UPDATE_CHECK`,
|
||||
`LLM_SECURITY_MCP_CACHE_FILE`, `LLM_SECURITY_IDE_ROOTS`.
|
||||
|
||||
**Also removed:** `riskScoreV1()` in `scanners/lib/severity.mjs`, the v1
|
||||
sum-and-cap scoring formula `@deprecated` since v7.0.0 and kept for reference
|
||||
only. It had no callers in code or tests. `riskScore()` (v2, severity-dominated)
|
||||
is unchanged, so no score, band, or verdict moves.
|
||||
|
||||
---
|
||||
|
||||
## Playground (v7.6.0)
|
||||
|
||||
A single-file SPA at `playground/llm-security-playground.html` provides
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue