feat(llm-security)!: v8 Phase 3 complete - riskScoreV1, posture heuristic, docs
Closes Phase 3 (B11) of the v8.0.0 plan. Three parts, all with the failing
test written first.
riskScoreV1 removed. scanners/lib/severity.mjs drops riskScoreV1() and its
SEVERITY_WEIGHTS_V1 table - @deprecated since v7.0.0, kept for diff/comparison,
zero callers in code or tests (re-verified, not taken from the plan). The v1
weights are recorded in CHANGELOG so an old score stays re-derivable. riskScore
(v2) is untouched; a test pins that one critical still lands in the 70-95 tier
and that 50 lows score below it, which is exactly the case v1 collapsed to 100.
Posture category 12 no longer keys off an identifier name. The check was
/TRIFECTA_MODE/i over the session-guard source, which measured what a constant
was CALLED rather than whether enforcement was configurable. With the env-var
gone, that regex would have dropped every correctly-migrated project from PASS
to PARTIAL - the gate punishing the migration it exists to encourage. It now
matches getPolicyValue('trifecta', 'mode', ...) and still accepts a pre-v8
vendored guard reading the old env-var, because a third-party project carries
its own hook copy and is equally configurable either way; the evidence line
says which of the two was found. The PARTIAL finding recommended setting an
env-var that v8 ignores; it now names the policy key. The grade-a fixture hook
moves to the policy-era form.
Two never-implemented env-vars deleted from the docs. LLM_SECURITY_SCR_OFFLINE
(ci-cd-guide) and LLM_SECURITY_OFFLINE (supply-chain-attack example) were
documented as OSV.dev / npm-audit kill-switches. No code has ever read either -
verified by grep across scanners, hooks and scripts, which finds them only in
markdown. A promised kill-switch that does nothing is worse than a documented
absence: it is trusted precisely when the run is meant to be air-gapped. The
docs now say there is none and that egress must be blocked at the network
layer. The LLM_SECURITY_AUDIT_* wildcard is narrowed to the one real key.
Docs. Migration section in README + CHANGELOG with the env-var -> policy-key
table, the detection commands (env + shell rc + .envrc + workflows), and the
explicit warning that a removed variable is now INERT rather than an error -
which is the failure mode that loses a project its configuration silently. The
hardening-guide env table splits into surviving vars and a removed-vars
migration table; its "promote to block" runbook named two variables that no
longer exist. Also swept: CLAUDE.md hook table, scanner-reference, ci-cd-guide,
both lethal-trifecta example docs, mitigation-matrix, injection-research.
Test counts in README/CLAUDE.md synced 2034 -> 2045.
Suite 2045 tests, 0 fail (2039 + 4 posture-trifecta + 2 riskScoreV1). The two
known parallel-load flakes did not recur this run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez
This commit is contained in:
parent
b6af9b46df
commit
fdec4b36ad
16 changed files with 333 additions and 62 deletions
55
CHANGELOG.md
55
CHANGELOG.md
|
|
@ -6,6 +6,61 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Removed — BREAKING
|
||||||
|
|
||||||
|
- **The four `LLM_SECURITY_*` configuration env-vars deprecated in v7.3.0.**
|
||||||
|
`.llm-security/policy.json` is now the only source. A removed variable is
|
||||||
|
**inert**: it neither warns nor configures, so a project relying on one
|
||||||
|
silently returns to the default. See the Migration section in `README.md`
|
||||||
|
for the detection commands.
|
||||||
|
|
||||||
|
| Removed env-var | Policy key | Default |
|
||||||
|
|-----------------|------------|---------|
|
||||||
|
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` |
|
||||||
|
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` |
|
||||||
|
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` |
|
||||||
|
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset |
|
||||||
|
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | dies with the warning it silenced |
|
||||||
|
|
||||||
|
Env-vars with no policy equivalent are unaffected:
|
||||||
|
`LLM_SECURITY_PRECOMPACT_MODE`, `LLM_SECURITY_PRECOMPACT_MAX_BYTES`,
|
||||||
|
`LLM_SECURITY_UPDATE_CHECK`, `LLM_SECURITY_MCP_CACHE_FILE`,
|
||||||
|
`LLM_SECURITY_IDE_ROOTS`.
|
||||||
|
|
||||||
|
`getPolicyValueWithEnvWarn` and its one-shot stderr deprecation line are
|
||||||
|
deleted from `scanners/lib/policy-loader.mjs`; the four call sites collapse
|
||||||
|
to `getPolicyValue`. Hook strings that advertised a removed variable as the
|
||||||
|
escape hatch now name the policy key — a blocked user following the old text
|
||||||
|
would have set a variable that does nothing.
|
||||||
|
|
||||||
|
- **`riskScoreV1()`** in `scanners/lib/severity.mjs`, plus its
|
||||||
|
`SEVERITY_WEIGHTS_V1` table. `@deprecated` since v7.0.0, kept for
|
||||||
|
diff/comparison, with zero callers in code or tests. The v1 weights are
|
||||||
|
recorded here for anyone re-deriving an old score:
|
||||||
|
`critical 25, high 10, medium 4, low 1, info 0`, summed and capped at 100.
|
||||||
|
`riskScore()` (v2) is untouched, so no score, band, or verdict moves.
|
||||||
|
|
||||||
|
- **Two documented env-vars that were never implemented.**
|
||||||
|
`LLM_SECURITY_SCR_OFFLINE` (`docs/ci-cd-guide.md`) and `LLM_SECURITY_OFFLINE`
|
||||||
|
(`examples/supply-chain-attack/README.md`) were promised as OSV.dev / npm-audit
|
||||||
|
kill-switches; no code has ever read either. The docs now say plainly that
|
||||||
|
there is no kill-switch and that an air-gapped run must block egress at the
|
||||||
|
network layer. The `LLM_SECURITY_AUDIT_*` wildcard phrasing (README,
|
||||||
|
`docs/scanner-reference.md`, `docs/security-hardening-guide.md`) is narrowed
|
||||||
|
to the single real key.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Posture category 12 (Rule of Two) no longer keys off the identifier
|
||||||
|
`TRIFECTA_MODE`.** The check was `/TRIFECTA_MODE/i` over the session-guard
|
||||||
|
source, which measured what a constant was *named* rather than whether
|
||||||
|
enforcement was configurable at all. It now matches
|
||||||
|
`getPolicyValue('trifecta', 'mode', …)`, and still accepts a pre-v8 vendored
|
||||||
|
guard reading `LLM_SECURITY_TRIFECTA_MODE` — third-party projects carry their
|
||||||
|
own hook copy and are equally configurable. Without this, every correctly
|
||||||
|
migrated project would have dropped from PASS to PARTIAL. The PARTIAL
|
||||||
|
finding now recommends the policy key instead of the removed env-var.
|
||||||
|
|
||||||
## [7.8.3] - 2026-07-18
|
## [7.8.3] - 2026-07-18
|
||||||
|
|
||||||
Security and correctness patch. 47 verified fixes from the v7.8.1/v7.8.2
|
Security and correctness patch. 47 verified fixes from the v7.8.1/v7.8.2
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# LLM Security Plugin (v7.8.3)
|
# LLM Security Plugin (v7.8.3)
|
||||||
|
|
||||||
Security scanning, auditing, and threat modeling for Claude Code projects. 5 frameworks: OWASP LLM Top 10, Agentic AI Top 10 (ASI, 2026 edition), Skills Top 10 (AST), MCP Top 10, AI Agent Traps (DeepMind). 2034+ unit, integration, and end-to-end tests (`tests/e2e/` covers the multi-hook attack chain, multi-session state simulation, and the full scan-orchestrator pipeline); mutation-testing coverage not published.
|
Security scanning, auditing, and threat modeling for Claude Code projects. 5 frameworks: OWASP LLM Top 10, Agentic AI Top 10 (ASI, 2026 edition), Skills Top 10 (AST), MCP Top 10, AI Agent Traps (DeepMind). 2045+ unit, integration, and end-to-end tests (`tests/e2e/` covers the multi-hook attack chain, multi-session state simulation, and the full scan-orchestrator pipeline); mutation-testing coverage not published.
|
||||||
|
|
||||||
Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on demand.
|
Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on demand.
|
||||||
|
|
||||||
|
|
@ -58,13 +58,13 @@ Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on d
|
||||||
|
|
||||||
| Script | Event | Matcher | Purpose |
|
| Script | Event | Matcher | Purpose |
|
||||||
|--------|-------|---------|---------|
|
|--------|-------|---------|---------|
|
||||||
| `pre-prompt-inject-scan.mjs` | UserPromptSubmit | — | Block prompt injection, warn on manipulation (incl. oversight evasion, HTML obfuscation, MEDIUM advisory for leetspeak/homoglyphs/zero-width/multi-lang). Unicode Tag steganography detection. Mode: `LLM_SECURITY_INJECTION_MODE=block\|warn\|off` |
|
| `pre-prompt-inject-scan.mjs` | UserPromptSubmit | — | Block prompt injection, warn on manipulation (incl. oversight evasion, HTML obfuscation, MEDIUM advisory for leetspeak/homoglyphs/zero-width/multi-lang). Unicode Tag steganography detection. Mode: policy key `injection.mode` = `block\|warn\|off` |
|
||||||
| `pre-edit-secrets.mjs` | PreToolUse | `Edit\|Write` | Block credentials in files |
|
| `pre-edit-secrets.mjs` | PreToolUse | `Edit\|Write` | Block credentials in files |
|
||||||
| `pre-bash-destructive.mjs` | PreToolUse | `Bash` | Block rm -rf, curl\|sh, fork bombs, eval. Bash evasion normalization (T1-T6 via `bash-normalize.mjs`) — defense-in-depth |
|
| `pre-bash-destructive.mjs` | PreToolUse | `Bash` | Block rm -rf, curl\|sh, fork bombs, eval. Bash evasion normalization (T1-T6 via `bash-normalize.mjs`) — defense-in-depth |
|
||||||
| `pre-install-supply-chain.mjs` | PreToolUse | `Bash` | Block compromised packages across ALL ecosystems. Bash evasion normalization before gate matching |
|
| `pre-install-supply-chain.mjs` | PreToolUse | `Bash` | Block compromised packages across ALL ecosystems. Bash evasion normalization before gate matching |
|
||||||
| `pre-write-pathguard.mjs` | PreToolUse | `Edit\|Write` | Block writes to .env, .ssh/, .aws/, credentials, settings |
|
| `pre-write-pathguard.mjs` | PreToolUse | `Edit\|Write` | Block writes to .env, .ssh/, .aws/, credentials, settings |
|
||||||
| `post-mcp-verify.mjs` | PostToolUse | — (all) | Injection scan on ALL tool output. MCP per-update drift + cumulative drift vs sticky baseline (E14, v7.3.0). Per-tool volume tracking |
|
| `post-mcp-verify.mjs` | PostToolUse | — (all) | Injection scan on ALL tool output. MCP per-update drift + cumulative drift vs sticky baseline (E14, v7.3.0). Per-tool volume tracking |
|
||||||
| `post-session-guard.mjs` | PostToolUse | — (all) | Runtime trifecta detection (Rule of Two). Sliding window + long-horizon. Behavioral drift (Jensen-Shannon). Mode: `LLM_SECURITY_TRIFECTA_MODE=block\|warn\|off` (default: warn) |
|
| `post-session-guard.mjs` | PostToolUse | — (all) | Runtime trifecta detection (Rule of Two). Sliding window + long-horizon. Behavioral drift (Jensen-Shannon). Mode: policy key `trifecta.mode` = `block\|warn\|off` (default: warn) |
|
||||||
| `update-check.mjs` | UserPromptSubmit | — | Checks for newer versions (max 1x/24h, cached). Disable: `LLM_SECURITY_UPDATE_CHECK=off` |
|
| `update-check.mjs` | UserPromptSubmit | — | Checks for newer versions (max 1x/24h, cached). Disable: `LLM_SECURITY_UPDATE_CHECK=off` |
|
||||||
| `pre-compact-scan.mjs` | PreCompact | — | Scan transcript for injection + credentials before context compaction. Reads at most last 512 KB. Mode: `LLM_SECURITY_PRECOMPACT_MODE=block\|warn\|off` (default: warn) |
|
| `pre-compact-scan.mjs` | PreCompact | — | Scan transcript for injection + credentials before context compaction. Reads at most last 512 KB. Mode: `LLM_SECURITY_PRECOMPACT_MODE=block\|warn\|off` (default: warn) |
|
||||||
|
|
||||||
|
|
|
||||||
59
README.md
59
README.md
|
|
@ -199,13 +199,13 @@ Hooks run on every operation — no commands needed. They activate the moment th
|
||||||
|
|
||||||
| Hook | Event | What it does |
|
| Hook | Event | What it does |
|
||||||
|------|-------|--------------|
|
|------|-------|--------------|
|
||||||
| **Prompt injection scan** | UserPromptSubmit | Blocks direct injection (override instructions, spoofed system headers, identity redefinition) and warns on subtle signals (leetspeak, homoglyphs, zero-width chars, multi-language). Decodes obfuscated payloads (Unicode Tag, hex, URL, base64, rot13) before matching. Mode: `LLM_SECURITY_INJECTION_MODE=block\|warn\|off` (default block) |
|
| **Prompt injection scan** | UserPromptSubmit | Blocks direct injection (override instructions, spoofed system headers, identity redefinition) and warns on subtle signals (leetspeak, homoglyphs, zero-width chars, multi-language). Decodes obfuscated payloads (Unicode Tag, hex, URL, base64, rot13) before matching. Mode: policy key `injection.mode` = `block\|warn\|off` (default block) |
|
||||||
| **Secret detection** | Edit, Write | Blocks AWS keys, Azure tokens, GitHub PATs, npm tokens, PEM keys, database URLs, Bearer tokens, and 30+ other secret patterns |
|
| **Secret detection** | Edit, Write | Blocks AWS keys, Azure tokens, GitHub PATs, npm tokens, PEM keys, database URLs, Bearer tokens, and 30+ other secret patterns |
|
||||||
| **Path guarding** | Write | Blocks writes to `.env*` (multi-segment-suffix-safe), `.ssh/`, `.aws/`, `.gnupg/`, credentials files, hook scripts, `/etc/`, `settings.json` |
|
| **Path guarding** | Write | Blocks writes to `.env*` (multi-segment-suffix-safe), `.ssh/`, `.aws/`, `.gnupg/`, credentials files, hook scripts, `/etc/`, `settings.json` |
|
||||||
| **Destructive commands** | Bash | Blocks `rm -rf /`, `chmod 777`, pipe-to-shell, fork bombs, eval-with-substitution, T8 base64-pipe-shell loaders. Bash-normalize T1-T9 collapses obfuscation (empty quotes, `${IFS}`, ANSI-C hex, process substitution, eval-via-variable) before pattern matching |
|
| **Destructive commands** | Bash | Blocks `rm -rf /`, `chmod 777`, pipe-to-shell, fork bombs, eval-with-substitution, T8 base64-pipe-shell loaders. Bash-normalize T1-T9 collapses obfuscation (empty quotes, `${IFS}`, ANSI-C hex, process substitution, eval-via-variable) before pattern matching |
|
||||||
| **Supply-chain guardrail** | Bash | Blocks known-compromised npm/pip packages, Levenshtein typosquats, age-gated installs (<72 h), OSV.dev CVE checks. Covers npm, pip, brew, docker, go, cargo, gem. v7.3.0: npm scope-hop typosquat advisory (E13) — `@evil/lodash`-class catches scope-jumping when the unscoped name matches a popular package |
|
| **Supply-chain guardrail** | Bash | Blocks known-compromised npm/pip packages, Levenshtein typosquats, age-gated installs (<72 h), OSV.dev CVE checks. Covers npm, pip, brew, docker, go, cargo, gem. v7.3.0: npm scope-hop typosquat advisory (E13) — `@evil/lodash`-class catches scope-jumping when the unscoped name matches a popular package |
|
||||||
| **Output verification** | All tools (post) | Advisory: scans ALL tool output for indirect injection (LLM01) and HITL traps (DeepMind kat. 6). Bash-specific: leaked secrets, unexpected URLs, oversized MCP responses. v7.3.0: per-update MCP description drift AND cumulative drift vs sticky baseline (E14) — slow-burn rug-pulls that stay under per-update thresholds but cumulatively diverge ≥25% emit `mcp-cumulative-drift` MEDIUM |
|
| **Output verification** | All tools (post) | Advisory: scans ALL tool output for indirect injection (LLM01) and HITL traps (DeepMind kat. 6). Bash-specific: leaked secrets, unexpected URLs, oversized MCP responses. v7.3.0: per-update MCP description drift AND cumulative drift vs sticky baseline (E14) — slow-burn rug-pulls that stay under per-update thresholds but cumulatively diverge ≥25% emit `mcp-cumulative-drift` MEDIUM |
|
||||||
| **Session guard** | All tools (post) | Advisory: monitors tool-call sequences for the lethal trifecta (untrusted input + sensitive read + exfiltration sink). 20-call sliding window + 100-call long-horizon window. Mode: `LLM_SECURITY_TRIFECTA_MODE=block\|warn\|off`. Sub-agent delegation tracking via Task/Agent tools surfaces escalation-after-input as a separate advisory |
|
| **Session guard** | All tools (post) | Advisory: monitors tool-call sequences for the lethal trifecta (untrusted input + sensitive read + exfiltration sink). 20-call sliding window + 100-call long-horizon window. Mode: policy key `trifecta.mode` = `block\|warn\|off`. Sub-agent delegation tracking via Task/Agent tools surfaces escalation-after-input as a separate advisory |
|
||||||
| **Pre-compact scan** | PreCompact | Scans transcript tail (max 512 KB, <500 ms) for injection patterns + credentials before context compaction. Prevents poisoned content from surviving in compact form. Mode: `LLM_SECURITY_PRECOMPACT_MODE=block\|warn\|off` (default warn) |
|
| **Pre-compact scan** | PreCompact | Scans transcript tail (max 512 KB, <500 ms) for injection patterns + credentials before context compaction. Prevents poisoned content from surviving in compact form. Mode: `LLM_SECURITY_PRECOMPACT_MODE=block\|warn\|off` (default warn) |
|
||||||
| **Update check** | UserPromptSubmit | Checks for newer plugin versions max 1× / 24 h, cached. Disable: `LLM_SECURITY_UPDATE_CHECK=off` |
|
| **Update check** | UserPromptSubmit | Checks for newer plugin versions max 1× / 24 h, cached. Disable: `LLM_SECURITY_UPDATE_CHECK=off` |
|
||||||
|
|
||||||
|
|
@ -329,9 +329,9 @@ Average ~69 %. Strongest at prompt injection (95 % with input + output scanning
|
||||||
| **Compliance mapping** | EU AI Act (Art. 9, 15, 17), NIST AI RMF (Map / Measure / Manage / Govern), ISO 42001 (Annex A), MITRE ATLAS techniques. Posture categories 14-16 assess readiness |
|
| **Compliance mapping** | EU AI Act (Art. 9, 15, 17), NIST AI RMF (Map / Measure / Manage / Govern), ISO 42001 (Annex A), MITRE ATLAS techniques. Posture categories 14-16 assess readiness |
|
||||||
| **Norwegian context** | Datatilsynet DPIA-for-AI guidance, NSM basic security principles, Digitaliseringsdirektoratet — relevant for Norwegian public-sector deployments |
|
| **Norwegian context** | Datatilsynet DPIA-for-AI guidance, NSM basic security principles, Digitaliseringsdirektoratet — relevant for Norwegian public-sector deployments |
|
||||||
| **SARIF 2.1.0 output** | `--format sarif` on scan / deep-scan produces OASIS SARIF for CI/CD ingestion (GitHub Advanced Security, Azure DevOps, SonarQube) |
|
| **SARIF 2.1.0 output** | `--format sarif` on scan / deep-scan produces OASIS SARIF for CI/CD ingestion (GitHub Advanced Security, Azure DevOps, SonarQube) |
|
||||||
| **Structured audit trail** | JSONL events with ISO 8601 timestamps and OWASP category tags (`LLM_SECURITY_AUDIT_*` env-vars or `audit.log_path` policy key) — SIEM-ready |
|
| **Structured audit trail** | JSONL events with ISO 8601 timestamps and OWASP category tags (`audit.log_path` policy key) — SIEM-ready |
|
||||||
| **AI-BOM** | CycloneDX 1.6 BOM for AI components — models, MCP servers, plugins, knowledge files, hooks (`llm-security audit-bom <target>`) |
|
| **AI-BOM** | CycloneDX 1.6 BOM for AI components — models, MCP servers, plugins, knowledge files, hooks (`llm-security audit-bom <target>`) |
|
||||||
| **Policy-as-code** | `.llm-security/policy.json` ships hook configuration with the team. v7.3.0 (D3) adds a one-time-per-process stderr deprecation line when both an env-var AND its `policy.json` equivalent are explicitly set; env still wins through the v7.x runway, env reads removed in v8.0.0. Suppress noise with `LLM_SECURITY_DEPRECATION_QUIET=1` |
|
| **Policy-as-code** | `.llm-security/policy.json` ships hook configuration with the team — the only source for injection, trifecta, and audit configuration since v8.0.0 removed the overlapping env-vars |
|
||||||
| **Standalone CLI** | `node bin/llm-security.mjs scan <target>` — runs scanners without Claude Code. Subcommands: `scan`, `deep-scan`, `posture`, `audit-bom`, `benchmark`. Schrems II compatible in default offline mode (optional OSV.dev enrichment is the only network call and is opt-in) |
|
| **Standalone CLI** | `node bin/llm-security.mjs scan <target>` — runs scanners without Claude Code. Subcommands: `scan`, `deep-scan`, `posture`, `audit-bom`, `benchmark`. Schrems II compatible in default offline mode (optional OSV.dev enrichment is the only network call and is opt-in) |
|
||||||
| **CI/CD integration** | `--fail-on <severity>` for threshold-based exit codes, `--compact` for one-liner output. Templates for GitHub Actions, Azure DevOps, GitLab CI in `ci/`. Guide: `docs/ci-cd-guide.md` |
|
| **CI/CD integration** | `--fail-on <severity>` for threshold-based exit codes, `--compact` for one-liner output. Templates for GitHub Actions, Azure DevOps, GitLab CI in `ci/`. Guide: `docs/ci-cd-guide.md` |
|
||||||
|
|
||||||
|
|
@ -419,14 +419,14 @@ These gaps are surfaced advisorily through `/security threat-model` and `/securi
|
||||||
|
|
||||||
This is a **solo open-source project in stabilization mode** as of 2026-05-01.
|
This is a **solo open-source project in stabilization mode** as of 2026-05-01.
|
||||||
The current feature set (5 frameworks, 22 scanners, 9 hooks, 6 agents,
|
The current feature set (5 frameworks, 22 scanners, 9 hooks, 6 agents,
|
||||||
20 commands, 23 knowledge files, 2034+ tests including a dedicated end-to-end suite) is the natural plateau for
|
20 commands, 23 knowledge files, 2045+ tests including a dedicated end-to-end suite) is the natural plateau for
|
||||||
what a deterministic + advisory plugin can defend against without crossing
|
what a deterministic + advisory plugin can defend against without crossing
|
||||||
into commercial-grade territory. Going forward, work focuses on:
|
into commercial-grade territory. Going forward, work focuses on:
|
||||||
|
|
||||||
- **Bug fixes** and security patches
|
- **Bug fixes** and security patches
|
||||||
- **Compatibility** with new Claude Code releases
|
- **Compatibility** with new Claude Code releases
|
||||||
- **Knowledge-base refresh** (OWASP updates, new published research, new attack patterns)
|
- **Knowledge-base refresh** (OWASP updates, new published research, new attack patterns)
|
||||||
- **Deprecation cleanup** — v8.0.0 removes the `LLM_SECURITY_*` env vars and `riskScoreV1` constant deprecated in v7.3.0
|
- **Deprecation cleanup** — v8.0.0 removed the four `LLM_SECURITY_*` mode env vars and `riskScoreV1`, both deprecated in v7.3.0 (see [Migration](#migrating-to-v800))
|
||||||
- **Opportunistic small additions** that fit the existing deterministic architecture
|
- **Opportunistic small additions** that fit the existing deterministic architecture
|
||||||
|
|
||||||
## Non-goals
|
## Non-goals
|
||||||
|
|
@ -479,6 +479,53 @@ Prompt injection is **structurally unsolvable** with current architectures (join
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Migrating to v8.0.0
|
||||||
|
|
||||||
|
v8.0.0 removes the four `LLM_SECURITY_*` configuration env-vars deprecated in
|
||||||
|
v7.3.0. Configuration moved to `.llm-security/policy.json`, which travels with
|
||||||
|
the repository instead of living in whoever's shell happened to launch Claude
|
||||||
|
Code.
|
||||||
|
|
||||||
|
**A removed variable is now inert.** It does not warn and it does not
|
||||||
|
configure — a project that relied on `LLM_SECURITY_INJECTION_MODE=off` silently
|
||||||
|
returns to the `block` default. Check for these before upgrading:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
env | grep '^LLM_SECURITY_\(INJECTION_MODE\|TRIFECTA_MODE\|ESCALATION_WINDOW\|AUDIT_LOG\|DEPRECATION_QUIET\)='
|
||||||
|
grep -rn 'LLM_SECURITY_\(INJECTION_MODE\|TRIFECTA_MODE\|ESCALATION_WINDOW\|AUDIT_LOG\)' \
|
||||||
|
~/.zshenv ~/.bashrc .envrc .github/workflows/ 2>/dev/null
|
||||||
|
```
|
||||||
|
|
||||||
|
| Removed env-var | Policy key | Default |
|
||||||
|
|-----------------|------------|---------|
|
||||||
|
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` |
|
||||||
|
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` |
|
||||||
|
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` |
|
||||||
|
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset (audit trail off) |
|
||||||
|
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | removed with the warning it silenced |
|
||||||
|
|
||||||
|
Translate each one you find into `.llm-security/policy.json`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"injection": { "mode": "block" },
|
||||||
|
"trifecta": { "mode": "warn", "escalation_window": 5 },
|
||||||
|
"audit": { "log_path": "/var/log/llm-security/audit.jsonl" }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Unaffected.** Env-vars with no policy equivalent keep working and are not
|
||||||
|
part of this change: `LLM_SECURITY_PRECOMPACT_MODE`,
|
||||||
|
`LLM_SECURITY_PRECOMPACT_MAX_BYTES`, `LLM_SECURITY_UPDATE_CHECK`,
|
||||||
|
`LLM_SECURITY_MCP_CACHE_FILE`, `LLM_SECURITY_IDE_ROOTS`.
|
||||||
|
|
||||||
|
**Also removed:** `riskScoreV1()` in `scanners/lib/severity.mjs`, the v1
|
||||||
|
sum-and-cap scoring formula `@deprecated` since v7.0.0 and kept for reference
|
||||||
|
only. It had no callers in code or tests. `riskScore()` (v2, severity-dominated)
|
||||||
|
is unchanged, so no score, band, or verdict moves.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Playground (v7.6.0)
|
## Playground (v7.6.0)
|
||||||
|
|
||||||
A single-file SPA at `playground/llm-security-playground.html` provides
|
A single-file SPA at `playground/llm-security-playground.html` provides
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ Integrate llm-security into your CI/CD pipeline for automated security scanning
|
||||||
|
|
||||||
**The standalone CLI makes zero network calls by default.** All 14 scanners operate locally on your source code using Shannon entropy analysis, regex pattern matching, AST traversal, and git log parsing. No data is transmitted to any external service.
|
**The standalone CLI makes zero network calls by default.** All 14 scanners operate locally on your source code using Shannon entropy analysis, regex pattern matching, AST traversal, and git log parsing. No data is transmitted to any external service.
|
||||||
|
|
||||||
**Exception: supply-chain-recheck** — When scanning lockfiles for known vulnerabilities, this scanner optionally queries the [OSV.dev](https://osv.dev/) batch API. This sends only package names and versions (not source code) over HTTPS. To disable: set `LLM_SECURITY_SCR_OFFLINE=1`.
|
**Exception: supply-chain-recheck** — When scanning lockfiles for known vulnerabilities, this scanner optionally queries the [OSV.dev](https://osv.dev/) batch API. This sends only package names and versions (not source code) over HTTPS. There is no kill-switch for this call today; block the host at the network layer if the run must be air-gapped.
|
||||||
|
|
||||||
**What about Claude Code integration?** The Claude Code plugin (hooks, agents, commands) uses AI models and sends data to Anthropic. These components are **not included** in the standalone CLI. When you run `npx llm-security scan`, only deterministic scanners execute.
|
**What about Claude Code integration?** The Claude Code plugin (hooks, agents, commands) uses AI models and sends data to Anthropic. These components are **not included** in the standalone CLI. When you run `npx llm-security scan`, only deterministic scanners execute.
|
||||||
|
|
||||||
|
|
@ -122,8 +122,11 @@ CLI flags always take precedence over policy file values.
|
||||||
|
|
||||||
| Variable | Description |
|
| Variable | Description |
|
||||||
|----------|-------------|
|
|----------|-------------|
|
||||||
| `LLM_SECURITY_SCR_OFFLINE=1` | Disable OSV.dev network calls in supply-chain-recheck |
|
| `LLM_SECURITY_PRECOMPACT_MODE` | `block\|warn\|off` for the PreCompact transcript scan (default `warn`) |
|
||||||
| `LLM_SECURITY_AUDIT_LOG=<path>` | Write structured JSONL audit trail (SIEM-ready) |
|
| `LLM_SECURITY_UPDATE_CHECK=off` | Disable the daily update-check HTTP call |
|
||||||
|
|
||||||
|
The audit trail moved to the policy file in v8.0.0: set `audit.log_path` in
|
||||||
|
`.llm-security/policy.json` instead of `LLM_SECURITY_AUDIT_LOG`.
|
||||||
|
|
||||||
## Exit Codes
|
## Exit Codes
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,7 @@ AST-taint (AST) shells out to a PARSE-ONLY python3 helper (`scanners/lib/py-ast-
|
||||||
Utility: `node scanners/lib/fs-utils.mjs <backup|restore|cleanup|tmppath> [args]`.
|
Utility: `node scanners/lib/fs-utils.mjs <backup|restore|cleanup|tmppath> [args]`.
|
||||||
|
|
||||||
Lib: `sarif-formatter.mjs` — converts scan output to OASIS SARIF 2.1.0 format. Used by `--format sarif` flag.
|
Lib: `sarif-formatter.mjs` — converts scan output to OASIS SARIF 2.1.0 format. Used by `--format sarif` flag.
|
||||||
Lib: `audit-trail.mjs` — writes structured JSONL audit events (ISO 8601, OWASP tags, SIEM-ready). Env: `LLM_SECURITY_AUDIT_*`.
|
Lib: `audit-trail.mjs` — writes structured JSONL audit events (ISO 8601, OWASP tags, SIEM-ready). Enabled by the `audit.log_path` key in `.llm-security/policy.json` (v8.0.0; was `LLM_SECURITY_AUDIT_LOG`).
|
||||||
Lib: `policy-loader.mjs` — reads `.llm-security/policy.json` for distributable hook configuration. Includes `ci` section (`failOn`, `compact`) for CI/CD defaults. Defaults match hardcoded values.
|
Lib: `policy-loader.mjs` — reads `.llm-security/policy.json` for distributable hook configuration. Includes `ci` section (`failOn`, `compact`) for CI/CD defaults. Defaults match hardcoded values.
|
||||||
|
|
||||||
**Standalone (8):** `posture-scanner.mjs` — deterministic posture assessment, 16 categories (incl. EU AI Act, NIST AI RMF, ISO 42001), <50ms. NOT in scan-orchestrator (meta-level, not code-level).
|
**Standalone (8):** `posture-scanner.mjs` — deterministic posture assessment, 16 categories (incl. EU AI Act, NIST AI RMF, ISO 42001), <50ms. NOT in scan-orchestrator (meta-level, not code-level).
|
||||||
|
|
|
||||||
|
|
@ -23,16 +23,33 @@ in production. Deviations are fine, but the defaults here are the tested path.
|
||||||
|
|
||||||
| Variable | Default | Modes |
|
| Variable | Default | Modes |
|
||||||
|----------|---------|-------|
|
|----------|---------|-------|
|
||||||
| `LLM_SECURITY_INJECTION_MODE` | `block` | `block` — exit 2 on critical/high injection patterns. `warn` — advisory via systemMessage. `off` — disables scan. |
|
|
||||||
| `LLM_SECURITY_TRIFECTA_MODE` | `warn` | `block` — exit 2 when lethal trifecta (untrusted input + sensitive data + exfiltration sink) detected. `warn` — advisory. `off` — disables. |
|
|
||||||
| `LLM_SECURITY_PRECOMPACT_MODE` | `warn` | `block` — exit 2 on findings during PreCompact. `warn` — advisory via systemMessage. `off` — disables scan. |
|
| `LLM_SECURITY_PRECOMPACT_MODE` | `warn` | `block` — exit 2 on findings during PreCompact. `warn` — advisory via systemMessage. `off` — disables scan. |
|
||||||
| `LLM_SECURITY_PRECOMPACT_MAX_BYTES` | `512000` | Tail size in bytes read from transcript for scanning. Higher values increase coverage at the cost of latency. |
|
| `LLM_SECURITY_PRECOMPACT_MAX_BYTES` | `512000` | Tail size in bytes read from transcript for scanning. Higher values increase coverage at the cost of latency. |
|
||||||
| `LLM_SECURITY_UPDATE_CHECK` | `on` | `off` disables the daily update-check HTTP call. |
|
| `LLM_SECURITY_UPDATE_CHECK` | `on` | `off` disables the daily update-check HTTP call. |
|
||||||
| `LLM_SECURITY_AUDIT_*` | unset | Audit trail configuration (destination, format, etc.) for SIEM-ready JSONL output. |
|
|
||||||
|
|
||||||
Apply env vars via shell profile, `.envrc`, or the host MDM. Do not write them
|
Apply env vars via shell profile, `.envrc`, or the host MDM. Do not write them
|
||||||
into the repository.
|
into the repository.
|
||||||
|
|
||||||
|
**Removed in v8.0.0.** Four modes moved to `.llm-security/policy.json`, which
|
||||||
|
travels with the repository instead of the operator's shell. Setting the old
|
||||||
|
variable is now inert — it neither warns nor configures.
|
||||||
|
|
||||||
|
| Removed variable | Policy key | Default |
|
||||||
|
|------------------|-----------|---------|
|
||||||
|
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` — exit 2 on critical injection patterns. `warn` — advisory via systemMessage. `off` — disables scan. |
|
||||||
|
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` — advisory. `block` — exit 2 when the lethal trifecta (untrusted input + sensitive data + exfiltration sink) is detected. `off` — disables. |
|
||||||
|
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` — calls between untrusted input and sub-agent delegation that still count as escalation-after-input. |
|
||||||
|
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset — set a path to write the SIEM-ready JSONL audit trail. |
|
||||||
|
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | Silenced the deprecation warning; removed with the warning. |
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"injection": { "mode": "block" },
|
||||||
|
"trifecta": { "mode": "warn", "escalation_window": 5 },
|
||||||
|
"audit": { "log_path": "/var/log/llm-security/audit.jsonl" }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Sandboxing
|
## 2. Sandboxing
|
||||||
|
|
@ -73,14 +90,16 @@ shell; avoid root, which disables user-namespace confinement.
|
||||||
### 3.1 Start in warn mode
|
### 3.1 Start in warn mode
|
||||||
|
|
||||||
Every new integration of `llm-security` should begin with all modes set to
|
Every new integration of `llm-security` should begin with all modes set to
|
||||||
`warn`. This yields advisories without breaking workflow, and lets the team
|
`warn` — `injection.mode` and `trifecta.mode` in `.llm-security/policy.json`,
|
||||||
calibrate false-positive rates against their actual repositories.
|
`LLM_SECURITY_PRECOMPACT_MODE` in the environment. This yields advisories
|
||||||
|
without breaking workflow, and lets the team calibrate false-positive rates
|
||||||
|
against their actual repositories.
|
||||||
|
|
||||||
### 3.2 Promote to block after baselining
|
### 3.2 Promote to block after baselining
|
||||||
|
|
||||||
After a baseline period (typically 1-2 weeks), flip each mode to `block` in this
|
After a baseline period (typically 1-2 weeks), flip each mode to `block` in this
|
||||||
order: `LLM_SECURITY_INJECTION_MODE`, `LLM_SECURITY_TRIFECTA_MODE`,
|
order: `injection.mode`, `trifecta.mode` (both in `.llm-security/policy.json`),
|
||||||
`LLM_SECURITY_PRECOMPACT_MODE`. The injection hook is first because false
|
then `LLM_SECURITY_PRECOMPACT_MODE`. The injection hook is first because false
|
||||||
positives there are the most visible; blocking comes last because the others
|
positives there are the most visible; blocking comes last because the others
|
||||||
build confidence.
|
build confidence.
|
||||||
|
|
||||||
|
|
@ -183,7 +202,9 @@ tier:
|
||||||
|
|
||||||
Verdict cutoffs (`BLOCK ≥65`, `WARNING ≥15`) are locked to the `riskBand()`
|
Verdict cutoffs (`BLOCK ≥65`, `WARNING ≥15`) are locked to the `riskBand()`
|
||||||
boundaries so you can't get a "BLOCK / Medium band" contradiction. The legacy
|
boundaries so you can't get a "BLOCK / Medium band" contradiction. The legacy
|
||||||
formula is kept as `riskScoreV1()` for reference only.
|
v1 formula was kept as `riskScoreV1()` for reference through the v7 line and
|
||||||
|
was removed in v8.0.0; see `CHANGELOG.md` for the v1 weights if you need them
|
||||||
|
to re-derive an old score.
|
||||||
|
|
||||||
**CI impact:** Pipelines with `--fail-on high` keep working (the severity
|
**CI impact:** Pipelines with `--fail-on high` keep working (the severity
|
||||||
gate is unaffected). Pipelines with score-based thresholds need recalibration
|
gate is unaffected). Pipelines with score-based thresholds need recalibration
|
||||||
|
|
|
||||||
|
|
@ -58,7 +58,7 @@ preview after step 3.
|
||||||
|
|
||||||
- **`hooks/scripts/post-session-guard.mjs`** — the only hook invoked.
|
- **`hooks/scripts/post-session-guard.mjs`** — the only hook invoked.
|
||||||
Configurable via `policy.json` `trifecta.mode` (`block` / `warn` /
|
Configurable via `policy.json` `trifecta.mode` (`block` / `warn` /
|
||||||
`off`; default `warn`) or env var `LLM_SECURITY_TRIFECTA_MODE`.
|
`off`; default `warn`) in `.llm-security/policy.json`.
|
||||||
|
|
||||||
This example uses `mode: warn` (default). In `block` mode the third
|
This example uses `mode: warn` (default). In `block` mode the third
|
||||||
call's advisory becomes a hard block (exit 2) and the agent action is
|
call's advisory becomes a hard block (exit 2) and the agent action is
|
||||||
|
|
@ -92,7 +92,7 @@ in a `finally` block before exiting. **Your real session state under
|
||||||
Jensen-Shannon divergence, or volume-threshold advisories.
|
Jensen-Shannon divergence, or volume-threshold advisories.
|
||||||
Those have their own unit tests under `tests/lib/post-session-guard.*`.
|
Those have their own unit tests under `tests/lib/post-session-guard.*`.
|
||||||
- This is deterministic detection. It does not exercise the
|
- This is deterministic detection. It does not exercise the
|
||||||
`block`-mode exit-2 path — flip `LLM_SECURITY_TRIFECTA_MODE=block`
|
`block`-mode exit-2 path — set `"trifecta": {"mode": "block"}`
|
||||||
and re-run if you want to see the script fail at step 3.
|
and re-run if you want to see the script fail at step 3.
|
||||||
|
|
||||||
## See also
|
## See also
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,7 @@ The `systemMessage` payload from step 3 must contain:
|
||||||
- The literal phrase `Rule of Two violation`
|
- The literal phrase `Rule of Two violation`
|
||||||
- A list of evidence items under `Untrusted input:`, `Data access:`,
|
- A list of evidence items under `Untrusted input:`, `Data access:`,
|
||||||
`Exfil sink:` headings
|
`Exfil sink:` headings
|
||||||
- A reference to `Set LLM_SECURITY_TRIFECTA_MODE=` for configuration
|
- A reference to the `trifecta.mode` policy key for configuration
|
||||||
- An OWASP tag mentioning `ASI01` or `ASI02`
|
- An OWASP tag mentioning `ASI01` or `ASI02`
|
||||||
|
|
||||||
Optional (depending on detail string and `policy.json` config):
|
Optional (depending on detail string and `policy.json` config):
|
||||||
|
|
@ -32,7 +32,7 @@ Optional (depending on detail string and `policy.json` config):
|
||||||
|
|
||||||
## Audit-trail side effect
|
## Audit-trail side effect
|
||||||
|
|
||||||
When `LLM_SECURITY_AUDIT_LOG` (or `policy.json` `audit.log_path`) is
|
When the `policy.json` key `audit.log_path` is
|
||||||
set, step 3 writes a JSONL event:
|
set, step 3 writes a JSONL event:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
|
|
@ -48,7 +48,7 @@ set, step 3 writes a JSONL event:
|
||||||
|
|
||||||
The walkthrough does not configure the audit log — `writeAuditEvent`
|
The walkthrough does not configure the audit log — `writeAuditEvent`
|
||||||
no-ops when no path is set. To observe the audit-trail behaviour,
|
no-ops when no path is set. To observe the audit-trail behaviour,
|
||||||
re-run with `LLM_SECURITY_AUDIT_LOG=/tmp/trifecta-audit.jsonl`.
|
re-run with `"audit": {"log_path": "/tmp/trifecta-audit.jsonl"}` in `.llm-security/policy.json`.
|
||||||
|
|
||||||
## State file
|
## State file
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -90,13 +90,12 @@ Expected: `5 pass, 0 fail`.
|
||||||
scope-hopping case the advisory is emitted before any network call.
|
scope-hopping case the advisory is emitted before any network call.
|
||||||
For the clean case it may attempt `npm view` — that runs against
|
For the clean case it may attempt `npm view` — that runs against
|
||||||
the public registry but is non-fatal if offline.
|
the public registry but is non-fatal if offline.
|
||||||
- **Stage B (dep-auditor)**: runs offline by default. If the env
|
- **Stage B (dep-auditor)**: runs offline by default. It may shell
|
||||||
var `LLM_SECURITY_OFFLINE=1` is unset, it may shell out to
|
out to `npm audit --json --offline=false` for CVE enrichment, but
|
||||||
`npm audit --json --offline=false` for CVE enrichment, but the
|
the fixture has no real npm install, so audit returns nothing.
|
||||||
fixture has no real npm install, so audit returns nothing.
|
|
||||||
|
|
||||||
If you need a fully air-gapped run, set `LLM_SECURITY_OFFLINE=1`
|
There is no environment kill-switch for these calls. If you need a
|
||||||
in the parent environment.
|
fully air-gapped run, block network egress for the process.
|
||||||
|
|
||||||
## OWASP / framework mapping
|
## OWASP / framework mapping
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ Attacker injects instructions via external content (files, web pages, tool outpu
|
||||||
| Prompt injection input scanning | Automated | `pre-prompt-inject-scan.mjs` detects CRITICAL/HIGH/MEDIUM injection patterns in user prompts | Hook file exists; MEDIUM advisory enabled |
|
| Prompt injection input scanning | Automated | `pre-prompt-inject-scan.mjs` detects CRITICAL/HIGH/MEDIUM injection patterns in user prompts | Hook file exists; MEDIUM advisory enabled |
|
||||||
| Unicode Tag steganography detection | Automated | `string-utils.mjs` decodes U+E0000-E007F tags; `injection-patterns.mjs` escalates to CRITICAL/HIGH | `decodeUnicodeTags()` in normalization pipeline |
|
| Unicode Tag steganography detection | Automated | `string-utils.mjs` decodes U+E0000-E007F tags; `injection-patterns.mjs` escalates to CRITICAL/HIGH | `decodeUnicodeTags()` in normalization pipeline |
|
||||||
| Bash evasion normalization | Automated | `bash-normalize.mjs` strips parameter expansion before pattern matching | `normalizeBashExpansion()` called by both bash hooks |
|
| Bash evasion normalization | Automated | `bash-normalize.mjs` strips parameter expansion before pattern matching | `normalizeBashExpansion()` called by both bash hooks |
|
||||||
| Rule of Two detection (block-mode opt-in) | Automated | `post-session-guard.mjs` detects trifecta (untrusted input + sensitive data + exfil); blocks only when `LLM_SECURITY_TRIFECTA_MODE=block` AND high-confidence trifecta is observed; default `warn` | `LLM_SECURITY_TRIFECTA_MODE` env var respected; block mode opt-in |
|
| Rule of Two detection (block-mode opt-in) | Automated | `post-session-guard.mjs` detects trifecta (untrusted input + sensitive data + exfil); blocks only when the policy key `trifecta.mode` is `block` AND high-confidence trifecta is observed; default `warn` | `trifecta.mode` in `.llm-security/policy.json` respected; block mode opt-in |
|
||||||
| Long-horizon monitoring | Automated | `post-session-guard.mjs` 100-call window + behavioral drift detection | Long-horizon window active alongside 20-call window |
|
| Long-horizon monitoring | Automated | `post-session-guard.mjs` 100-call window + behavioral drift detection | Long-horizon window active alongside 20-call window |
|
||||||
| HITL trap detection | Automated | `injection-patterns.mjs` HIGH patterns for approval urgency, summary suppression, scope minimization | HITL patterns present in HIGH_PATTERNS array |
|
| HITL trap detection | Automated | `injection-patterns.mjs` HIGH patterns for approval urgency, summary suppression, scope minimization | HITL patterns present in HIGH_PATTERNS array |
|
||||||
| Hybrid attack detection | Automated | `injection-patterns.mjs` HYBRID_PATTERNS for P2SQL, recursive injection, XSS | Hybrid patterns checked in tool output scanning |
|
| Hybrid attack detection | Automated | `injection-patterns.mjs` HYBRID_PATTERNS for P2SQL, recursive injection, XSS | Hybrid patterns checked in tool output scanning |
|
||||||
|
|
|
||||||
|
|
@ -65,7 +65,7 @@ Research summary for the llm-security plugin. Documents what the field has learn
|
||||||
- Sensitive path patterns need expansion as new sensitive files emerge
|
- Sensitive path patterns need expansion as new sensitive files emerge
|
||||||
|
|
||||||
**Plugin controls:**
|
**Plugin controls:**
|
||||||
- `post-session-guard.mjs`: `LLM_SECURITY_TRIFECTA_MODE=block|warn|off`
|
- `post-session-guard.mjs`: policy key `trifecta.mode` = `block|warn|off`
|
||||||
- Block mode: exit 2 for MCP-concentrated trifecta or sensitive path + exfil
|
- Block mode: exit 2 for MCP-concentrated trifecta or sensitive path + exfil
|
||||||
- Default warn mode preserves backward compatibility
|
- Default warn mode preserves backward compatibility
|
||||||
- **Gap:** Rule of Two is approximate — false positives possible for legitimate multi-tool workflows
|
- **Gap:** Rule of Two is approximate — false positives possible for legitimate multi-tool workflows
|
||||||
|
|
|
||||||
|
|
@ -9,9 +9,6 @@ export const SEVERITY = Object.freeze({
|
||||||
INFO: 'info',
|
INFO: 'info',
|
||||||
});
|
});
|
||||||
|
|
||||||
// Legacy weights — used only by riskScoreV1() for backwards-compat reference.
|
|
||||||
const SEVERITY_WEIGHTS_V1 = { critical: 25, high: 10, medium: 4, low: 1, info: 0 };
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Calculate aggregate risk score from severity counts (v2 model — v7.0.0+).
|
* Calculate aggregate risk score from severity counts (v2 model — v7.0.0+).
|
||||||
*
|
*
|
||||||
|
|
@ -56,24 +53,6 @@ export function riskScore(counts) {
|
||||||
return Math.round(Math.min(100, base));
|
return Math.round(Math.min(100, base));
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Legacy v1 risk score formula — kept for diff/comparison only.
|
|
||||||
* Not exported in production paths; reference for CI re-calibration.
|
|
||||||
*
|
|
||||||
* @deprecated Since v7.0.0. Use riskScore() instead. Kept for diff/comparison only — not used in production paths.
|
|
||||||
* @param {{ critical: number, high: number, medium: number, low: number, info: number }} counts
|
|
||||||
* @returns {number} 0-100 capped score (sum-and-cap model)
|
|
||||||
*/
|
|
||||||
export function riskScoreV1(counts) {
|
|
||||||
const raw =
|
|
||||||
(counts.critical || 0) * SEVERITY_WEIGHTS_V1.critical +
|
|
||||||
(counts.high || 0) * SEVERITY_WEIGHTS_V1.high +
|
|
||||||
(counts.medium || 0) * SEVERITY_WEIGHTS_V1.medium +
|
|
||||||
(counts.low || 0) * SEVERITY_WEIGHTS_V1.low +
|
|
||||||
(counts.info || 0) * SEVERITY_WEIGHTS_V1.info;
|
|
||||||
return Math.min(raw, 100);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Derive verdict from severity counts and risk score (v7.0.0 thresholds).
|
* Derive verdict from severity counts and risk score (v7.0.0 thresholds).
|
||||||
* Aligned to v2 riskBand cutoffs so verdict and band are co-monotonic:
|
* Aligned to v2 riskBand cutoffs so verdict and band are co-monotonic:
|
||||||
|
|
|
||||||
|
|
@ -1122,9 +1122,22 @@ async function checkRuleOfTwo(projectRoot, hooksJson) {
|
||||||
if (hookActive) {
|
if (hookActive) {
|
||||||
const scriptPath = join(projectRoot, 'hooks', 'scripts', 'post-session-guard.mjs');
|
const scriptPath = join(projectRoot, 'hooks', 'scripts', 'post-session-guard.mjs');
|
||||||
const content = await readText(scriptPath);
|
const content = await readText(scriptPath);
|
||||||
if (content && /TRIFECTA_MODE/i.test(content)) {
|
// What this category measures is whether enforcement is *configurable*, not
|
||||||
|
// what the constant holding it is called. v8.0.0 moved the mode to the
|
||||||
|
// policy.json key `trifecta.mode`; a pre-v8 vendored guard still resolving
|
||||||
|
// LLM_SECURITY_TRIFECTA_MODE is equally configurable and still passes.
|
||||||
|
const readsPolicyMode =
|
||||||
|
content && /getPolicyValue\w*\(\s*['"]trifecta['"]\s*,\s*['"]mode['"]/.test(content);
|
||||||
|
const readsLegacyEnv = content && /LLM_SECURITY_TRIFECTA_MODE/.test(content);
|
||||||
|
if (readsPolicyMode) {
|
||||||
hasTrifectaMode = true;
|
hasTrifectaMode = true;
|
||||||
evidence.push('TRIFECTA_MODE configurable: yes');
|
evidence.push('Enforcement mode configurable: policy.json trifecta.mode');
|
||||||
|
} else if (readsLegacyEnv) {
|
||||||
|
hasTrifectaMode = true;
|
||||||
|
evidence.push(
|
||||||
|
'Enforcement mode configurable: LLM_SECURITY_TRIFECTA_MODE (pre-v8.0.0 env-var; ' +
|
||||||
|
'migrate to the policy.json key trifecta.mode)'
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1146,10 +1159,10 @@ async function checkRuleOfTwo(projectRoot, hooksJson) {
|
||||||
scanner: 'PST',
|
scanner: 'PST',
|
||||||
severity: SEVERITY.MEDIUM,
|
severity: SEVERITY.MEDIUM,
|
||||||
title: 'Session guard lacks configurable trifecta mode',
|
title: 'Session guard lacks configurable trifecta mode',
|
||||||
description: 'post-session-guard does not support LLM_SECURITY_TRIFECTA_MODE (block/warn/off).',
|
description: 'post-session-guard resolves no enforcement mode (block/warn/off) from the policy.json key trifecta.mode.',
|
||||||
file: 'hooks/scripts/post-session-guard.mjs',
|
file: 'hooks/scripts/post-session-guard.mjs',
|
||||||
owasp: 'ASI02',
|
owasp: 'ASI02',
|
||||||
recommendation: 'Upgrade to v5.0 session guard with configurable enforcement mode.',
|
recommendation: 'Upgrade to a session guard that reads trifecta.mode from .llm-security/policy.json.',
|
||||||
}));
|
}));
|
||||||
return { status: STATUS.PARTIAL, findings, evidence };
|
return { status: STATUS.PARTIAL, findings, evidence };
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,12 @@
|
||||||
#!/usr/bin/env node
|
#!/usr/bin/env node
|
||||||
// post-session-guard.mjs — Runtime trifecta detection (Rule of Two)
|
// post-session-guard.mjs — Runtime trifecta detection (Rule of Two)
|
||||||
// v5.0: Configurable TRIFECTA_MODE (block|warn|off), long-horizon 100-call window,
|
// v5.0: configurable enforcement mode (block|warn|off), long-horizon 100-call
|
||||||
// behavioral drift via Jensen-Shannon divergence
|
// window, behavioral drift via Jensen-Shannon divergence
|
||||||
|
// v8.0.0: mode comes from the policy.json key trifecta.mode
|
||||||
import { readFileSync, appendFileSync } from 'node:fs';
|
import { readFileSync, appendFileSync } from 'node:fs';
|
||||||
|
import { getPolicyValue } from '../../scanners/lib/policy-loader.mjs';
|
||||||
|
|
||||||
const TRIFECTA_MODE = (process.env.LLM_SECURITY_TRIFECTA_MODE || 'warn').toLowerCase();
|
const TRIFECTA_MODE = String(getPolicyValue('trifecta', 'mode', 'warn')).toLowerCase();
|
||||||
const SLIDING_WINDOW = 20;
|
const SLIDING_WINDOW = 20;
|
||||||
const LONG_HORIZON_WINDOW = 100;
|
const LONG_HORIZON_WINDOW = 100;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -242,3 +242,26 @@ describe('B11 — the env-var deprecation mechanism is gone from source', () =>
|
||||||
assert.deepEqual(offenders, [], `removed env-vars are still read:\n${offenders.join('\n')}`);
|
assert.deepEqual(offenders, [], `removed env-vars are still read:\n${offenders.join('\n')}`);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// riskScoreV1 — the other v7.3.0 deprecation shipping in the same major
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
describe('B11 — riskScoreV1 is removed', () => {
|
||||||
|
it('severity.mjs no longer exports riskScoreV1', async () => {
|
||||||
|
const mod = await import('../../scanners/lib/severity.mjs');
|
||||||
|
assert.equal(
|
||||||
|
mod.riskScoreV1,
|
||||||
|
undefined,
|
||||||
|
'the v1 sum-and-cap formula was @deprecated in v7.3.0 with zero consumers'
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('riskScore (v2) is untouched and still severity-dominated', async () => {
|
||||||
|
const { riskScore } = await import('../../scanners/lib/severity.mjs');
|
||||||
|
const oneCritical = riskScore({ critical: 1, high: 0, medium: 0, low: 0, info: 0 });
|
||||||
|
const manyLow = riskScore({ critical: 0, high: 0, medium: 0, low: 50, info: 0 });
|
||||||
|
assert.ok(oneCritical >= 70 && oneCritical <= 95, `one critical -> ${oneCritical}`);
|
||||||
|
assert.ok(manyLow < oneCritical, 'v1 collapsed this case to 100; v2 must not');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
|
||||||
129
tests/scanners/posture-trifecta-mode.test.mjs
Normal file
129
tests/scanners/posture-trifecta-mode.test.mjs
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
// posture-trifecta-mode.test.mjs — B11: posture category 12 (Rule of Two)
|
||||||
|
// recognises a policy-era session guard.
|
||||||
|
//
|
||||||
|
// Pre-v8 the check was `/TRIFECTA_MODE/i` over the hook source. That matched
|
||||||
|
// the *identifier* `TRIFECTA_MODE`, not the configuration mechanism, so a hook
|
||||||
|
// that resolves the mode from `.llm-security/policy.json` without naming a
|
||||||
|
// constant that way scored PARTIAL despite being strictly more configurable.
|
||||||
|
// With LLM_SECURITY_TRIFECTA_MODE removed in v8.0.0, that regex would push
|
||||||
|
// every correctly-migrated project off PASS.
|
||||||
|
|
||||||
|
import { describe, it, beforeEach, afterEach } from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, cpSync } from 'node:fs';
|
||||||
|
import { join, resolve } from 'node:path';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||||
|
import { scan } from '../../scanners/posture-scanner.mjs';
|
||||||
|
|
||||||
|
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||||
|
const GRADE_A_FIXTURE = resolve(__dirname, '../fixtures/posture-scan/grade-a-project');
|
||||||
|
|
||||||
|
const GUARD_REL = join('hooks', 'scripts', 'post-session-guard.mjs');
|
||||||
|
|
||||||
|
// A guard that is configurable, but names nothing `TRIFECTA_MODE`.
|
||||||
|
const POLICY_ERA_GUARD = `#!/usr/bin/env node
|
||||||
|
// post-session-guard.mjs — Runtime trifecta detection (Rule of Two)
|
||||||
|
// v8.0.0: enforcement mode comes from .llm-security/policy.json.
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
import { getPolicyValue } from '../../scanners/lib/policy-loader.mjs';
|
||||||
|
|
||||||
|
const mode = String(getPolicyValue('trifecta', 'mode', 'warn')).toLowerCase();
|
||||||
|
|
||||||
|
const input = JSON.parse(readFileSync('/dev/stdin', 'utf-8'));
|
||||||
|
if (mode === 'off') {
|
||||||
|
process.stdout.write(JSON.stringify({ decision: 'allow' }));
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
const LONG_HORIZON_WINDOW = 100;
|
||||||
|
process.stdout.write(JSON.stringify({ decision: 'allow', window: LONG_HORIZON_WINDOW }));
|
||||||
|
`;
|
||||||
|
|
||||||
|
// A guard with no configurable mode at all — hardcoded enforcement.
|
||||||
|
const UNCONFIGURABLE_GUARD = `#!/usr/bin/env node
|
||||||
|
// post-session-guard.mjs — Runtime trifecta detection, no configuration.
|
||||||
|
import { readFileSync } from 'node:fs';
|
||||||
|
const input = JSON.parse(readFileSync('/dev/stdin', 'utf-8'));
|
||||||
|
const LONG_HORIZON_WINDOW = 100;
|
||||||
|
process.stdout.write(JSON.stringify({ decision: 'allow', window: LONG_HORIZON_WINDOW }));
|
||||||
|
`;
|
||||||
|
|
||||||
|
let root = null;
|
||||||
|
|
||||||
|
/** Copy of grade-a-project whose session guard is replaced with `source`. */
|
||||||
|
function projectWithGuard(source) {
|
||||||
|
root = mkdtempSync(join(tmpdir(), 'llmsec-posture-trifecta-'));
|
||||||
|
cpSync(GRADE_A_FIXTURE, root, { recursive: true });
|
||||||
|
mkdirSync(join(root, 'hooks', 'scripts'), { recursive: true });
|
||||||
|
writeFileSync(join(root, GUARD_REL), source);
|
||||||
|
return root;
|
||||||
|
}
|
||||||
|
|
||||||
|
function ruleOfTwo(result) {
|
||||||
|
return result.categories.find((c) => c.id === 12);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Category findings live on the top-level `findings` array, not on the category. */
|
||||||
|
function ruleOfTwoFindings(result) {
|
||||||
|
return result.findings.filter((f) => /trifecta mode|Rule of Two/i.test(f.title));
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('posture category 12 — Rule of Two mode detection (B11)', () => {
|
||||||
|
afterEach(() => {
|
||||||
|
if (root) rmSync(root, { recursive: true, force: true });
|
||||||
|
root = null;
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
resetCounter();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('PASSes a policy-era guard that reads trifecta.mode from policy.json', async () => {
|
||||||
|
const result = await scan(projectWithGuard(POLICY_ERA_GUARD));
|
||||||
|
const cat = ruleOfTwo(result);
|
||||||
|
assert.equal(
|
||||||
|
cat.status,
|
||||||
|
'PASS',
|
||||||
|
'a guard configured through policy.json is configurable; the identifier name is not the mechanism'
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
cat.evidence.some((e) => /trifecta\.mode|policy\.json/i.test(e)),
|
||||||
|
`expected evidence naming the policy key, got: ${JSON.stringify(cat.evidence)}`
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('PASSes a legacy guard still using the pre-v8 env-var', async () => {
|
||||||
|
// Third-party projects vendor their own hook copy and may not have migrated.
|
||||||
|
// Their enforcement mode is still configurable, so the category still holds.
|
||||||
|
const legacy = POLICY_ERA_GUARD.replace(
|
||||||
|
"String(getPolicyValue('trifecta', 'mode', 'warn'))",
|
||||||
|
"String(process.env.LLM_SECURITY_TRIFECTA_MODE || 'warn')"
|
||||||
|
);
|
||||||
|
const result = await scan(projectWithGuard(legacy));
|
||||||
|
assert.equal(ruleOfTwo(result).status, 'PASS');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not PASS a guard with no configurable mode', async () => {
|
||||||
|
const result = await scan(projectWithGuard(UNCONFIGURABLE_GUARD));
|
||||||
|
const cat = ruleOfTwo(result);
|
||||||
|
assert.equal(cat.status, 'PARTIAL', 'hardcoded enforcement is not a configurable mode');
|
||||||
|
assert.ok(
|
||||||
|
ruleOfTwoFindings(result).some((f) => /configurable trifecta mode/i.test(f.title)),
|
||||||
|
'expected the PARTIAL finding to name the missing configurability'
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('the PARTIAL finding recommends the policy key, not a removed env-var', async () => {
|
||||||
|
const result = await scan(projectWithGuard(UNCONFIGURABLE_GUARD));
|
||||||
|
const f = ruleOfTwoFindings(result).find((x) => /configurable trifecta mode/i.test(x.title));
|
||||||
|
assert.ok(f, 'expected the PARTIAL finding');
|
||||||
|
const text = `${f.description} ${f.recommendation}`;
|
||||||
|
assert.doesNotMatch(
|
||||||
|
text,
|
||||||
|
/LLM_SECURITY_TRIFECTA_MODE/,
|
||||||
|
'must not tell the user to set an env-var that v8 ignores'
|
||||||
|
);
|
||||||
|
assert.match(text, /trifecta\.mode/, 'should name the policy key');
|
||||||
|
});
|
||||||
|
});
|
||||||
Loading…
Add table
Add a link
Reference in a new issue