feat(llm-security)!: v8 Phase 3 complete - riskScoreV1, posture heuristic, docs
Closes Phase 3 (B11) of the v8.0.0 plan. Three parts, all with the failing
test written first.
riskScoreV1 removed. scanners/lib/severity.mjs drops riskScoreV1() and its
SEVERITY_WEIGHTS_V1 table - @deprecated since v7.0.0, kept for diff/comparison,
zero callers in code or tests (re-verified, not taken from the plan). The v1
weights are recorded in CHANGELOG so an old score stays re-derivable. riskScore
(v2) is untouched; a test pins that one critical still lands in the 70-95 tier
and that 50 lows score below it, which is exactly the case v1 collapsed to 100.
Posture category 12 no longer keys off an identifier name. The check was
/TRIFECTA_MODE/i over the session-guard source, which measured what a constant
was CALLED rather than whether enforcement was configurable. With the env-var
gone, that regex would have dropped every correctly-migrated project from PASS
to PARTIAL - the gate punishing the migration it exists to encourage. It now
matches getPolicyValue('trifecta', 'mode', ...) and still accepts a pre-v8
vendored guard reading the old env-var, because a third-party project carries
its own hook copy and is equally configurable either way; the evidence line
says which of the two was found. The PARTIAL finding recommended setting an
env-var that v8 ignores; it now names the policy key. The grade-a fixture hook
moves to the policy-era form.
Two never-implemented env-vars deleted from the docs. LLM_SECURITY_SCR_OFFLINE
(ci-cd-guide) and LLM_SECURITY_OFFLINE (supply-chain-attack example) were
documented as OSV.dev / npm-audit kill-switches. No code has ever read either -
verified by grep across scanners, hooks and scripts, which finds them only in
markdown. A promised kill-switch that does nothing is worse than a documented
absence: it is trusted precisely when the run is meant to be air-gapped. The
docs now say there is none and that egress must be blocked at the network
layer. The LLM_SECURITY_AUDIT_* wildcard is narrowed to the one real key.
Docs. Migration section in README + CHANGELOG with the env-var -> policy-key
table, the detection commands (env + shell rc + .envrc + workflows), and the
explicit warning that a removed variable is now INERT rather than an error -
which is the failure mode that loses a project its configuration silently. The
hardening-guide env table splits into surviving vars and a removed-vars
migration table; its "promote to block" runbook named two variables that no
longer exist. Also swept: CLAUDE.md hook table, scanner-reference, ci-cd-guide,
both lethal-trifecta example docs, mitigation-matrix, injection-research.
Test counts in README/CLAUDE.md synced 2034 -> 2045.
Suite 2045 tests, 0 fail (2039 + 4 posture-trifecta + 2 riskScoreV1). The two
known parallel-load flakes did not recur this run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez
This commit is contained in:
parent
b6af9b46df
commit
fdec4b36ad
16 changed files with 333 additions and 62 deletions
55
CHANGELOG.md
55
CHANGELOG.md
|
|
@ -6,6 +6,61 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Removed — BREAKING
|
||||
|
||||
- **The four `LLM_SECURITY_*` configuration env-vars deprecated in v7.3.0.**
|
||||
`.llm-security/policy.json` is now the only source. A removed variable is
|
||||
**inert**: it neither warns nor configures, so a project relying on one
|
||||
silently returns to the default. See the Migration section in `README.md`
|
||||
for the detection commands.
|
||||
|
||||
| Removed env-var | Policy key | Default |
|
||||
|-----------------|------------|---------|
|
||||
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` |
|
||||
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` |
|
||||
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` |
|
||||
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset |
|
||||
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | dies with the warning it silenced |
|
||||
|
||||
Env-vars with no policy equivalent are unaffected:
|
||||
`LLM_SECURITY_PRECOMPACT_MODE`, `LLM_SECURITY_PRECOMPACT_MAX_BYTES`,
|
||||
`LLM_SECURITY_UPDATE_CHECK`, `LLM_SECURITY_MCP_CACHE_FILE`,
|
||||
`LLM_SECURITY_IDE_ROOTS`.
|
||||
|
||||
`getPolicyValueWithEnvWarn` and its one-shot stderr deprecation line are
|
||||
deleted from `scanners/lib/policy-loader.mjs`; the four call sites collapse
|
||||
to `getPolicyValue`. Hook strings that advertised a removed variable as the
|
||||
escape hatch now name the policy key — a blocked user following the old text
|
||||
would have set a variable that does nothing.
|
||||
|
||||
- **`riskScoreV1()`** in `scanners/lib/severity.mjs`, plus its
|
||||
`SEVERITY_WEIGHTS_V1` table. `@deprecated` since v7.0.0, kept for
|
||||
diff/comparison, with zero callers in code or tests. The v1 weights are
|
||||
recorded here for anyone re-deriving an old score:
|
||||
`critical 25, high 10, medium 4, low 1, info 0`, summed and capped at 100.
|
||||
`riskScore()` (v2) is untouched, so no score, band, or verdict moves.
|
||||
|
||||
- **Two documented env-vars that were never implemented.**
|
||||
`LLM_SECURITY_SCR_OFFLINE` (`docs/ci-cd-guide.md`) and `LLM_SECURITY_OFFLINE`
|
||||
(`examples/supply-chain-attack/README.md`) were promised as OSV.dev / npm-audit
|
||||
kill-switches; no code has ever read either. The docs now say plainly that
|
||||
there is no kill-switch and that an air-gapped run must block egress at the
|
||||
network layer. The `LLM_SECURITY_AUDIT_*` wildcard phrasing (README,
|
||||
`docs/scanner-reference.md`, `docs/security-hardening-guide.md`) is narrowed
|
||||
to the single real key.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Posture category 12 (Rule of Two) no longer keys off the identifier
|
||||
`TRIFECTA_MODE`.** The check was `/TRIFECTA_MODE/i` over the session-guard
|
||||
source, which measured what a constant was *named* rather than whether
|
||||
enforcement was configurable at all. It now matches
|
||||
`getPolicyValue('trifecta', 'mode', …)`, and still accepts a pre-v8 vendored
|
||||
guard reading `LLM_SECURITY_TRIFECTA_MODE` — third-party projects carry their
|
||||
own hook copy and are equally configurable. Without this, every correctly
|
||||
migrated project would have dropped from PASS to PARTIAL. The PARTIAL
|
||||
finding now recommends the policy key instead of the removed env-var.
|
||||
|
||||
## [7.8.3] - 2026-07-18
|
||||
|
||||
Security and correctness patch. 47 verified fixes from the v7.8.1/v7.8.2
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# LLM Security Plugin (v7.8.3)
|
||||
|
||||
Security scanning, auditing, and threat modeling for Claude Code projects. 5 frameworks: OWASP LLM Top 10, Agentic AI Top 10 (ASI, 2026 edition), Skills Top 10 (AST), MCP Top 10, AI Agent Traps (DeepMind). 2034+ unit, integration, and end-to-end tests (`tests/e2e/` covers the multi-hook attack chain, multi-session state simulation, and the full scan-orchestrator pipeline); mutation-testing coverage not published.
|
||||
Security scanning, auditing, and threat modeling for Claude Code projects. 5 frameworks: OWASP LLM Top 10, Agentic AI Top 10 (ASI, 2026 edition), Skills Top 10 (AST), MCP Top 10, AI Agent Traps (DeepMind). 2045+ unit, integration, and end-to-end tests (`tests/e2e/` covers the multi-hook attack chain, multi-session state simulation, and the full scan-orchestrator pipeline); mutation-testing coverage not published.
|
||||
|
||||
Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on demand.
|
||||
|
||||
|
|
@ -58,13 +58,13 @@ Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on d
|
|||
|
||||
| Script | Event | Matcher | Purpose |
|
||||
|--------|-------|---------|---------|
|
||||
| `pre-prompt-inject-scan.mjs` | UserPromptSubmit | — | Block prompt injection, warn on manipulation (incl. oversight evasion, HTML obfuscation, MEDIUM advisory for leetspeak/homoglyphs/zero-width/multi-lang). Unicode Tag steganography detection. Mode: `LLM_SECURITY_INJECTION_MODE=block\|warn\|off` |
|
||||
| `pre-prompt-inject-scan.mjs` | UserPromptSubmit | — | Block prompt injection, warn on manipulation (incl. oversight evasion, HTML obfuscation, MEDIUM advisory for leetspeak/homoglyphs/zero-width/multi-lang). Unicode Tag steganography detection. Mode: policy key `injection.mode` = `block\|warn\|off` |
|
||||
| `pre-edit-secrets.mjs` | PreToolUse | `Edit\|Write` | Block credentials in files |
|
||||
| `pre-bash-destructive.mjs` | PreToolUse | `Bash` | Block rm -rf, curl\|sh, fork bombs, eval. Bash evasion normalization (T1-T6 via `bash-normalize.mjs`) — defense-in-depth |
|
||||
| `pre-install-supply-chain.mjs` | PreToolUse | `Bash` | Block compromised packages across ALL ecosystems. Bash evasion normalization before gate matching |
|
||||
| `pre-write-pathguard.mjs` | PreToolUse | `Edit\|Write` | Block writes to .env, .ssh/, .aws/, credentials, settings |
|
||||
| `post-mcp-verify.mjs` | PostToolUse | — (all) | Injection scan on ALL tool output. MCP per-update drift + cumulative drift vs sticky baseline (E14, v7.3.0). Per-tool volume tracking |
|
||||
| `post-session-guard.mjs` | PostToolUse | — (all) | Runtime trifecta detection (Rule of Two). Sliding window + long-horizon. Behavioral drift (Jensen-Shannon). Mode: `LLM_SECURITY_TRIFECTA_MODE=block\|warn\|off` (default: warn) |
|
||||
| `post-session-guard.mjs` | PostToolUse | — (all) | Runtime trifecta detection (Rule of Two). Sliding window + long-horizon. Behavioral drift (Jensen-Shannon). Mode: policy key `trifecta.mode` = `block\|warn\|off` (default: warn) |
|
||||
| `update-check.mjs` | UserPromptSubmit | — | Checks for newer versions (max 1x/24h, cached). Disable: `LLM_SECURITY_UPDATE_CHECK=off` |
|
||||
| `pre-compact-scan.mjs` | PreCompact | — | Scan transcript for injection + credentials before context compaction. Reads at most last 512 KB. Mode: `LLM_SECURITY_PRECOMPACT_MODE=block\|warn\|off` (default: warn) |
|
||||
|
||||
|
|
|
|||
59
README.md
59
README.md
|
|
@ -199,13 +199,13 @@ Hooks run on every operation — no commands needed. They activate the moment th
|
|||
|
||||
| Hook | Event | What it does |
|
||||
|------|-------|--------------|
|
||||
| **Prompt injection scan** | UserPromptSubmit | Blocks direct injection (override instructions, spoofed system headers, identity redefinition) and warns on subtle signals (leetspeak, homoglyphs, zero-width chars, multi-language). Decodes obfuscated payloads (Unicode Tag, hex, URL, base64, rot13) before matching. Mode: `LLM_SECURITY_INJECTION_MODE=block\|warn\|off` (default block) |
|
||||
| **Prompt injection scan** | UserPromptSubmit | Blocks direct injection (override instructions, spoofed system headers, identity redefinition) and warns on subtle signals (leetspeak, homoglyphs, zero-width chars, multi-language). Decodes obfuscated payloads (Unicode Tag, hex, URL, base64, rot13) before matching. Mode: policy key `injection.mode` = `block\|warn\|off` (default block) |
|
||||
| **Secret detection** | Edit, Write | Blocks AWS keys, Azure tokens, GitHub PATs, npm tokens, PEM keys, database URLs, Bearer tokens, and 30+ other secret patterns |
|
||||
| **Path guarding** | Write | Blocks writes to `.env*` (multi-segment-suffix-safe), `.ssh/`, `.aws/`, `.gnupg/`, credentials files, hook scripts, `/etc/`, `settings.json` |
|
||||
| **Destructive commands** | Bash | Blocks `rm -rf /`, `chmod 777`, pipe-to-shell, fork bombs, eval-with-substitution, T8 base64-pipe-shell loaders. Bash-normalize T1-T9 collapses obfuscation (empty quotes, `${IFS}`, ANSI-C hex, process substitution, eval-via-variable) before pattern matching |
|
||||
| **Supply-chain guardrail** | Bash | Blocks known-compromised npm/pip packages, Levenshtein typosquats, age-gated installs (<72 h), OSV.dev CVE checks. Covers npm, pip, brew, docker, go, cargo, gem. v7.3.0: npm scope-hop typosquat advisory (E13) — `@evil/lodash`-class catches scope-jumping when the unscoped name matches a popular package |
|
||||
| **Output verification** | All tools (post) | Advisory: scans ALL tool output for indirect injection (LLM01) and HITL traps (DeepMind kat. 6). Bash-specific: leaked secrets, unexpected URLs, oversized MCP responses. v7.3.0: per-update MCP description drift AND cumulative drift vs sticky baseline (E14) — slow-burn rug-pulls that stay under per-update thresholds but cumulatively diverge ≥25% emit `mcp-cumulative-drift` MEDIUM |
|
||||
| **Session guard** | All tools (post) | Advisory: monitors tool-call sequences for the lethal trifecta (untrusted input + sensitive read + exfiltration sink). 20-call sliding window + 100-call long-horizon window. Mode: `LLM_SECURITY_TRIFECTA_MODE=block\|warn\|off`. Sub-agent delegation tracking via Task/Agent tools surfaces escalation-after-input as a separate advisory |
|
||||
| **Session guard** | All tools (post) | Advisory: monitors tool-call sequences for the lethal trifecta (untrusted input + sensitive read + exfiltration sink). 20-call sliding window + 100-call long-horizon window. Mode: policy key `trifecta.mode` = `block\|warn\|off`. Sub-agent delegation tracking via Task/Agent tools surfaces escalation-after-input as a separate advisory |
|
||||
| **Pre-compact scan** | PreCompact | Scans transcript tail (max 512 KB, <500 ms) for injection patterns + credentials before context compaction. Prevents poisoned content from surviving in compact form. Mode: `LLM_SECURITY_PRECOMPACT_MODE=block\|warn\|off` (default warn) |
|
||||
| **Update check** | UserPromptSubmit | Checks for newer plugin versions max 1× / 24 h, cached. Disable: `LLM_SECURITY_UPDATE_CHECK=off` |
|
||||
|
||||
|
|
@ -329,9 +329,9 @@ Average ~69 %. Strongest at prompt injection (95 % with input + output scanning
|
|||
| **Compliance mapping** | EU AI Act (Art. 9, 15, 17), NIST AI RMF (Map / Measure / Manage / Govern), ISO 42001 (Annex A), MITRE ATLAS techniques. Posture categories 14-16 assess readiness |
|
||||
| **Norwegian context** | Datatilsynet DPIA-for-AI guidance, NSM basic security principles, Digitaliseringsdirektoratet — relevant for Norwegian public-sector deployments |
|
||||
| **SARIF 2.1.0 output** | `--format sarif` on scan / deep-scan produces OASIS SARIF for CI/CD ingestion (GitHub Advanced Security, Azure DevOps, SonarQube) |
|
||||
| **Structured audit trail** | JSONL events with ISO 8601 timestamps and OWASP category tags (`LLM_SECURITY_AUDIT_*` env-vars or `audit.log_path` policy key) — SIEM-ready |
|
||||
| **Structured audit trail** | JSONL events with ISO 8601 timestamps and OWASP category tags (`audit.log_path` policy key) — SIEM-ready |
|
||||
| **AI-BOM** | CycloneDX 1.6 BOM for AI components — models, MCP servers, plugins, knowledge files, hooks (`llm-security audit-bom <target>`) |
|
||||
| **Policy-as-code** | `.llm-security/policy.json` ships hook configuration with the team. v7.3.0 (D3) adds a one-time-per-process stderr deprecation line when both an env-var AND its `policy.json` equivalent are explicitly set; env still wins through the v7.x runway, env reads removed in v8.0.0. Suppress noise with `LLM_SECURITY_DEPRECATION_QUIET=1` |
|
||||
| **Policy-as-code** | `.llm-security/policy.json` ships hook configuration with the team — the only source for injection, trifecta, and audit configuration since v8.0.0 removed the overlapping env-vars |
|
||||
| **Standalone CLI** | `node bin/llm-security.mjs scan <target>` — runs scanners without Claude Code. Subcommands: `scan`, `deep-scan`, `posture`, `audit-bom`, `benchmark`. Schrems II compatible in default offline mode (optional OSV.dev enrichment is the only network call and is opt-in) |
|
||||
| **CI/CD integration** | `--fail-on <severity>` for threshold-based exit codes, `--compact` for one-liner output. Templates for GitHub Actions, Azure DevOps, GitLab CI in `ci/`. Guide: `docs/ci-cd-guide.md` |
|
||||
|
||||
|
|
@ -419,14 +419,14 @@ These gaps are surfaced advisorily through `/security threat-model` and `/securi
|
|||
|
||||
This is a **solo open-source project in stabilization mode** as of 2026-05-01.
|
||||
The current feature set (5 frameworks, 22 scanners, 9 hooks, 6 agents,
|
||||
20 commands, 23 knowledge files, 2034+ tests including a dedicated end-to-end suite) is the natural plateau for
|
||||
20 commands, 23 knowledge files, 2045+ tests including a dedicated end-to-end suite) is the natural plateau for
|
||||
what a deterministic + advisory plugin can defend against without crossing
|
||||
into commercial-grade territory. Going forward, work focuses on:
|
||||
|
||||
- **Bug fixes** and security patches
|
||||
- **Compatibility** with new Claude Code releases
|
||||
- **Knowledge-base refresh** (OWASP updates, new published research, new attack patterns)
|
||||
- **Deprecation cleanup** — v8.0.0 removes the `LLM_SECURITY_*` env vars and `riskScoreV1` constant deprecated in v7.3.0
|
||||
- **Deprecation cleanup** — v8.0.0 removed the four `LLM_SECURITY_*` mode env vars and `riskScoreV1`, both deprecated in v7.3.0 (see [Migration](#migrating-to-v800))
|
||||
- **Opportunistic small additions** that fit the existing deterministic architecture
|
||||
|
||||
## Non-goals
|
||||
|
|
@ -479,6 +479,53 @@ Prompt injection is **structurally unsolvable** with current architectures (join
|
|||
|
||||
---
|
||||
|
||||
## Migrating to v8.0.0
|
||||
|
||||
v8.0.0 removes the four `LLM_SECURITY_*` configuration env-vars deprecated in
|
||||
v7.3.0. Configuration moved to `.llm-security/policy.json`, which travels with
|
||||
the repository instead of living in whoever's shell happened to launch Claude
|
||||
Code.
|
||||
|
||||
**A removed variable is now inert.** It does not warn and it does not
|
||||
configure — a project that relied on `LLM_SECURITY_INJECTION_MODE=off` silently
|
||||
returns to the `block` default. Check for these before upgrading:
|
||||
|
||||
```bash
|
||||
env | grep '^LLM_SECURITY_\(INJECTION_MODE\|TRIFECTA_MODE\|ESCALATION_WINDOW\|AUDIT_LOG\|DEPRECATION_QUIET\)='
|
||||
grep -rn 'LLM_SECURITY_\(INJECTION_MODE\|TRIFECTA_MODE\|ESCALATION_WINDOW\|AUDIT_LOG\)' \
|
||||
~/.zshenv ~/.bashrc .envrc .github/workflows/ 2>/dev/null
|
||||
```
|
||||
|
||||
| Removed env-var | Policy key | Default |
|
||||
|-----------------|------------|---------|
|
||||
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` |
|
||||
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` |
|
||||
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` |
|
||||
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset (audit trail off) |
|
||||
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | removed with the warning it silenced |
|
||||
|
||||
Translate each one you find into `.llm-security/policy.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"injection": { "mode": "block" },
|
||||
"trifecta": { "mode": "warn", "escalation_window": 5 },
|
||||
"audit": { "log_path": "/var/log/llm-security/audit.jsonl" }
|
||||
}
|
||||
```
|
||||
|
||||
**Unaffected.** Env-vars with no policy equivalent keep working and are not
|
||||
part of this change: `LLM_SECURITY_PRECOMPACT_MODE`,
|
||||
`LLM_SECURITY_PRECOMPACT_MAX_BYTES`, `LLM_SECURITY_UPDATE_CHECK`,
|
||||
`LLM_SECURITY_MCP_CACHE_FILE`, `LLM_SECURITY_IDE_ROOTS`.
|
||||
|
||||
**Also removed:** `riskScoreV1()` in `scanners/lib/severity.mjs`, the v1
|
||||
sum-and-cap scoring formula `@deprecated` since v7.0.0 and kept for reference
|
||||
only. It had no callers in code or tests. `riskScore()` (v2, severity-dominated)
|
||||
is unchanged, so no score, band, or verdict moves.
|
||||
|
||||
---
|
||||
|
||||
## Playground (v7.6.0)
|
||||
|
||||
A single-file SPA at `playground/llm-security-playground.html` provides
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ Integrate llm-security into your CI/CD pipeline for automated security scanning
|
|||
|
||||
**The standalone CLI makes zero network calls by default.** All 14 scanners operate locally on your source code using Shannon entropy analysis, regex pattern matching, AST traversal, and git log parsing. No data is transmitted to any external service.
|
||||
|
||||
**Exception: supply-chain-recheck** — When scanning lockfiles for known vulnerabilities, this scanner optionally queries the [OSV.dev](https://osv.dev/) batch API. This sends only package names and versions (not source code) over HTTPS. To disable: set `LLM_SECURITY_SCR_OFFLINE=1`.
|
||||
**Exception: supply-chain-recheck** — When scanning lockfiles for known vulnerabilities, this scanner optionally queries the [OSV.dev](https://osv.dev/) batch API. This sends only package names and versions (not source code) over HTTPS. There is no kill-switch for this call today; block the host at the network layer if the run must be air-gapped.
|
||||
|
||||
**What about Claude Code integration?** The Claude Code plugin (hooks, agents, commands) uses AI models and sends data to Anthropic. These components are **not included** in the standalone CLI. When you run `npx llm-security scan`, only deterministic scanners execute.
|
||||
|
||||
|
|
@ -122,8 +122,11 @@ CLI flags always take precedence over policy file values.
|
|||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `LLM_SECURITY_SCR_OFFLINE=1` | Disable OSV.dev network calls in supply-chain-recheck |
|
||||
| `LLM_SECURITY_AUDIT_LOG=<path>` | Write structured JSONL audit trail (SIEM-ready) |
|
||||
| `LLM_SECURITY_PRECOMPACT_MODE` | `block\|warn\|off` for the PreCompact transcript scan (default `warn`) |
|
||||
| `LLM_SECURITY_UPDATE_CHECK=off` | Disable the daily update-check HTTP call |
|
||||
|
||||
The audit trail moved to the policy file in v8.0.0: set `audit.log_path` in
|
||||
`.llm-security/policy.json` instead of `LLM_SECURITY_AUDIT_LOG`.
|
||||
|
||||
## Exit Codes
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ AST-taint (AST) shells out to a PARSE-ONLY python3 helper (`scanners/lib/py-ast-
|
|||
Utility: `node scanners/lib/fs-utils.mjs <backup|restore|cleanup|tmppath> [args]`.
|
||||
|
||||
Lib: `sarif-formatter.mjs` — converts scan output to OASIS SARIF 2.1.0 format. Used by `--format sarif` flag.
|
||||
Lib: `audit-trail.mjs` — writes structured JSONL audit events (ISO 8601, OWASP tags, SIEM-ready). Env: `LLM_SECURITY_AUDIT_*`.
|
||||
Lib: `audit-trail.mjs` — writes structured JSONL audit events (ISO 8601, OWASP tags, SIEM-ready). Enabled by the `audit.log_path` key in `.llm-security/policy.json` (v8.0.0; was `LLM_SECURITY_AUDIT_LOG`).
|
||||
Lib: `policy-loader.mjs` — reads `.llm-security/policy.json` for distributable hook configuration. Includes `ci` section (`failOn`, `compact`) for CI/CD defaults. Defaults match hardcoded values.
|
||||
|
||||
**Standalone (8):** `posture-scanner.mjs` — deterministic posture assessment, 16 categories (incl. EU AI Act, NIST AI RMF, ISO 42001), <50ms. NOT in scan-orchestrator (meta-level, not code-level).
|
||||
|
|
|
|||
|
|
@ -23,16 +23,33 @@ in production. Deviations are fine, but the defaults here are the tested path.
|
|||
|
||||
| Variable | Default | Modes |
|
||||
|----------|---------|-------|
|
||||
| `LLM_SECURITY_INJECTION_MODE` | `block` | `block` — exit 2 on critical/high injection patterns. `warn` — advisory via systemMessage. `off` — disables scan. |
|
||||
| `LLM_SECURITY_TRIFECTA_MODE` | `warn` | `block` — exit 2 when lethal trifecta (untrusted input + sensitive data + exfiltration sink) detected. `warn` — advisory. `off` — disables. |
|
||||
| `LLM_SECURITY_PRECOMPACT_MODE` | `warn` | `block` — exit 2 on findings during PreCompact. `warn` — advisory via systemMessage. `off` — disables scan. |
|
||||
| `LLM_SECURITY_PRECOMPACT_MAX_BYTES` | `512000` | Tail size in bytes read from transcript for scanning. Higher values increase coverage at the cost of latency. |
|
||||
| `LLM_SECURITY_UPDATE_CHECK` | `on` | `off` disables the daily update-check HTTP call. |
|
||||
| `LLM_SECURITY_AUDIT_*` | unset | Audit trail configuration (destination, format, etc.) for SIEM-ready JSONL output. |
|
||||
|
||||
Apply env vars via shell profile, `.envrc`, or the host MDM. Do not write them
|
||||
into the repository.
|
||||
|
||||
**Removed in v8.0.0.** Four modes moved to `.llm-security/policy.json`, which
|
||||
travels with the repository instead of the operator's shell. Setting the old
|
||||
variable is now inert — it neither warns nor configures.
|
||||
|
||||
| Removed variable | Policy key | Default |
|
||||
|------------------|-----------|---------|
|
||||
| `LLM_SECURITY_INJECTION_MODE` | `injection.mode` | `block` — exit 2 on critical injection patterns. `warn` — advisory via systemMessage. `off` — disables scan. |
|
||||
| `LLM_SECURITY_TRIFECTA_MODE` | `trifecta.mode` | `warn` — advisory. `block` — exit 2 when the lethal trifecta (untrusted input + sensitive data + exfiltration sink) is detected. `off` — disables. |
|
||||
| `LLM_SECURITY_ESCALATION_WINDOW` | `trifecta.escalation_window` | `5` — calls between untrusted input and sub-agent delegation that still count as escalation-after-input. |
|
||||
| `LLM_SECURITY_AUDIT_LOG` | `audit.log_path` | unset — set a path to write the SIEM-ready JSONL audit trail. |
|
||||
| `LLM_SECURITY_DEPRECATION_QUIET` | *(none)* | Silenced the deprecation warning; removed with the warning. |
|
||||
|
||||
```json
|
||||
{
|
||||
"injection": { "mode": "block" },
|
||||
"trifecta": { "mode": "warn", "escalation_window": 5 },
|
||||
"audit": { "log_path": "/var/log/llm-security/audit.jsonl" }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Sandboxing
|
||||
|
|
@ -73,14 +90,16 @@ shell; avoid root, which disables user-namespace confinement.
|
|||
### 3.1 Start in warn mode
|
||||
|
||||
Every new integration of `llm-security` should begin with all modes set to
|
||||
`warn`. This yields advisories without breaking workflow, and lets the team
|
||||
calibrate false-positive rates against their actual repositories.
|
||||
`warn` — `injection.mode` and `trifecta.mode` in `.llm-security/policy.json`,
|
||||
`LLM_SECURITY_PRECOMPACT_MODE` in the environment. This yields advisories
|
||||
without breaking workflow, and lets the team calibrate false-positive rates
|
||||
against their actual repositories.
|
||||
|
||||
### 3.2 Promote to block after baselining
|
||||
|
||||
After a baseline period (typically 1-2 weeks), flip each mode to `block` in this
|
||||
order: `LLM_SECURITY_INJECTION_MODE`, `LLM_SECURITY_TRIFECTA_MODE`,
|
||||
`LLM_SECURITY_PRECOMPACT_MODE`. The injection hook is first because false
|
||||
order: `injection.mode`, `trifecta.mode` (both in `.llm-security/policy.json`),
|
||||
then `LLM_SECURITY_PRECOMPACT_MODE`. The injection hook is first because false
|
||||
positives there are the most visible; blocking comes last because the others
|
||||
build confidence.
|
||||
|
||||
|
|
@ -183,7 +202,9 @@ tier:
|
|||
|
||||
Verdict cutoffs (`BLOCK ≥65`, `WARNING ≥15`) are locked to the `riskBand()`
|
||||
boundaries so you can't get a "BLOCK / Medium band" contradiction. The legacy
|
||||
formula is kept as `riskScoreV1()` for reference only.
|
||||
v1 formula was kept as `riskScoreV1()` for reference through the v7 line and
|
||||
was removed in v8.0.0; see `CHANGELOG.md` for the v1 weights if you need them
|
||||
to re-derive an old score.
|
||||
|
||||
**CI impact:** Pipelines with `--fail-on high` keep working (the severity
|
||||
gate is unaffected). Pipelines with score-based thresholds need recalibration
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ preview after step 3.
|
|||
|
||||
- **`hooks/scripts/post-session-guard.mjs`** — the only hook invoked.
|
||||
Configurable via `policy.json` `trifecta.mode` (`block` / `warn` /
|
||||
`off`; default `warn`) or env var `LLM_SECURITY_TRIFECTA_MODE`.
|
||||
`off`; default `warn`) in `.llm-security/policy.json`.
|
||||
|
||||
This example uses `mode: warn` (default). In `block` mode the third
|
||||
call's advisory becomes a hard block (exit 2) and the agent action is
|
||||
|
|
@ -92,7 +92,7 @@ in a `finally` block before exiting. **Your real session state under
|
|||
Jensen-Shannon divergence, or volume-threshold advisories.
|
||||
Those have their own unit tests under `tests/lib/post-session-guard.*`.
|
||||
- This is deterministic detection. It does not exercise the
|
||||
`block`-mode exit-2 path — flip `LLM_SECURITY_TRIFECTA_MODE=block`
|
||||
`block`-mode exit-2 path — set `"trifecta": {"mode": "block"}`
|
||||
and re-run if you want to see the script fail at step 3.
|
||||
|
||||
## See also
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ The `systemMessage` payload from step 3 must contain:
|
|||
- The literal phrase `Rule of Two violation`
|
||||
- A list of evidence items under `Untrusted input:`, `Data access:`,
|
||||
`Exfil sink:` headings
|
||||
- A reference to `Set LLM_SECURITY_TRIFECTA_MODE=` for configuration
|
||||
- A reference to the `trifecta.mode` policy key for configuration
|
||||
- An OWASP tag mentioning `ASI01` or `ASI02`
|
||||
|
||||
Optional (depending on detail string and `policy.json` config):
|
||||
|
|
@ -32,7 +32,7 @@ Optional (depending on detail string and `policy.json` config):
|
|||
|
||||
## Audit-trail side effect
|
||||
|
||||
When `LLM_SECURITY_AUDIT_LOG` (or `policy.json` `audit.log_path`) is
|
||||
When the `policy.json` key `audit.log_path` is
|
||||
set, step 3 writes a JSONL event:
|
||||
|
||||
```json
|
||||
|
|
@ -48,7 +48,7 @@ set, step 3 writes a JSONL event:
|
|||
|
||||
The walkthrough does not configure the audit log — `writeAuditEvent`
|
||||
no-ops when no path is set. To observe the audit-trail behaviour,
|
||||
re-run with `LLM_SECURITY_AUDIT_LOG=/tmp/trifecta-audit.jsonl`.
|
||||
re-run with `"audit": {"log_path": "/tmp/trifecta-audit.jsonl"}` in `.llm-security/policy.json`.
|
||||
|
||||
## State file
|
||||
|
||||
|
|
|
|||
|
|
@ -90,13 +90,12 @@ Expected: `5 pass, 0 fail`.
|
|||
scope-hopping case the advisory is emitted before any network call.
|
||||
For the clean case it may attempt `npm view` — that runs against
|
||||
the public registry but is non-fatal if offline.
|
||||
- **Stage B (dep-auditor)**: runs offline by default. If the env
|
||||
var `LLM_SECURITY_OFFLINE=1` is unset, it may shell out to
|
||||
`npm audit --json --offline=false` for CVE enrichment, but the
|
||||
fixture has no real npm install, so audit returns nothing.
|
||||
- **Stage B (dep-auditor)**: runs offline by default. It may shell
|
||||
out to `npm audit --json --offline=false` for CVE enrichment, but
|
||||
the fixture has no real npm install, so audit returns nothing.
|
||||
|
||||
If you need a fully air-gapped run, set `LLM_SECURITY_OFFLINE=1`
|
||||
in the parent environment.
|
||||
There is no environment kill-switch for these calls. If you need a
|
||||
fully air-gapped run, block network egress for the process.
|
||||
|
||||
## OWASP / framework mapping
|
||||
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ Attacker injects instructions via external content (files, web pages, tool outpu
|
|||
| Prompt injection input scanning | Automated | `pre-prompt-inject-scan.mjs` detects CRITICAL/HIGH/MEDIUM injection patterns in user prompts | Hook file exists; MEDIUM advisory enabled |
|
||||
| Unicode Tag steganography detection | Automated | `string-utils.mjs` decodes U+E0000-E007F tags; `injection-patterns.mjs` escalates to CRITICAL/HIGH | `decodeUnicodeTags()` in normalization pipeline |
|
||||
| Bash evasion normalization | Automated | `bash-normalize.mjs` strips parameter expansion before pattern matching | `normalizeBashExpansion()` called by both bash hooks |
|
||||
| Rule of Two detection (block-mode opt-in) | Automated | `post-session-guard.mjs` detects trifecta (untrusted input + sensitive data + exfil); blocks only when `LLM_SECURITY_TRIFECTA_MODE=block` AND high-confidence trifecta is observed; default `warn` | `LLM_SECURITY_TRIFECTA_MODE` env var respected; block mode opt-in |
|
||||
| Rule of Two detection (block-mode opt-in) | Automated | `post-session-guard.mjs` detects trifecta (untrusted input + sensitive data + exfil); blocks only when the policy key `trifecta.mode` is `block` AND high-confidence trifecta is observed; default `warn` | `trifecta.mode` in `.llm-security/policy.json` respected; block mode opt-in |
|
||||
| Long-horizon monitoring | Automated | `post-session-guard.mjs` 100-call window + behavioral drift detection | Long-horizon window active alongside 20-call window |
|
||||
| HITL trap detection | Automated | `injection-patterns.mjs` HIGH patterns for approval urgency, summary suppression, scope minimization | HITL patterns present in HIGH_PATTERNS array |
|
||||
| Hybrid attack detection | Automated | `injection-patterns.mjs` HYBRID_PATTERNS for P2SQL, recursive injection, XSS | Hybrid patterns checked in tool output scanning |
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ Research summary for the llm-security plugin. Documents what the field has learn
|
|||
- Sensitive path patterns need expansion as new sensitive files emerge
|
||||
|
||||
**Plugin controls:**
|
||||
- `post-session-guard.mjs`: `LLM_SECURITY_TRIFECTA_MODE=block|warn|off`
|
||||
- `post-session-guard.mjs`: policy key `trifecta.mode` = `block|warn|off`
|
||||
- Block mode: exit 2 for MCP-concentrated trifecta or sensitive path + exfil
|
||||
- Default warn mode preserves backward compatibility
|
||||
- **Gap:** Rule of Two is approximate — false positives possible for legitimate multi-tool workflows
|
||||
|
|
|
|||
|
|
@ -9,9 +9,6 @@ export const SEVERITY = Object.freeze({
|
|||
INFO: 'info',
|
||||
});
|
||||
|
||||
// Legacy weights — used only by riskScoreV1() for backwards-compat reference.
|
||||
const SEVERITY_WEIGHTS_V1 = { critical: 25, high: 10, medium: 4, low: 1, info: 0 };
|
||||
|
||||
/**
|
||||
* Calculate aggregate risk score from severity counts (v2 model — v7.0.0+).
|
||||
*
|
||||
|
|
@ -56,24 +53,6 @@ export function riskScore(counts) {
|
|||
return Math.round(Math.min(100, base));
|
||||
}
|
||||
|
||||
/**
|
||||
* Legacy v1 risk score formula — kept for diff/comparison only.
|
||||
* Not exported in production paths; reference for CI re-calibration.
|
||||
*
|
||||
* @deprecated Since v7.0.0. Use riskScore() instead. Kept for diff/comparison only — not used in production paths.
|
||||
* @param {{ critical: number, high: number, medium: number, low: number, info: number }} counts
|
||||
* @returns {number} 0-100 capped score (sum-and-cap model)
|
||||
*/
|
||||
export function riskScoreV1(counts) {
|
||||
const raw =
|
||||
(counts.critical || 0) * SEVERITY_WEIGHTS_V1.critical +
|
||||
(counts.high || 0) * SEVERITY_WEIGHTS_V1.high +
|
||||
(counts.medium || 0) * SEVERITY_WEIGHTS_V1.medium +
|
||||
(counts.low || 0) * SEVERITY_WEIGHTS_V1.low +
|
||||
(counts.info || 0) * SEVERITY_WEIGHTS_V1.info;
|
||||
return Math.min(raw, 100);
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive verdict from severity counts and risk score (v7.0.0 thresholds).
|
||||
* Aligned to v2 riskBand cutoffs so verdict and band are co-monotonic:
|
||||
|
|
|
|||
|
|
@ -1122,9 +1122,22 @@ async function checkRuleOfTwo(projectRoot, hooksJson) {
|
|||
if (hookActive) {
|
||||
const scriptPath = join(projectRoot, 'hooks', 'scripts', 'post-session-guard.mjs');
|
||||
const content = await readText(scriptPath);
|
||||
if (content && /TRIFECTA_MODE/i.test(content)) {
|
||||
// What this category measures is whether enforcement is *configurable*, not
|
||||
// what the constant holding it is called. v8.0.0 moved the mode to the
|
||||
// policy.json key `trifecta.mode`; a pre-v8 vendored guard still resolving
|
||||
// LLM_SECURITY_TRIFECTA_MODE is equally configurable and still passes.
|
||||
const readsPolicyMode =
|
||||
content && /getPolicyValue\w*\(\s*['"]trifecta['"]\s*,\s*['"]mode['"]/.test(content);
|
||||
const readsLegacyEnv = content && /LLM_SECURITY_TRIFECTA_MODE/.test(content);
|
||||
if (readsPolicyMode) {
|
||||
hasTrifectaMode = true;
|
||||
evidence.push('TRIFECTA_MODE configurable: yes');
|
||||
evidence.push('Enforcement mode configurable: policy.json trifecta.mode');
|
||||
} else if (readsLegacyEnv) {
|
||||
hasTrifectaMode = true;
|
||||
evidence.push(
|
||||
'Enforcement mode configurable: LLM_SECURITY_TRIFECTA_MODE (pre-v8.0.0 env-var; ' +
|
||||
'migrate to the policy.json key trifecta.mode)'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1146,10 +1159,10 @@ async function checkRuleOfTwo(projectRoot, hooksJson) {
|
|||
scanner: 'PST',
|
||||
severity: SEVERITY.MEDIUM,
|
||||
title: 'Session guard lacks configurable trifecta mode',
|
||||
description: 'post-session-guard does not support LLM_SECURITY_TRIFECTA_MODE (block/warn/off).',
|
||||
description: 'post-session-guard resolves no enforcement mode (block/warn/off) from the policy.json key trifecta.mode.',
|
||||
file: 'hooks/scripts/post-session-guard.mjs',
|
||||
owasp: 'ASI02',
|
||||
recommendation: 'Upgrade to v5.0 session guard with configurable enforcement mode.',
|
||||
recommendation: 'Upgrade to a session guard that reads trifecta.mode from .llm-security/policy.json.',
|
||||
}));
|
||||
return { status: STATUS.PARTIAL, findings, evidence };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
#!/usr/bin/env node
|
||||
// post-session-guard.mjs — Runtime trifecta detection (Rule of Two)
|
||||
// v5.0: Configurable TRIFECTA_MODE (block|warn|off), long-horizon 100-call window,
|
||||
// behavioral drift via Jensen-Shannon divergence
|
||||
// v5.0: configurable enforcement mode (block|warn|off), long-horizon 100-call
|
||||
// window, behavioral drift via Jensen-Shannon divergence
|
||||
// v8.0.0: mode comes from the policy.json key trifecta.mode
|
||||
import { readFileSync, appendFileSync } from 'node:fs';
|
||||
import { getPolicyValue } from '../../scanners/lib/policy-loader.mjs';
|
||||
|
||||
const TRIFECTA_MODE = (process.env.LLM_SECURITY_TRIFECTA_MODE || 'warn').toLowerCase();
|
||||
const TRIFECTA_MODE = String(getPolicyValue('trifecta', 'mode', 'warn')).toLowerCase();
|
||||
const SLIDING_WINDOW = 20;
|
||||
const LONG_HORIZON_WINDOW = 100;
|
||||
|
||||
|
|
|
|||
|
|
@ -242,3 +242,26 @@ describe('B11 — the env-var deprecation mechanism is gone from source', () =>
|
|||
assert.deepEqual(offenders, [], `removed env-vars are still read:\n${offenders.join('\n')}`);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// riskScoreV1 — the other v7.3.0 deprecation shipping in the same major
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('B11 — riskScoreV1 is removed', () => {
|
||||
it('severity.mjs no longer exports riskScoreV1', async () => {
|
||||
const mod = await import('../../scanners/lib/severity.mjs');
|
||||
assert.equal(
|
||||
mod.riskScoreV1,
|
||||
undefined,
|
||||
'the v1 sum-and-cap formula was @deprecated in v7.3.0 with zero consumers'
|
||||
);
|
||||
});
|
||||
|
||||
it('riskScore (v2) is untouched and still severity-dominated', async () => {
|
||||
const { riskScore } = await import('../../scanners/lib/severity.mjs');
|
||||
const oneCritical = riskScore({ critical: 1, high: 0, medium: 0, low: 0, info: 0 });
|
||||
const manyLow = riskScore({ critical: 0, high: 0, medium: 0, low: 50, info: 0 });
|
||||
assert.ok(oneCritical >= 70 && oneCritical <= 95, `one critical -> ${oneCritical}`);
|
||||
assert.ok(manyLow < oneCritical, 'v1 collapsed this case to 100; v2 must not');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
129
tests/scanners/posture-trifecta-mode.test.mjs
Normal file
129
tests/scanners/posture-trifecta-mode.test.mjs
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
// posture-trifecta-mode.test.mjs — B11: posture category 12 (Rule of Two)
|
||||
// recognises a policy-era session guard.
|
||||
//
|
||||
// Pre-v8 the check was `/TRIFECTA_MODE/i` over the hook source. That matched
|
||||
// the *identifier* `TRIFECTA_MODE`, not the configuration mechanism, so a hook
|
||||
// that resolves the mode from `.llm-security/policy.json` without naming a
|
||||
// constant that way scored PARTIAL despite being strictly more configurable.
|
||||
// With LLM_SECURITY_TRIFECTA_MODE removed in v8.0.0, that regex would push
|
||||
// every correctly-migrated project off PASS.
|
||||
|
||||
import { describe, it, beforeEach, afterEach } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, cpSync } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/posture-scanner.mjs';
|
||||
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
const GRADE_A_FIXTURE = resolve(__dirname, '../fixtures/posture-scan/grade-a-project');
|
||||
|
||||
const GUARD_REL = join('hooks', 'scripts', 'post-session-guard.mjs');
|
||||
|
||||
// A guard that is configurable, but names nothing `TRIFECTA_MODE`.
|
||||
const POLICY_ERA_GUARD = `#!/usr/bin/env node
|
||||
// post-session-guard.mjs — Runtime trifecta detection (Rule of Two)
|
||||
// v8.0.0: enforcement mode comes from .llm-security/policy.json.
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { getPolicyValue } from '../../scanners/lib/policy-loader.mjs';
|
||||
|
||||
const mode = String(getPolicyValue('trifecta', 'mode', 'warn')).toLowerCase();
|
||||
|
||||
const input = JSON.parse(readFileSync('/dev/stdin', 'utf-8'));
|
||||
if (mode === 'off') {
|
||||
process.stdout.write(JSON.stringify({ decision: 'allow' }));
|
||||
process.exit(0);
|
||||
}
|
||||
const LONG_HORIZON_WINDOW = 100;
|
||||
process.stdout.write(JSON.stringify({ decision: 'allow', window: LONG_HORIZON_WINDOW }));
|
||||
`;
|
||||
|
||||
// A guard with no configurable mode at all — hardcoded enforcement.
|
||||
const UNCONFIGURABLE_GUARD = `#!/usr/bin/env node
|
||||
// post-session-guard.mjs — Runtime trifecta detection, no configuration.
|
||||
import { readFileSync } from 'node:fs';
|
||||
const input = JSON.parse(readFileSync('/dev/stdin', 'utf-8'));
|
||||
const LONG_HORIZON_WINDOW = 100;
|
||||
process.stdout.write(JSON.stringify({ decision: 'allow', window: LONG_HORIZON_WINDOW }));
|
||||
`;
|
||||
|
||||
let root = null;
|
||||
|
||||
/** Copy of grade-a-project whose session guard is replaced with `source`. */
|
||||
function projectWithGuard(source) {
|
||||
root = mkdtempSync(join(tmpdir(), 'llmsec-posture-trifecta-'));
|
||||
cpSync(GRADE_A_FIXTURE, root, { recursive: true });
|
||||
mkdirSync(join(root, 'hooks', 'scripts'), { recursive: true });
|
||||
writeFileSync(join(root, GUARD_REL), source);
|
||||
return root;
|
||||
}
|
||||
|
||||
function ruleOfTwo(result) {
|
||||
return result.categories.find((c) => c.id === 12);
|
||||
}
|
||||
|
||||
/** Category findings live on the top-level `findings` array, not on the category. */
|
||||
function ruleOfTwoFindings(result) {
|
||||
return result.findings.filter((f) => /trifecta mode|Rule of Two/i.test(f.title));
|
||||
}
|
||||
|
||||
describe('posture category 12 — Rule of Two mode detection (B11)', () => {
|
||||
afterEach(() => {
|
||||
if (root) rmSync(root, { recursive: true, force: true });
|
||||
root = null;
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
resetCounter();
|
||||
});
|
||||
|
||||
it('PASSes a policy-era guard that reads trifecta.mode from policy.json', async () => {
|
||||
const result = await scan(projectWithGuard(POLICY_ERA_GUARD));
|
||||
const cat = ruleOfTwo(result);
|
||||
assert.equal(
|
||||
cat.status,
|
||||
'PASS',
|
||||
'a guard configured through policy.json is configurable; the identifier name is not the mechanism'
|
||||
);
|
||||
assert.ok(
|
||||
cat.evidence.some((e) => /trifecta\.mode|policy\.json/i.test(e)),
|
||||
`expected evidence naming the policy key, got: ${JSON.stringify(cat.evidence)}`
|
||||
);
|
||||
});
|
||||
|
||||
it('PASSes a legacy guard still using the pre-v8 env-var', async () => {
|
||||
// Third-party projects vendor their own hook copy and may not have migrated.
|
||||
// Their enforcement mode is still configurable, so the category still holds.
|
||||
const legacy = POLICY_ERA_GUARD.replace(
|
||||
"String(getPolicyValue('trifecta', 'mode', 'warn'))",
|
||||
"String(process.env.LLM_SECURITY_TRIFECTA_MODE || 'warn')"
|
||||
);
|
||||
const result = await scan(projectWithGuard(legacy));
|
||||
assert.equal(ruleOfTwo(result).status, 'PASS');
|
||||
});
|
||||
|
||||
it('does not PASS a guard with no configurable mode', async () => {
|
||||
const result = await scan(projectWithGuard(UNCONFIGURABLE_GUARD));
|
||||
const cat = ruleOfTwo(result);
|
||||
assert.equal(cat.status, 'PARTIAL', 'hardcoded enforcement is not a configurable mode');
|
||||
assert.ok(
|
||||
ruleOfTwoFindings(result).some((f) => /configurable trifecta mode/i.test(f.title)),
|
||||
'expected the PARTIAL finding to name the missing configurability'
|
||||
);
|
||||
});
|
||||
|
||||
it('the PARTIAL finding recommends the policy key, not a removed env-var', async () => {
|
||||
const result = await scan(projectWithGuard(UNCONFIGURABLE_GUARD));
|
||||
const f = ruleOfTwoFindings(result).find((x) => /configurable trifecta mode/i.test(x.title));
|
||||
assert.ok(f, 'expected the PARTIAL finding');
|
||||
const text = `${f.description} ${f.recommendation}`;
|
||||
assert.doesNotMatch(
|
||||
text,
|
||||
/LLM_SECURITY_TRIFECTA_MODE/,
|
||||
'must not tell the user to set an env-var that v8 ignores'
|
||||
);
|
||||
assert.match(text, /trifecta\.mode/, 'should name the policy key');
|
||||
});
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue