llm-security/docs/plans/skillspector-gap-analysis.md

# SkillSpector Gap Analysis & Build Brief

**Date:** 2026-06-20
**Plugin version at time of writing:** 7.7.2
**Status:** Analysis complete, awaiting implementation go-ahead. No code written yet.
**Reference:** NVIDIA/SkillSpector — https://github.com/nvidia/skillspector (Apache-2.0, Python, v2.2.3 as researched)

---

## 1. Framing

SkillSpector and llm-security solve partially different problems, and that determines what "missing"
actually means:

- **SkillSpector** is a static **pre-install gate** for skill bundles (`SKILL.md` + code + deps).
  Python, `pip install`, runs in CI, emits pass/fail. Its depth is in *static code analysis of the
  bundle itself* — real Python AST, variable-level taint, YARA signatures.
- **llm-security** is a broad **runtime + posture defense** for an entire Claude Code setup: 9 hooks,
  multi-session behavioral analysis, drift, baseline/diff, watch, IDE scanning, 7 package ecosystems,
  git forensics, MCP live-inspect.

We are **broader and own a runtime layer they lack entirely**. They are **deeper on a few static
techniques** we either lack or do more coarsely. "Fill what's missing" therefore means closing a
small number of sharp holes in our static analysis — not becoming SkillSpector.

The goal for each item below is an llm-security-native implementation (zero external npm
dependencies, integrated with our decode/normalize pipeline and policy system) — not a port.

---

## 2. Verification log

Per the verification duty, every claim this brief depends on was checked against actual source —
SkillSpector via raw source on `main`, our own plugin via direct file reads. Verdicts below are
load-bearing; the priorities in §6 follow from them.

### SkillSpector (all VERIFIED against source)

| Claim | Verdict | Source file |
|---|---|---|
| 4 YARA rulesets (malware, webshells, cryptominers, hacktools) | VERIFIED | `src/skillspector/yara_rules/*.yar` |
| Python `ast`-based behavioral analysis (exec/eval/subprocess) | VERIFIED | `nodes/analyzers/behavioral_ast.py` |
| Variable-level taint tracking via AST (source→sink) | VERIFIED | `nodes/analyzers/behavioral_taint_tracking.py` |
| Trigger-abuse rules TR1 (broad), TR2 (shadow), TR3 (baiting) | VERIFIED | `nodes/analyzers/static_patterns_supply_chain.py` |
| 0–100 risk score + exit 0/1/2 at threshold 50 | VERIFIED | `cli.py` (`risk_score > 50 → Exit(1)`) |
| Manifest-vs-code least privilege LP1–LP4 | VERIFIED | `nodes/analyzers/mcp_least_privilege.py` |
| SARIF 2.1.0 output | VERIFIED | `cli.py` + `sarif_models.py` |

### llm-security (our actual status)

| Capability | Verdict | Evidence |
|---|---|---|
| #1 Known-malware signature engine | **MISSING** | No YARA/webshell/cryptominer/signature engine in `scanners/`, `scanners/lib/`, `knowledge/`. Entropy/taint detect *shape*, not known-bad *identity*. |
| #2 Real AST analysis | **MISSING** | `taint-tracer.mjs:1–12` self-documents regex matching, ~70% recall, no scope/cross-file. No AST library imported anywhere (`package.json` has zero deps; no acorn/babel/espree/tree-sitter). |
| #3 Trigger/activation-abuse detection | **MISSING** | `permission-mapper.mjs` has exactly 6 checks (purpose-vs-tools, dangerous combos, ghost hooks, haiku model, over-privilege, missing-doc). None inspect `name`/`description` for broad/baiting/shadowing triggers. |
| #4 Per-scan 0–100 risk score | **HAVE** | `severity.mjs` `riskScore(counts)` → `output.aggregate.risk_score`; logged in orchestrator. |
| #4 CI exit-code gating | **HAVE** | `scan-orchestrator.mjs:324–333`: `--fail-on <level>` exits 1 over threshold; default exits 2/1/0 on BLOCK/WARNING/ALLOW. Policy key `ci.failOn`. |

**Correction to the initial analysis:** #4 was provisionally listed as a gap. Verification REFUTES that
— it is fully implemented. #4 is therefore reclassified from "build" to "already covered, optional
polish" (§5). It is intentionally *not* a build item.

---

## 3. What we already beat them on (scope guard — do not rebuild)

Runtime hooks (9), multi-session lethal-trifecta + Jensen-Shannon behavioral drift, MCP description
drift with immutable baseline, baseline/diff, watch mode, IDE-extension scanning with VSIX sandbox,
7 package ecosystems (vs their PyPI+npm), git forensics, pre-compact transcript scan, evidence-package
sanitization for remote repos, skill-registry fingerprinting. None of these exist in SkillSpector.
No inspiration to draw there.

---

## 4. Build items

Three genuine gaps. Each lists the *our-own-way* differentiator, concrete plug-in points (verified
from the orchestrator), framework mapping, risks, and per-item verification criteria.

Shared plug-in facts (verified):
- New deterministic scanner = export `scan(targetPath, discovery)` (add `requiresPriorResults: true`
  for cross-scanner correlation), import it in `scan-orchestrator.mjs`, append to the `SCANNERS` array
  (`scan-orchestrator.mjs:101–125`).
- Findings use `finding()` (`scanners/lib/output.mjs:30–44`): `scanner` (prefix), `severity`, `title`,
  `description`, `file`, `line`, `evidence`, `owasp`, `recommendation`. IDs auto-format as `DS-<PREFIX>-NNN`.
- Configurable thresholds: add a section to `DEFAULT_POLICY` and read via
  `getPolicyValue('section','key',default)`. User policy at `<project>/.llm-security/policy.json`.

### 4.1 — Signature engine (#1) · proposed prefix `SIG`

**Gap:** We detect *shape* (entropy, obfuscation, dangerous flows) but never match against *known-bad
identities* — reverse shells, PHP/JSP/ASPX webshells, cryptominer stratum strings, offensive-tool
references. SkillSpector ships 4 YARA rulesets for exactly this.

**Our-own-way (better, not a copy):**
- **Pure-Node signature engine, zero deps.** Rules live as data in `knowledge/signatures.json`
  (string/regex/codepoint matchers grouped by family: webshell, reverse-shell, cryptominer, hacktool,
  c2). No `yara-python` dependency.
- **Match against the decode/normalize pipeline output, not just raw bytes.** This is the differentiator:
  YARA matches raw file content, so a base64- or hex-obfuscated webshell evades it. We already have
  `normalizeForScan()` (Unicode/hex/URL/base64/HTML-entity/zero-width/BIDI/homoglyph/rot13 decode).
  Running signatures against every decoded variant catches obfuscated known-malware SkillSpector misses.
- **Optional acceleration, graceful degradation.** If a `yara` binary is present on PATH, optionally
  shell out to it for the heavy lifting (we already shell out to `npm audit`/`git`/`pip`); otherwise
  the pure-Node engine is authoritative. Never a hard dependency.

**Plug-in:** new `scanners/signature-scanner.mjs`, prefix `SIG`, registered in `SCANNERS`. Rule data in
`knowledge/signatures.json`. Policy section `signatures` (enable/disable families, custom rule path).

**Frameworks:** LLM03 (supply chain) primary; webshell/C2 hits also LLM02 (exfiltration); ASI04.

**Risks:** signature corpus is maintenance surface and can go stale — keep it small, high-confidence,
and family-scoped; document provenance of each rule. False positives on security tooling/education —
mitigate with context down-weighting (§5, confidence) and path exclusions (`knowledge/`, tests).

**Verification criteria:**
- Fixture containing a known PHP webshell pattern → `SIG` flags it (HIGH/CRITICAL).
- Same payload base64-wrapped → still flagged (decode-pipeline integration; this is what beats raw YARA).
- Benign file with the word "shell" in prose → not flagged.
- `yara` binary absent → scanner still runs and flags via pure-Node path.

### 4.2 — AST analysis (#2) · augments `TNT`, proposed behavioral prefix `AST`

**Gap:** `taint-tracer.mjs` is regex/line-based: ~70% recall, no scope awareness, no cross-file, no
flow through arrays/objects/closures (self-documented at `taint-tracer.mjs:1–12`). SkillSpector parses
a real Python AST and tracks taint at variable level with high confidence. This is our deepest
methodology gap.

**Constraint:** zero npm deps (`package.json` is empty of dependencies). A real JS AST would require a
parser dependency. This shapes the recommendation.

**Our-own-way — staged, dep-free:**
- **Phase A (recommended first): Python AST via a shipped helper.** Ship `scanners/lib/py-ast-taint.py`
  invoked as `python3 <helper> <file>` (same optional-python pattern as `dep-auditor`'s `pip audit`).
  The helper does `ast.parse` + `ast.walk`, runs source→sink taint at variable level, and emits JSON
  findings back to Node. Graceful degradation: if `python3` is absent, fall back to the existing regex
  tracer. *Rationale for going Python-first:* skill bundles' executable payloads are most often Python,
  it is exactly where SkillSpector's AST advantage is real and where our regex tracer is weakest, and
  it is achievable with no dependency.
- **Phase B (smaller, optional): bracket-aware scope pass for JS/TS.** Improve the existing JS regex
  tracer with balanced-bracket/function-scope tokenization to cut the "variable named `input` taints
  the whole file" class of false positives — incremental, still dep-free, no full parser.
- **Phase C (deferred): full JS/TS AST.** Requires either vendoring a micro-parser (maintenance cost)
  or accepting a dependency (breaks the zero-dep invariant). Defer until A+B are measured; decide then.

**Plug-in:** Python findings flow through the existing `taint` scanner result (or a new `AST` prefix if
we want them separable in reports). Add policy section `ast` (enable, python path override). The helper
emits the same `finding()` fields so the orchestrator/SARIF path is unchanged.

**Frameworks:** LLM01 (injection sinks), LLM02 (exfiltration sinks) — same mapping as `TNT`, higher
precision and recall.

**Risks:** python3 availability varies (handled by fallback). Subprocess to an interpreter on untrusted
code — the helper must only *parse* (`ast.parse` never executes), never `exec`/`eval`/import the target.
State this as a hard invariant and test it.

**Verification criteria:**
- Python fixture: `k = os.environ["AWS_SECRET"]; requests.post(url, data=k)` → flagged CRITICAL (creds→net),
  where the regex tracer's recall is shaky on the intermediate variable.
- Python fixture with `input` reused in two functions, only one tainted → only the tainted use flagged
  (scope awareness the regex tracer lacks).
- Helper invoked on a file that calls `os.system("rm -rf /")` → parses without executing it (parse-only
  invariant; assert no side effects).
- `python3` removed from PATH → orchestrator still completes via regex fallback (no crash).

### 4.3 — Trigger / activation-abuse scanner (#3) · proposed prefix `TRG`

**Gap:** Cleanest uncovered hole and the most skill-native. Claude skills/commands auto-activate on
their `description`. A skill can be engineered to hijack activation — a single common-word trigger, a
maximally-activating description ("use this for anything", "always", "all messages"), or a `name` that
shadows a built-in command. SkillSpector covers this as TR1–TR3; we have nothing. `permission-mapper`'s
6 checks never read `name`/`description` for activation-surface abuse.

**Our-own-way:** dedicated `scanners/trigger-abuse.mjs`, reusing our existing frontmatter parsing across
`commands/`, `agents/`, `skills/`:
- **TRG-broad:** single common-word or ≤2-char trigger names; description that claims universal
  applicability.
- **TRG-baiting:** maximally-activating phrasing ("anything", "everything", "always", "all
  files/messages", "whenever") — word lists in policy, extensible.
- **TRG-shadow:** `name` colliding with built-in command/tool names (configurable built-in list).
- Differentiator vs SkillSpector: run the description through `normalizeForScan()` first, so a baiting
  description hidden behind homoglyphs/zero-width still trips — consistent with the rest of our pipeline.

**Plug-in:** new scanner registered in `SCANNERS`, prefix `TRG`, policy section `triggers`
(word lists, built-in names, severity per rule).

**Frameworks:** AST04 (Scope Creep) primary; LLM06 (Excessive Agency). Cross-reference
`knowledge/owasp-skills-top10.md`.

**Risks:** legitimate broad utilities exist (e.g. a general formatter) — keep severity advisory
(MEDIUM) by default, allowlist via policy, and require the *combination* of broad-name + universal-claim
for the higher tier to cut false positives.

**Verification criteria:**
- Skill with `name: run` and `description: "use for anything"` → TRG-shadow + TRG-baiting fire.
- Skill with a specific, scoped description → no finding.
- Baiting phrase obfuscated with zero-width chars → still flagged (normalize-first differentiator).
- New scanner appears in `scan-orchestrator` JSON and SARIF output with correct `owasp` tag.

---

## 5. #4 — already covered (no build)

Per §2, the 0–100 risk score and CI exit-code gate exist. Optional, low-cost polish only:

- **Per-finding `confidence`.** The `finding()` schema has no `confidence` field; SkillSpector uses
  confidence pervasively and adjusts it by context (e.g. P5 down-weights safety/education contexts).
  Adding an optional `confidence` field and feeding it into `riskScore()` would (a) let the new
  signature/AST/trigger scanners express uncertainty and (b) reduce false-positive weight on
  docs/test/fixture contexts. This is the one cross-cutting enhancement that connects #1/#2/#3 to the
  existing score. Small, optional, do-after.
- **Document the CI story.** `docs/ci-cd-guide.md` exists; confirm `--fail-on` + `risk_score` +
  SARIF upload are documented end-to-end. Doc-only.

---

## 6. Priority & sequencing

Recommended order (value × independence × risk):

1. **#3 Trigger-abuse (`TRG`)** — smallest, lowest-risk, fully dep-free, most skill-native, reuses
   existing frontmatter + normalize machinery. Best first win; establishes the new-scanner pattern.
2. **#1 Signature engine (`SIG`)** — high value, dep-free, the decode-pipeline integration is a clear
   "better than YARA" story. Main cost is curating a small high-confidence rule corpus.
3. **#2 AST, Phase A (Python helper)** — deepest methodology lift but most design surface (subprocess,
   parse-only invariant, fallback). Do after the new-scanner pattern is proven by #3/#1. Phases B/C
   gated on measuring A.

Each item is independent and shippable on its own; no item blocks another. TDD per the Iron Law —
failing fixture test first for every rule.

---

## 7. Out of scope / deliberate non-goals

- **Harmful-content moderation (SkillSpector P5: toxic substances / dangerous actions).** Content
  safety is not our mission — we are a code/config security scanner, not a content moderator. Excluded.
- **LLM semantic analyzers (SkillSpector SSD/SDI/SQP).** We already cover semantic analysis through our
  agents (`skill-scanner-agent`, etc.); not reimplementing as inline LLM nodes.
- **Standalone `pip install` distribution.** Our identity is the runtime Claude Code plugin;
  `scan-orchestrator.mjs` already works as a CLI. Repackaging for non-Claude CI is deprioritized.
- **Manifest-vs-code least privilege (SkillSpector LP1–LP4).** Partially covered by `permission-mapper`
  (intent-vs-tools, over-privilege). The under-declared dimension (code reaches network but tool not
  declared) is a *possible later extension* of `permission-mapper`, not part of this brief's three items.

---

## 8. Consolidated verification criteria

Acceptance for the build phase (each maps to a fixture test):

- [ ] `SIG` flags a known webshell fixture, including a base64-obfuscated variant; ignores benign prose;
      runs without a `yara` binary.
- [ ] `AST` (Python helper) flags creds→network with an intermediate variable; respects function scope;
      never executes target code; orchestrator survives missing `python3`.
- [ ] `TRG` flags shadow+baiting skills, including obfuscated baiting; passes scoped skills; appears in
      JSON + SARIF with correct OWASP tags.
- [ ] All three appear in `scan-orchestrator.mjs` `SCANNERS`, emit standard `finding()` objects, and are
      policy-configurable via `.llm-security/policy.json`.
- [ ] No new npm dependency added (`package.json` dependencies remain empty).
- [ ] Existing test suite (1820+) stays green; new rules covered by failing-first tests.

---

## Sources

- NVIDIA/SkillSpector source (`main`), verified file-by-file 2026-06-20 — paths in §2.
- NVIDIA Technical Blog: developer.nvidia.com/blog (verified-agent-skills capability governance).
- OWASP Agentic Skills Top 10 (AST) — `knowledge/owasp-skills-top10.md`.
- llm-security internals verified by direct read: `scan-orchestrator.mjs`, `scanners/taint-tracer.mjs`,
  `scanners/permission-mapper.mjs`, `scanners/lib/{output,severity,policy-loader}.mjs`.