llm-security/docs/plans/skillspector-gap-analysis.md

SkillSpector Gap Analysis & Build Brief

Date: 2026-06-20 Plugin version at time of writing: 7.7.2 Status: Analysis complete, awaiting implementation go-ahead. No code written yet. Reference: NVIDIA/SkillSpector — https://github.com/nvidia/skillspector (Apache-2.0, Python, v2.2.3 as researched)

---

1. Framing

SkillSpector and llm-security solve partially different problems, and that determines what "missing" actually means:

• SkillSpector is a static pre-install gate for skill bundles (SKILL.md + code + deps). Python, pip install, runs in CI, emits pass/fail. Its depth is in static code analysis of the bundle itself — real Python AST, variable-level taint, YARA signatures.
• llm-security is a broad runtime + posture defense for an entire Claude Code setup: 9 hooks, multi-session behavioral analysis, drift, baseline/diff, watch, IDE scanning, 7 package ecosystems, git forensics, MCP live-inspect.

We are broader and own a runtime layer they lack entirely. They are deeper on a few static techniques we either lack or do more coarsely. "Fill what's missing" therefore means closing a small number of sharp holes in our static analysis — not becoming SkillSpector.

The goal for each item below is an llm-security-native implementation (zero external npm dependencies, integrated with our decode/normalize pipeline and policy system) — not a port.

---

2. Verification log

Per the verification duty, every claim this brief depends on was checked against actual source — SkillSpector via raw source on main, our own plugin via direct file reads. Verdicts below are load-bearing; the priorities in §6 follow from them.

SkillSpector (all VERIFIED against source)

Claim	Verdict	Source file
4 YARA rulesets (malware, webshells, cryptominers, hacktools)	VERIFIED	src/skillspector/yara_rules/*.yar
Python ast-based behavioral analysis (exec/eval/subprocess)	VERIFIED	nodes/analyzers/behavioral_ast.py
Variable-level taint tracking via AST (source→sink)	VERIFIED	nodes/analyzers/behavioral_taint_tracking.py
Trigger-abuse rules TR1 (broad), TR2 (shadow), TR3 (baiting)	VERIFIED	nodes/analyzers/static_patterns_supply_chain.py
0–100 risk score + exit 0/1/2 at threshold 50	VERIFIED	cli.py (risk_score > 50 → Exit(1))
Manifest-vs-code least privilege LP1–LP4	VERIFIED	nodes/analyzers/mcp_least_privilege.py
SARIF 2.1.0 output	VERIFIED	cli.py + sarif_models.py

llm-security (our actual status)

Capability	Verdict	Evidence
#1 Known-malware signature engine	MISSING	No YARA/webshell/cryptominer/signature engine in scanners/, scanners/lib/, knowledge/. Entropy/taint detect shape, not known-bad identity.
#2 Real AST analysis	MISSING	taint-tracer.mjs:1–12 self-documents regex matching, ~70% recall, no scope/cross-file. No AST library imported anywhere (package.json has zero deps; no acorn/babel/espree/tree-sitter).
#3 Trigger/activation-abuse detection	MISSING	permission-mapper.mjs has exactly 6 checks (purpose-vs-tools, dangerous combos, ghost hooks, haiku model, over-privilege, missing-doc). None inspect name/description for broad/baiting/shadowing triggers.
#4 Per-scan 0–100 risk score	HAVE	severity.mjs riskScore(counts) → output.aggregate.risk_score; logged in orchestrator.
#4 CI exit-code gating	HAVE	scan-orchestrator.mjs:324–333: --fail-on <level> exits 1 over threshold; default exits 2/1/0 on BLOCK/WARNING/ALLOW. Policy key ci.failOn.

Correction to the initial analysis: #4 was provisionally listed as a gap. Verification REFUTES that — it is fully implemented. #4 is therefore reclassified from "build" to "already covered, optional polish" (§5). It is intentionally not a build item.

---

3. What we already beat them on (scope guard — do not rebuild)

Runtime hooks (9), multi-session lethal-trifecta + Jensen-Shannon behavioral drift, MCP description drift with immutable baseline, baseline/diff, watch mode, IDE-extension scanning with VSIX sandbox, 7 package ecosystems (vs their PyPI+npm), git forensics, pre-compact transcript scan, evidence-package sanitization for remote repos, skill-registry fingerprinting. None of these exist in SkillSpector. No inspiration to draw there.

---

4. Build items

Three genuine gaps. Each lists the our-own-way differentiator, concrete plug-in points (verified from the orchestrator), framework mapping, risks, and per-item verification criteria.

Shared plug-in facts (verified):

• New deterministic scanner = export scan(targetPath, discovery) (add requiresPriorResults: true for cross-scanner correlation), import it in scan-orchestrator.mjs, append to the SCANNERS array (scan-orchestrator.mjs:101–125).
• Findings use finding() (scanners/lib/output.mjs:30–44): scanner (prefix), severity, title, description, file, line, evidence, owasp, recommendation. IDs auto-format as DS-<PREFIX>-NNN.
• Configurable thresholds: add a section to DEFAULT_POLICY and read via getPolicyValue('section','key',default). User policy at <project>/.llm-security/policy.json.

4.1 — Signature engine (#1) · proposed prefix SIG

Gap: We detect shape (entropy, obfuscation, dangerous flows) but never match against known-bad identities — reverse shells, PHP/JSP/ASPX webshells, cryptominer stratum strings, offensive-tool references. SkillSpector ships 4 YARA rulesets for exactly this.

Our-own-way (better, not a copy):

• Pure-Node signature engine, zero deps. Rules live as data in knowledge/signatures.json (string/regex/codepoint matchers grouped by family: webshell, reverse-shell, cryptominer, hacktool, c2). No yara-python dependency.
• Match against the decode/normalize pipeline output, not just raw bytes. This is the differentiator: YARA matches raw file content, so a base64- or hex-obfuscated webshell evades it. We already have normalizeForScan() (Unicode/hex/URL/base64/HTML-entity/zero-width/BIDI/homoglyph/rot13 decode). Running signatures against every decoded variant catches obfuscated known-malware SkillSpector misses.
• Optional acceleration, graceful degradation. If a yara binary is present on PATH, optionally shell out to it for the heavy lifting (we already shell out to npm audit/git/pip); otherwise the pure-Node engine is authoritative. Never a hard dependency.

Plug-in: new scanners/signature-scanner.mjs, prefix SIG, registered in SCANNERS. Rule data in knowledge/signatures.json. Policy section signatures (enable/disable families, custom rule path).

Frameworks: LLM03 (supply chain) primary; webshell/C2 hits also LLM02 (exfiltration); ASI04.

Risks: signature corpus is maintenance surface and can go stale — keep it small, high-confidence, and family-scoped; document provenance of each rule. False positives on security tooling/education — mitigate with context down-weighting (§5, confidence) and path exclusions (knowledge/, tests).

Verification criteria:

• Fixture containing a known PHP webshell pattern → SIG flags it (HIGH/CRITICAL).
• Same payload base64-wrapped → still flagged (decode-pipeline integration; this is what beats raw YARA).
• Benign file with the word "shell" in prose → not flagged.
• yara binary absent → scanner still runs and flags via pure-Node path.

4.2 — AST analysis (#2) · augments TNT, proposed behavioral prefix AST

Gap: taint-tracer.mjs is regex/line-based: ~70% recall, no scope awareness, no cross-file, no flow through arrays/objects/closures (self-documented at taint-tracer.mjs:1–12). SkillSpector parses a real Python AST and tracks taint at variable level with high confidence. This is our deepest methodology gap.

Constraint: zero npm deps (package.json is empty of dependencies). A real JS AST would require a parser dependency. This shapes the recommendation.

Our-own-way — staged, dep-free:

• Phase A (recommended first): Python AST via a shipped helper. Ship scanners/lib/py-ast-taint.py invoked as python3 <helper> <file> (same optional-python pattern as dep-auditor's pip audit). The helper does ast.parse + ast.walk, runs source→sink taint at variable level, and emits JSON findings back to Node. Graceful degradation: if python3 is absent, fall back to the existing regex tracer. Rationale for going Python-first: skill bundles' executable payloads are most often Python, it is exactly where SkillSpector's AST advantage is real and where our regex tracer is weakest, and it is achievable with no dependency.
• Phase B (smaller, optional): bracket-aware scope pass for JS/TS. Improve the existing JS regex tracer with balanced-bracket/function-scope tokenization to cut the "variable named input taints the whole file" class of false positives — incremental, still dep-free, no full parser.
• Phase C (deferred): full JS/TS AST. Requires either vendoring a micro-parser (maintenance cost) or accepting a dependency (breaks the zero-dep invariant). Defer until A+B are measured; decide then.

Plug-in: Python findings flow through the existing taint scanner result (or a new AST prefix if we want them separable in reports). Add policy section ast (enable, python path override). The helper emits the same finding() fields so the orchestrator/SARIF path is unchanged.

Frameworks: LLM01 (injection sinks), LLM02 (exfiltration sinks) — same mapping as TNT, higher precision and recall.

Risks: python3 availability varies (handled by fallback). Subprocess to an interpreter on untrusted code — the helper must only parse (ast.parse never executes), never exec/eval/import the target. State this as a hard invariant and test it.

Verification criteria:

• Python fixture: k = os.environ["AWS_SECRET"]; requests.post(url, data=k) → flagged CRITICAL (creds→net), where the regex tracer's recall is shaky on the intermediate variable.
• Python fixture with input reused in two functions, only one tainted → only the tainted use flagged (scope awareness the regex tracer lacks).
• Helper invoked on a file that calls os.system("rm -rf /") → parses without executing it (parse-only invariant; assert no side effects).
• python3 removed from PATH → orchestrator still completes via regex fallback (no crash).

4.3 — Trigger / activation-abuse scanner (#3) · proposed prefix TRG

Gap: Cleanest uncovered hole and the most skill-native. Claude skills/commands auto-activate on their description. A skill can be engineered to hijack activation — a single common-word trigger, a maximally-activating description ("use this for anything", "always", "all messages"), or a name that shadows a built-in command. SkillSpector covers this as TR1–TR3; we have nothing. permission-mapper's 6 checks never read name/description for activation-surface abuse.

Our-own-way: dedicated scanners/trigger-abuse.mjs, reusing our existing frontmatter parsing across commands/, agents/, skills/:

• TRG-broad: single common-word or ≤2-char trigger names; description that claims universal applicability.
• TRG-baiting: maximally-activating phrasing ("anything", "everything", "always", "all files/messages", "whenever") — word lists in policy, extensible.
• TRG-shadow: name colliding with built-in command/tool names (configurable built-in list).
• Differentiator vs SkillSpector: run the description through normalizeForScan() first, so a baiting description hidden behind homoglyphs/zero-width still trips — consistent with the rest of our pipeline.

Plug-in: new scanner registered in SCANNERS, prefix TRG, policy section triggers (word lists, built-in names, severity per rule).

Frameworks: AST04 (Scope Creep) primary; LLM06 (Excessive Agency). Cross-reference knowledge/owasp-skills-top10.md.

Risks: legitimate broad utilities exist (e.g. a general formatter) — keep severity advisory (MEDIUM) by default, allowlist via policy, and require the combination of broad-name + universal-claim for the higher tier to cut false positives.

Verification criteria:

• Skill with name: run and description: "use for anything" → TRG-shadow + TRG-baiting fire.
• Skill with a specific, scoped description → no finding.
• Baiting phrase obfuscated with zero-width chars → still flagged (normalize-first differentiator).
• New scanner appears in scan-orchestrator JSON and SARIF output with correct owasp tag.

---

5. #4 — already covered (no build)

Per §2, the 0–100 risk score and CI exit-code gate exist. Optional, low-cost polish only:

• Per-finding confidence. The finding() schema has no confidence field; SkillSpector uses confidence pervasively and adjusts it by context (e.g. P5 down-weights safety/education contexts). Adding an optional confidence field and feeding it into riskScore() would (a) let the new signature/AST/trigger scanners express uncertainty and (b) reduce false-positive weight on docs/test/fixture contexts. This is the one cross-cutting enhancement that connects #1/#2/#3 to the existing score. Small, optional, do-after.
• Document the CI story. docs/ci-cd-guide.md exists; confirm --fail-on + risk_score + SARIF upload are documented end-to-end. Doc-only.

---

6. Priority & sequencing

Recommended order (value × independence × risk):

1. #3 Trigger-abuse (TRG) — smallest, lowest-risk, fully dep-free, most skill-native, reuses existing frontmatter + normalize machinery. Best first win; establishes the new-scanner pattern.
2. #1 Signature engine (SIG) — high value, dep-free, the decode-pipeline integration is a clear "better than YARA" story. Main cost is curating a small high-confidence rule corpus.
3. #2 AST, Phase A (Python helper) — deepest methodology lift but most design surface (subprocess, parse-only invariant, fallback). Do after the new-scanner pattern is proven by #3/#1. Phases B/C gated on measuring A.

Each item is independent and shippable on its own; no item blocks another. TDD per the Iron Law — failing fixture test first for every rule.

---

7. Out of scope / deliberate non-goals

• Harmful-content moderation (SkillSpector P5: toxic substances / dangerous actions). Content safety is not our mission — we are a code/config security scanner, not a content moderator. Excluded.
• LLM semantic analyzers (SkillSpector SSD/SDI/SQP). We already cover semantic analysis through our agents (skill-scanner-agent, etc.); not reimplementing as inline LLM nodes.
• Standalone pip install distribution. Our identity is the runtime Claude Code plugin; scan-orchestrator.mjs already works as a CLI. Repackaging for non-Claude CI is deprioritized.
• Manifest-vs-code least privilege (SkillSpector LP1–LP4). Partially covered by permission-mapper (intent-vs-tools, over-privilege). The under-declared dimension (code reaches network but tool not declared) is a possible later extension of permission-mapper, not part of this brief's three items.

---

8. Consolidated verification criteria

Acceptance for the build phase (each maps to a fixture test):

• SIG flags a known webshell fixture, including a base64-obfuscated variant; ignores benign prose; runs without a yara binary.
• AST (Python helper) flags creds→network with an intermediate variable; respects function scope; never executes target code; orchestrator survives missing python3.
• TRG flags shadow+baiting skills, including obfuscated baiting; passes scoped skills; appears in JSON + SARIF with correct OWASP tags.
• All three appear in scan-orchestrator.mjs SCANNERS, emit standard finding() objects, and are policy-configurable via .llm-security/policy.json.
• No new npm dependency added (package.json dependencies remain empty).
• Existing test suite (1820+) stays green; new rules covered by failing-first tests.

---

Sources

• NVIDIA/SkillSpector source (main), verified file-by-file 2026-06-20 — paths in §2.
• NVIDIA Technical Blog: developer.nvidia.com/blog (verified-agent-skills capability governance).
• OWASP Agentic Skills Top 10 (AST) — knowledge/owasp-skills-top10.md.
• llm-security internals verified by direct read: scan-orchestrator.mjs, scanners/taint-tracer.mjs, scanners/permission-mapper.mjs, scanners/lib/{output,severity,policy-loader}.mjs.