open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

feat(egress): decode-rescan feeds base64 plaintext to secret-egress (review MINOR)

Browse source 
Output gate step 3 now runs scan_secret_egress over every decoded base64
blob's plaintext, not only scan_lexicon. A base64-wrapped credential that
formerly vanished (decode fed the lexicon, which has no secret patterns)
now surfaces as decoded:egress:* carrying the blob offset. Evidence stays
length-only, so the decoded finding never leaks the secret value.

Hex-wrapped secrets remain a documented honest-limit (entropy exposes
decoded plaintext for base64 only). README honest-limits + CLAUDE.md
Kontekst updated; 3 tests added (347 passed, was 344).
This commit is contained in:
Kjell Tore Guttormsen 2026-07-15 07:11:29 +02:00
commit f4e89d2885
4 changed files with 62 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

27
src/llm_ingestion_guard/output.py
View file

@ -15,12 +15,13 @@ input-side scanners do not cover:
2. :func:`~llm_ingestion_guard.entropy.scan_entropy` over the output — encoded /
high-entropy carrier blobs.
3. **Decode-and-rescan** — every base64 blob ``entropy`` decoded to printable
text is fed back through ``scan_lexicon``. This is what turns "a blob is
present" into "an injection is hidden *inside* this blob". Findings from the
decoded plaintext are re-labelled ``decoded:<label>`` and carry the blob's
offset in the original text. (Scope: base64 only — ``entropy`` exposes
decoded plaintext for base64, not hex; a base64-*wrapped secret* is a
documented gap, since decode-rescan feeds the lexicon, not the egress set.)
text is fed back through ``scan_lexicon`` **and** ``scan_secret_egress``.
This is what turns "a blob is present" into "an injection — or a wrapped
credential — is hidden *inside* this blob". Findings from the decoded
plaintext are re-labelled ``decoded:<label>`` (e.g.
``decoded:egress:aws-access-key-id``) and carry the blob's offset in the
original text. (Scope: base64 only — ``entropy`` exposes decoded plaintext
for base64, not hex; a hex-*wrapped* secret stays a documented honest-limit.)
4. **Secret / credential egress** (:func:`scan_secret_egress`, OWASP LLM02 —
Sensitive Information Disclosure) — cloud/provider API keys, PEM private-key
headers, DB connection strings, JWTs, and labelled password/secret/api-key
@ -286,11 +287,17 @@ def scan_output(
entropy_result = scan_entropy(scan_text, source)
report.extend(entropy_result.report.findings)
# 3. Decode-and-rescan: run the lexicon over each decoded blob's plaintext,
# re-labelled so the finding is attributable to the hiding blob.
# 3. Decode-and-rescan: run the lexicon AND the egress scanner over each
# decoded blob's plaintext, re-labelled so the finding is attributable to
# the hiding blob. Feeding the egress set here (not only the lexicon) is
# what catches a base64-*wrapped* secret: the plaintext credential reaches
# scan_secret_egress as a decoded:egress:* finding instead of vanishing.
# (Scope: base64 only — entropy exposes decoded plaintext for base64, not
# hex; a hex-wrapped secret stays a documented honest-limit.)
for blob in entropy_result.decoded:
hidden = scan_lexicon(blob.decoded, source, max_scan_chars)
for finding in hidden.findings:
hidden = scan_lexicon(blob.decoded, source, max_scan_chars).findings
leaked = scan_secret_egress(blob.decoded, source).findings
for finding in [*hidden, *leaked]:
report.add(
replace(
finding,