open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 6 Activity

feat(output): decode hex-wrapped credentials at the persist gate

Browse source 
Gap 1 closed. The output gate finds runs of 32+ hex digits
(calibration.HEX_WRAP_MIN_LEN), decodes both byte alignments, keeps text
that is at least 80% printable (the base64 bar, now one shared helper in
entropy) and runs the credential patterns on it. A hex-wrapped AWS key
surfaces as decoded:egress:aws-access-key-id with "in hex run @ N" as
evidence; the value is never in it. Credentials only: a hex-wrapped
injection is not decoded.

Measured before and after on the same trees: all 3583 clean documents
(three FP populations and the 78 pinned OKF concepts) are identical line
for line under both presets (disposition, labels, severity, count,
offset); OKF import stays 47/31. They and 569 READMEs hold 2403 hex runs
of 32+ digits, none of which decodes to text. scan_output CPU 122.9 s
before, 123.4 s after; one solid hex run at the 1M cap 4.2 s -> 5.8 s.

14 new tests (10 red for the right reason first), 6 of 6 mutations red.
Matrix 146/146 + 5/5 gaps, suite 1274 passed, ReDoS sweep 169 patterns.
This commit is contained in:
Kjell Tore Guttormsen 2026-11-01 17:06:44 +01:00
commit 0eab4345bf
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
14 changed files with 265 additions and 40 deletions
Download patch file Download diff file Expand all files Collapse all files

6
tests/test_docs_measurement_scripts.py
View file

@ -272,13 +272,13 @@ def test_redos_sweep_floor_and_flag_match_the_published_numbers():
assert "2.6 flag threshold" in text
def test_redos_sweep_collector_covers_168_patterns_across_11_tables():
# The count docs/LIMITATIONS.md carries as "all 168 compiled patterns across
def test_redos_sweep_collector_covers_169_patterns_across_11_tables():
# The count docs/LIMITATIONS.md carries as "all 169 compiled patterns across
# all eleven regex-bearing modules". A pattern added or removed in `src/`
# without re-measuring would drift the doc's claim silently otherwise.
assert len(redos_sweep.TABLES) == 11
total = sum(len(collect()) for collect in redos_sweep.TABLES.values())
assert total == 168
assert total == 169
def test_redos_sweep_collector_walks_a_dict_of_patterns():