open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

refactor(calibration): consolidate tunable thresholds into calibration.py

Browse source 
Session D: move every calibration constant (entropy floors 5.4/128, 5.1/64,
4.7/40 + shape floors; MAX_SCAN_CHARS; rot13-min; cognitive-load lengths
2000/2500; disposition ranks; active-content severities) into one documented
calibration.py, so a parallel Node/TS port can mirror exactly the same numbers.

Pure refactor, zero behavior change: calibration is a leaf module (imports only
report.Severity) that entropy/lexicon/disposition/active_content now source
their thresholds from. MAX_SCAN_CHARS is re-exported from lexicon so output.py
and existing callers are unaffected. The 347 pre-existing tests pass unmodified;
new test_calibration.py freezes the values and asserts each detector actually
reads its threshold from calibration (identity-checked, not a dead copy).
This commit is contained in:
Kjell Tore Guttormsen 2026-07-15 09:44:53 +02:00
commit ee402e4ea8
6 changed files with 212 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

24
src/llm_ingestion_guard/entropy.py
View file

@ -39,19 +39,21 @@ import math
import re
from dataclasses import dataclass, field
from .calibration import (
ENTROPY_BASE64_FLOOR_LEN as _BASE64_FLOOR_LEN,
ENTROPY_CRITICAL_H as _CRITICAL_H,
ENTROPY_CRITICAL_LEN as _CRITICAL_LEN,
ENTROPY_HEX_FLOOR_LEN as _HEX_FLOOR_LEN,
ENTROPY_HIGH_H as _HIGH_H,
ENTROPY_HIGH_LEN as _HIGH_LEN,
ENTROPY_MEDIUM_H as _MEDIUM_H,
ENTROPY_MEDIUM_LEN as _MEDIUM_LEN,
)
from .report import Finding, Report, Severity, Source
# --- length-calibrated entropy thresholds (bits/char, min length) -----------
# Empirically calibrated against real distributions in the seed scanner:
# plaintext prose H ~3.5-4.2; base64 len64 H ~5.2; base64 len128 H ~5.6.
_CRITICAL_H, _CRITICAL_LEN = 5.4, 128
_HIGH_H, _HIGH_LEN = 5.1, 64
_MEDIUM_H, _MEDIUM_LEN = 4.7, 40
# Shape-floor lengths: structured encodings that are suspicious by size even
# when their entropy sits below the classification floor.
_BASE64_FLOOR_LEN = 100
_HEX_FLOOR_LEN = 64
# Length-calibrated entropy thresholds (bits/char, min length) and shape-floor
# lengths now live in `calibration` — the single source of truth the Node port
# shares. See that module for the calibration rationale.
# Candidate extraction: maximal runs of the base64 alphabet (hex is a subset),
# with optional trailing padding. Bounded class, no nested quantifier -> linear.