open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

refactor(calibration): consolidate tunable thresholds into calibration.py

Browse source 
Session D: move every calibration constant (entropy floors 5.4/128, 5.1/64,
4.7/40 + shape floors; MAX_SCAN_CHARS; rot13-min; cognitive-load lengths
2000/2500; disposition ranks; active-content severities) into one documented
calibration.py, so a parallel Node/TS port can mirror exactly the same numbers.

Pure refactor, zero behavior change: calibration is a leaf module (imports only
report.Severity) that entropy/lexicon/disposition/active_content now source
their thresholds from. MAX_SCAN_CHARS is re-exported from lexicon so output.py
and existing callers are unaffected. The 347 pre-existing tests pass unmodified;
new test_calibration.py freezes the values and asserts each detector actually
reads its threshold from calibration (identity-checked, not a dead copy).
This commit is contained in:
Kjell Tore Guttormsen 2026-07-15 09:44:53 +02:00
commit ee402e4ea8
6 changed files with 212 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

15
src/llm_ingestion_guard/active_content.py
View file

@ -37,7 +37,8 @@ from __future__ import annotations
import re
from .report import Finding, Report, Severity, Source
from .calibration import ACTIVE_CONTENT_SEVERITY as _SEVERITY
from .report import Finding, Report, Source
# --- URL defang (shared primitive) -------------------------------------------
# Rewrite a URL to a form no renderer will resolve, while keeping it readable.
@ -138,15 +139,9 @@ def _always(url: str) -> bool:
return True
_SEVERITY = {
# zero-click auto-fetch / auto-execute -> HIGH; click-required -> MEDIUM.
"markdown-image": Severity.HIGH,
"markdown-link": Severity.MEDIUM,
"reference-link": Severity.MEDIUM,
"autolink": Severity.MEDIUM,
"raw-html": Severity.HIGH,
"data-uri": Severity.HIGH,
}
# Per-construct severities (_SEVERITY, imported above) live in `calibration` —
# zero-click auto-fetch/execute -> HIGH, click-required -> MEDIUM — the Node port
# shares them.
def scan_active_content(text: str, source: Source = Source.OUTPUT) -> Report: