feat(grounding): SourceGroundingCheck protocol + pass-through default — the semantic-poisoning seam (TDD) [skip-docs]
Module 10 of the build order — interface only in the stdlib core (PLAN §67/§94, BRIEF §7/§10). lexicon and entropy see surface signals; they are structurally blind to semantic poisoning — a factually false or subtly biased claim, in clean prose, carrying no suspicious token. Catching that needs grounding against a source of truth (embedding classifier, retrieval check, LLM judge) — a model call, and models never live in this core. So this module ships exactly two things and no detector logic: - SourceGroundingCheck: the text -> Report protocol a [judge] implementation must satisfy to plug in. runtime_checkable for a coarse callable-vs-not isinstance gate; a conforming impl may be a plain function or a stateful callable holding a retriever/client — the grounding source of truth is the impl's concern, never a seam parameter. Its findings flow through disposition like any other detector's, with zero grounding-specific plumbing. - no_grounding_check (bound as DEFAULT_GROUNDING_CHECK): the pass-through default returning an empty Report. The honest scope note made structural — semantic poisoning is not solved at the text layer, and the core does not pretend to. Design choice, tested: the default returns NO finding (not an INFO "unchecked" marker). An INFO finding would flip every artifact to found and trip the quarantine_default floor in disposition, quarantining every upload for a check that never ran. The seam stays silent; the honesty lives in the contract. 8 new tests (pass-through empties, silence on semantic poison, protocol conformance incl. a stateful judge stub, and a composition test proving a plugged-in check's HIGH finding fails secure through disposition unchanged); 169 green total. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
This commit is contained in:
parent
9b75bf0bb7
commit
1e63643157
2 changed files with 177 additions and 0 deletions
71
src/llm_ingestion_guard/grounding.py
Normal file
71
src/llm_ingestion_guard/grounding.py
Normal file
|
|
@ -0,0 +1,71 @@
|
||||||
|
"""grounding — the seam for semantic / factual poisoning (BRIEF §7/§10, PLAN §67/§94).
|
||||||
|
|
||||||
|
Interface only in the stdlib core. :mod:`lexicon` and :mod:`entropy` see
|
||||||
|
*surface* signals — known injection phrasing, obfuscation, invisible carriers.
|
||||||
|
They are structurally blind to **semantic poisoning**: a factually false or
|
||||||
|
subtly biased claim, in clean prose, carrying no suspicious token. Catching that
|
||||||
|
means grounding the content against a source of truth — an embedding classifier,
|
||||||
|
a retrieval check, an LLM judge — which is a model call, and models never live
|
||||||
|
in this core (BRIEF §7: ML detectors are optional, pluggable extras behind the
|
||||||
|
``[judge]`` extra).
|
||||||
|
|
||||||
|
So this module ships exactly two things and no detector logic:
|
||||||
|
|
||||||
|
* :class:`SourceGroundingCheck` — the ``text -> Report`` protocol a ``[judge]``
|
||||||
|
implementation must satisfy to plug into a pipeline. Its findings then flow
|
||||||
|
through :mod:`disposition` like any other detector's, with no
|
||||||
|
grounding-specific plumbing.
|
||||||
|
* :func:`no_grounding_check` (bound as :data:`DEFAULT_GROUNDING_CHECK`) — the
|
||||||
|
shipped default: a pass-through returning an empty Report. The honest scope
|
||||||
|
note made structural — *semantic poisoning is not solved at the text layer*,
|
||||||
|
and the deterministic core does not pretend to. A pipeline that needs it
|
||||||
|
installs a ``[judge]`` implementation; one that does not gets a safe no-op that
|
||||||
|
never breaks composition.
|
||||||
|
|
||||||
|
The default returns *no* finding — not an INFO "unchecked" marker — on purpose.
|
||||||
|
An INFO finding would flip every artifact to :attr:`Report.found` and trip the
|
||||||
|
``quarantine_default`` floor in :mod:`disposition`, quarantining every upload for
|
||||||
|
a check that never ran. The seam stays silent; the honesty lives here in the
|
||||||
|
contract, not in a finding that pollutes the gate.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Protocol, runtime_checkable
|
||||||
|
|
||||||
|
from .report import Report, Source
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class SourceGroundingCheck(Protocol):
|
||||||
|
"""A semantic-grounding detector: ``text -> Report`` (BRIEF §7/§10).
|
||||||
|
|
||||||
|
Structurally identical to the other detectors so its findings compose with
|
||||||
|
:mod:`disposition` unchanged. A conforming ``[judge]`` implementation may be
|
||||||
|
a plain function or a stateful callable (an object holding a model client or
|
||||||
|
a trusted-corpus retriever) — the grounding *source of truth* is the
|
||||||
|
implementation's concern, never a parameter of this seam.
|
||||||
|
|
||||||
|
Runtime-checkable for a coarse ``isinstance`` structural gate (callable vs
|
||||||
|
not); the intended ``source`` keyword and return type are enforced by a type
|
||||||
|
checker, not at runtime.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __call__(self, text: str, *, source: Source = Source.OUTPUT) -> Report:
|
||||||
|
...
|
||||||
|
|
||||||
|
|
||||||
|
def no_grounding_check(text: str, *, source: Source = Source.OUTPUT) -> Report:
|
||||||
|
"""The pass-through default (PLAN §94): return an empty :class:`Report`.
|
||||||
|
|
||||||
|
The stdlib core cannot judge semantics, so it reports nothing rather than
|
||||||
|
guessing. This is deliberate and **not** an assertion that ``text`` is
|
||||||
|
semantically clean — only that no grounding check ran. Install a ``[judge]``
|
||||||
|
:class:`SourceGroundingCheck` to actually verify factual grounding.
|
||||||
|
"""
|
||||||
|
return Report()
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_GROUNDING_CHECK: SourceGroundingCheck = no_grounding_check
|
||||||
|
"""The grounding check the core ships: :func:`no_grounding_check`. A pipeline
|
||||||
|
rebinds this to a ``[judge]`` :class:`SourceGroundingCheck` when it needs
|
||||||
|
semantic grounding."""
|
||||||
106
tests/test_grounding.py
Normal file
106
tests/test_grounding.py
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
"""Tests for grounding — the semantic-poisoning seam (BRIEF §7/§10, PLAN §67/§94).
|
||||||
|
|
||||||
|
Interface only in the stdlib core. lexicon and entropy see *surface* signals;
|
||||||
|
they are structurally blind to a factually false claim in clean prose. Detecting
|
||||||
|
that needs a model call, which never lives in this core — so grounding ships a
|
||||||
|
`text -> Report` protocol plus a pass-through default, and a `[judge]`
|
||||||
|
implementation plugs in behind an extra. These tests pin the seam: the default
|
||||||
|
is silent (and honest about why), a conforming impl satisfies the protocol, and
|
||||||
|
its findings compose through :mod:`disposition` unchanged.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from llm_ingestion_guard.grounding import (
|
||||||
|
DEFAULT_GROUNDING_CHECK,
|
||||||
|
SourceGroundingCheck,
|
||||||
|
no_grounding_check,
|
||||||
|
)
|
||||||
|
from llm_ingestion_guard.report import Finding, Report, Severity, Source
|
||||||
|
from llm_ingestion_guard.disposition import (
|
||||||
|
Disposition,
|
||||||
|
PRESET_USER_UPLOAD,
|
||||||
|
decide,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# A factually false claim in clean prose: no injection token, no obfuscation, no
|
||||||
|
# invisible carrier. lexicon and entropy — and therefore the deterministic core
|
||||||
|
# — are structurally blind to it. Only a grounding check against a source of
|
||||||
|
# truth can catch it.
|
||||||
|
_SEMANTIC_POISON = "The capital of France is Berlin."
|
||||||
|
|
||||||
|
|
||||||
|
# --- the pass-through default ----------------------------------------------
|
||||||
|
|
||||||
|
def test_no_grounding_check_returns_empty_report():
|
||||||
|
report = no_grounding_check("any content", source=Source.OUTPUT)
|
||||||
|
assert isinstance(report, Report)
|
||||||
|
assert not report.found
|
||||||
|
assert report.findings == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_grounding_check_default_source_is_output_and_accepts_source():
|
||||||
|
# source is part of the seam signature; the default ignores it but accepts it.
|
||||||
|
assert not no_grounding_check("x").found # default source
|
||||||
|
assert not no_grounding_check("x", source=Source.INPUT).found
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_grounding_check_is_silent_on_semantic_poisoning():
|
||||||
|
# The honest scope note, made a test: semantic poisoning is NOT solved at the
|
||||||
|
# text layer, and the core does not pretend to. An empty report here is "no
|
||||||
|
# check ran", not "verified clean".
|
||||||
|
assert not no_grounding_check(_SEMANTIC_POISON).found
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_grounding_check_is_the_no_op():
|
||||||
|
assert DEFAULT_GROUNDING_CHECK is no_grounding_check
|
||||||
|
|
||||||
|
|
||||||
|
# --- the protocol (structural seam) ----------------------------------------
|
||||||
|
|
||||||
|
def test_no_grounding_check_satisfies_protocol():
|
||||||
|
assert isinstance(no_grounding_check, SourceGroundingCheck)
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_callable_does_not_satisfy_protocol():
|
||||||
|
assert not isinstance(object(), SourceGroundingCheck)
|
||||||
|
assert not isinstance("not a check", SourceGroundingCheck)
|
||||||
|
|
||||||
|
|
||||||
|
class _FactJudge:
|
||||||
|
"""A stand-in `[judge]` SourceGroundingCheck: a stateful callable that flags
|
||||||
|
one known-false claim. Real impls hold a retriever / model client; the
|
||||||
|
grounding source of truth is the impl's concern, never a seam parameter."""
|
||||||
|
|
||||||
|
def __call__(self, text: str, *, source: Source = Source.OUTPUT) -> Report:
|
||||||
|
report = Report()
|
||||||
|
if "capital of france is berlin" in text.lower():
|
||||||
|
report.add(Finding(
|
||||||
|
label="grounding:factual-contradiction",
|
||||||
|
severity=Severity.HIGH,
|
||||||
|
source=source,
|
||||||
|
detector="grounding",
|
||||||
|
evidence="claim contradicts trusted source",
|
||||||
|
))
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def test_stateful_judge_impl_satisfies_protocol():
|
||||||
|
assert isinstance(_FactJudge(), SourceGroundingCheck)
|
||||||
|
|
||||||
|
|
||||||
|
# --- the seam is real: findings flow through disposition -------------------
|
||||||
|
|
||||||
|
def test_grounding_findings_compose_with_disposition():
|
||||||
|
judge = _FactJudge()
|
||||||
|
|
||||||
|
# the core misses it; the plugged-in judge catches the same text.
|
||||||
|
assert not no_grounding_check(_SEMANTIC_POISON).found
|
||||||
|
report = judge(_SEMANTIC_POISON, source=Source.OUTPUT)
|
||||||
|
assert report.found
|
||||||
|
assert report.findings[0].detector == "grounding"
|
||||||
|
|
||||||
|
# and its finding disposes like any other detector's: HIGH under an untrusted
|
||||||
|
# upload halts. No grounding-specific plumbing in disposition.
|
||||||
|
result = decide(report, PRESET_USER_UPLOAD)
|
||||||
|
assert result.disposition is Disposition.FAIL_SECURE
|
||||||
Loading…
Add table
Add a link
Reference in a new issue