fix(s51): close 2 review MINORs — non-UTF-8 skip + affected_codes parity

Both live in the tolerant inbox-read path and both contradicted the module's
own "skipped, never raised" / loader-parity docstrings (S5.1 review: ALLOW,
2 MINOR, non-gating):

- UnicodeDecodeError (a *.json hand-saved in Latin-1 with Norwegian æ/ø/å is
  invalid UTF-8) now SKIPPED in hitl._load_json_dict AND
  verdicts.load_verdicts_from_dir — was: crashed pending/route with a raw
  traceback (a ValueError subclass, caught by neither OSError nor
  JSONDecodeError). Fix symmetric across both readers.
- hitl._inbox_verdict_ids now skips a non-iterable affected_codes exactly as
  load_verdicts_from_dir does (frozenset() raises TypeError) — was: marked the
  proposal judged on key-presence alone → a silent false-negative in the
  operator pending queue. hitl-only: verdicts is the correct reference.

TDD: 3 RED-then-green (both UnicodeDecodeError twins + the parity gap) + 1
ground-truth pin locking the loader side so the parity cannot rot.

Gate: pytest → 389 passed, 4 skipped (was 385); ruff check + format clean; mypy clean.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-15 20:59:01 +02:00
commit 7e86896bc8
5 changed files with 131 additions and 4 deletions

View file

@ -111,8 +111,11 @@ def _inbox_verdict_ids(verdict_dir: str) -> set[str]:
"""Collect the ``id`` of every inbox verdict file that ``load_verdicts_from_dir`` WOULD accept —
the same tolerant predicate mirrored inline: a dict carrying all ``_REQUIRED_VERDICT_KEYS``, a
``decision`` in the binary vocabulary ``{approved, rejected}``, and a ``proposal_features`` dict
carrying the inner keys ``verdict_from_dict`` reads. A file that would be skipped or raise in the
real loader is skipped here too, so ``pending`` never treats it as a delivered dom."""
carrying the inner keys ``verdict_from_dict`` reads AND an ``affected_codes`` that
``frozenset(...)`` accepts (a non-iterable ``null``/int/float/bool makes the real loader's
``frozenset(pf['affected_codes'])`` raise ``TypeError``, so it is skipped here too). A file that
would be skipped or raise in the real loader is skipped here too, so ``pending`` never treats it as
a delivered dom."""
directory = Path(verdict_dir)
if not directory.is_dir():
return set()
@ -126,6 +129,12 @@ def _inbox_verdict_ids(verdict_dir: str) -> set[str]:
features = data.get("proposal_features")
if not isinstance(features, dict) or not _REQUIRED_FEATURE_KEYS <= features.keys():
continue
# Mirror verdict_from_dict EXACTLY: frozenset(affected_codes) raises TypeError on a
# non-iterable (null/int/float/bool), which the real loader skips — so skip it here too.
try:
frozenset(features["affected_codes"])
except TypeError:
continue
ids.add(data["id"])
return ids
@ -237,7 +246,9 @@ def _load_json_dict(file: Path) -> dict[str, Any] | None:
unreadable / non-JSON / non-object file is skipped by every reader here)."""
try:
data = json.loads(file.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
# UnicodeDecodeError: a *.json file hand-saved in Latin-1 (Norwegian æ/ø/å) is invalid UTF-8 —
# skip it, not raise (a ValueError subclass, so neither OSError nor JSONDecodeError catches it).
return None
return data if isinstance(data, dict) else None

View file

@ -212,7 +212,10 @@ def load_verdicts_from_dir(
for file in files:
try:
data = json.loads(file.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
# UnicodeDecodeError: a *.json file hand-saved in Latin-1 (Norwegian æ/ø/å) is invalid
# UTF-8 — a per-file tolerant skip, not a raise (a ValueError subclass, caught by neither
# OSError nor JSONDecodeError). Kept in lockstep with hitl._load_json_dict.
continue
if not isinstance(data, dict) or not _REQUIRED_VERDICT_KEYS <= data.keys():
continue