is_within_dir called os.path.realpath OUTSIDE its try block, so a path with
an embedded null byte (which makes realpath raise ValueError, not OSError)
leaked that ValueError to callers. Via safe_resolve -> okf._load_file (which
catches only PathSecurityError) it propagated uncaught out of
navigate_bundle, breaking the OKF SPEC §4 guarantee (okf.py:8-9) that a
broken cross-link is silently skipped, never raised: an index link like
`](a\x00b.md)` has no `/`, slips past the same-dir pre-filter, and reached
path resolution.
Move both realpath calls inside the existing try so an uncanonicalisable
path is treated as not-within (fail-closed): safe_resolve raises
PathSecurityError -> _load_file returns None -> the link is skipped. Fixes
the class for both callers of the seam (navigate_bundle + retrieve).
Tests (load-bearing — RED before, GREEN after):
- test_navigate_skips_null_byte_link: the reported regression.
- test_null_byte_path_rejected: the seam directly (is_within_dir/safe_resolve).
- test_navigate_skips_bundle_escaping_symlink_link: closes the previously
untested path-safety-at-link-resolution branch of navigate_bundle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVHLJuwBzp7MXkrUXJYARS
Door A (manifest -> connector -> deterministic materialization -> index) is no
longer implemented here. src/portfolio_optimiser/ingest.py becomes a thin
consumer seam over the shared library, git-pinned to v0.3.1 on the same Forgejo
channel portfolio-optimiser-claude uses. Net -626/+385; ingest.py 599 -> 145 lines.
shared/ingest-spec.md remains the normative spec: the library implements it, it
does not replace it. Spec changes continue to go via commons.
Acceptance criterion met and proven: all three golden bundles (file/sql/http)
are byte-exact before and after, including the idempotence re-run. examples/ and
shared/ carry ZERO modifications -- the fasit was not adjusted to fit.
The rejection set was verified equivalent, not assumed: all 22 malformations the
repo's pydantic models refused are refused by the library, with typed codes
(okf_type_reserved, credential_embedded, extraction_id_duplicate, ...).
Test rebinding (invariants preserved, vehicle changed): the library has zero
runtime dependencies by design, so pydantic is unavailable to it.
ManifestV1.model_validate(dict) -> load_manifest_bytes(bytes); ValidationError ->
ManifestError; model_fields -> dataclasses.fields; PathSecurityError ->
SourceError(path_escape); ValueError -> MaterializationError(ingested_at_invalid).
Tests now also pin the refusal `code`, the library's documented stability
contract -- a sharper assertion than "some validation error was raised".
Two accepted behavioural deltas, recorded rather than silently dropped:
- Title whitespace is stored verbatim instead of collapsed at validation, so the
frontmatter title and the index label are no longer guaranteed identical for
irregular whitespace. Both behaviours are spec-conformant (the spec is SILENT;
the old one was a repo-local pinned decision). Queued as a commons-amendment
candidate so both stacks pin the same answer. Goldens unaffected.
- The section 8 audit log moves to logger llm_ingestion_okf.materialize. Nothing
in the repo consumed the old channel.
Also: the `type` discriminator is no longer a dataclass field, so the spec
cross-check asserts it explicitly -- without that line the swap would have
silently narrowed the test.
New tests/test_ingest_library_seam.py pins the seam itself: the restated section 5
stamp formula against the stamp the library actually writes (the one place the
adapter does not purely delegate, since v0.3.1 exposes no stamp helper), the
local-only allow_network default, the list[Path] unwrapping, and a guard that the
adapter never regrows local Door A machinery. All four verified RED when detached,
as were both golden regressions under a byte-level render mutation.
Door A is UNGATED: it calls no guard before writing to disk. Gating untrusted
content remains the caller's responsibility (guard wiring still planned).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4jNN186eVqfe1x5DnTU6r
Map untrusted-ingest surface (ingest.materialize http/I6, verdict-inbox
load, promote_verdict, future received-bundle) vs first-party paths;
verdict = planned, wire scan/sanitize before M3 as S2.4/S2.5 extension.
Plan only — guard not wired. shared/ hardening owned by commons session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145ZKPLMVeqM47z2jxxokym
Both live in the tolerant inbox-read path and both contradicted the module's
own "skipped, never raised" / loader-parity docstrings (S5.1 review: ALLOW,
2 MINOR, non-gating):
- UnicodeDecodeError (a *.json hand-saved in Latin-1 with Norwegian æ/ø/å is
invalid UTF-8) now SKIPPED in hitl._load_json_dict AND
verdicts.load_verdicts_from_dir — was: crashed pending/route with a raw
traceback (a ValueError subclass, caught by neither OSError nor
JSONDecodeError). Fix symmetric across both readers.
- hitl._inbox_verdict_ids now skips a non-iterable affected_codes exactly as
load_verdicts_from_dir does (frozenset() raises TypeError) — was: marked the
proposal judged on key-presence alone → a silent false-negative in the
operator pending queue. hitl-only: verdicts is the correct reference.
TDD: 3 RED-then-green (both UnicodeDecodeError twins + the parity gap) + 1
ground-truth pin locking the loader side so the parity cannot rot.
Gate: pytest → 389 passed, 4 skipped (was 385); ruff check + format clean; mypy clean.
Post-hoc /trekreview of S4.2 surfaced two confirmed findings; both closed via TDD.
S42-001 (MAJOR): the new --live-dry-run CLI tests read PORTFOLIO_MODEL_MAP /
PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT via resolve_model/AzureFoundryBackend but did
not isolate them, so both arms inverted their rc in a Foundry-configured env.
Add an autouse fixture that delenvs both, mirroring test_backends/test_preflight.
S42-002 (MINOR): the --live-dry-run except ValueError attached the azure-preflight
remediation to every offline-path ValueError (unknown project_id, empty docs_dir,
bundle mismatch). Scope the hint to args.profile == "azure"; structured refusal +
rc 1 preserved for all. New test proves a LOCAL unknown-project refusal carries no
azure hint.
Gate: pytest 358 passed / 4 skipped, ruff check + format clean, mypy 24 files.
Post-hoc /trekreview (WARN, scope 0670b2a..b513e64) surfaced 2 findings, both closed:
- MAJOR MISSING_TEST: the non-https scheme guard (preflight.py:61) and non-azure
profile refusal (:82-86) had zero test coverage — added
test_non_https_endpoint_refused + test_non_azure_profile_refused so an inverted
condition can't regress silently.
- MINOR MISSING_ERROR_HANDLING: an existing-but-unreadable PORTFOLIO_MODEL_MAP
raised PermissionError (an OSError, not ValueError) past the except at :95 →
widened to OSError (subsumes FileNotFoundError). test_unreadable_override_is_
structured_refusal proves the traceback-free invariant (red pre-fix).
Suite 348->351/4, ruff+format+mypy clean.
Closes the one MAJOR from /trekreview of S3.6 (verdict WARN): the
user-facing NOK-string formatter had only an isinstance(...,str) check,
so a regression (wrong divmod, dropped padding, øre/kr transposition,
or non-breaking-space → ASCII) would ship a wrong money figure with the
suite green. Adds a load-bearing value-level test (thousands grouping,
sub-100-øre padding, negative-sign branch) pinning the actual Norwegian
\xa0-separated formatting; verified red under a divmod(...,1000) mutation
and reverted. Suite 333 passed / 4 skipped, ruff+mypy clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145ZKPLMVeqM47z2jxxokym
