feat(examples): SEGMENTED_V1 golden fixture and decision record

docs/plan/concept-granularity.md

@@ -0,0 +1,145 @@
+# Door B concept granularity: one document becomes many concepts
+
+Record of what was decided and why, written after the work landed. The repo's
+pattern: `cf65635` after structure derivation, `d35bcb2` after Door C.
+
+## The defect this closes
+
+`process_inbox` emitted exactly one flat concept per dropped file. OKF v0.2 §2
+defines a concept as "a single unit of knowledge within a bundle" and a concept
+ID as "the path of the concept's file within the bundle" — neither ties a
+concept to a source file, and Appendix A presents v0.1 → v0.2 as a
+de-monolithization. Door B therefore implemented the shape the SPEC names as
+the one being migrated away from.
+
+No conformance test caught it, and none could. §11 checks that every
+non-reserved `.md` has parsable frontmatter with a non-empty `type`, so a
+bundle of one giant concept is fully conformant. **Conformance is the floor,
+not the proof.** The measurement that found it was a reading of §2's
+terminology, not a count of keys.
+
+## What was built
+
+A new profile, `SEGMENTED_V1`, and a caller-owned, human-adjudicated
+segmentation plan. `DEFAULT`, `STRICT_V1`, `STRUCTURED_V1` and `OKF_V0_2` are
+byte-identical; their goldens did not move, which is asserted against the
+baseline commit rather than assumed.
+
+The splitting decision is made once, written down as data, adjudicated by a
+human, and thereafter replayed deterministically. The run path still makes
+**zero model calls**, and a from-scratch rebuild is byte-identical to an
+incremental update.
+
+| Piece | Where |
+|---|---|
+| Plan model, parser, cache key, slicing | `segmentation.py` |
+| `SegmentationPolicy` + `SEGMENTED_V1` | `profiles.py` |
+| Root frontmatter, per-concept mirroring, 1→N, collision gate, ownership, per-directory indexes | `inbox.py` |
+| Proposer (outside the package) | `tools/okf_propose_segments.py` |
+| Golden | `examples/ingest-golden-segmented/` |
+
+## The identity decision, and what it costs
+
+Order `…2527032751` settled the form: **(c) a root-frontmatter bundle
+identifier consumers join on**, combined with **(a) `sources[].resource` as an
+absolute URL wherever the source has a stable one**. (c) carries identity, (a)
+carries provenance.
+
+**No step implemented form (a), and that is the correct outcome.** It is
+already emitted, at `materialize.py:205,214`, from the manifest locator —
+Door A's surface. Door B ingests dropped files, which have no stable URL by
+construction.
+
+**S4b is resolved as ONE branch.** The root index is the SOURCE of `bundle_id`
+— the caller supplies it exactly once, so D5 stays intact — and every concept
+MIRRORS it. Two bundles built from identical inputs therefore hold concepts
+whose paths **collide by construction** and whose identity values are disjoint.
+The collision is the expected behaviour under form (c), not a defect.
+
+**Two identity mechanisms, deliberately.** Within a bundle, Door C keys on a
+content hash; across bundles, identity is this caller-assigned `bundle_id`. A
+bundle is a collection the caller delimits, so hashing it would hand it a new
+identity every time a document was added. The cost is stated rather than
+hidden: a consumer meets both mechanisms.
+
+## Decisions worth their own line
+
+- **The capability is `profile.segmentation is not None`, never
+  `IndexPolicy.per_directory`.** `STRICT_V1` already sets `per_directory=True`
+  while Door B ignores it, so keying there would have changed a shipped
+  profile's output and broken its byte-stability pin.
+- **Offsets index the CANONICAL EXTRACTED TEXT, never source bytes.** A `.csv`
+  is re-rendered before it becomes a concept body, and a `.pdf` has no
+  sliceable prose at all. The golden and the `.csv` fixture both pin this; on a
+  `.md` fixture bytes and text are identical and the assertion would pass for
+  the wrong reason.
+- **The adjudication cache key is the triple `(source_sha256, extractor_id,
+  extractor_version)`, not the hash alone** (S5b). Source bytes cannot see an
+  extractor swap or a version bump, either of which silently invalidates every
+  stored offset while the hash still matches. Door B observes the hash and the
+  extractor id; the parser *version* is the caller's to declare, for the same
+  reason `bundle_id`'s value is — the library names the key, whoever knows the
+  value owns it.
+- **All N segments are gated before any is written.** Gating and writing one at
+  a time would leave a half-screened document on disk the moment segment 3 of 5
+  quarantines. A refusal is reported once, for the document.
+- **Ownership asks "which concepts are owned by source_file X?"**, recursively,
+  and only under the capability. Keyed on the concept filename, a round naming
+  fewer segments orphaned the rest: they survived incrementally, were absent
+  from a scratch rebuild, and the two diverged silently.
+- **A retired directory loses its index too.** A scratch rebuild writes an
+  index only where a concept lives, so an orphaned one is a one-sided
+  difference. Found by S7 on 2026-09-01, once its fixture was made to actually
+  retire a segment — with round 2 a superset of round 1, S7 stayed green with
+  retirement disabled entirely.
+- **Frontmatter values are FLOW form, never block.** `source_offset: [12, 48]`.
+  This library's parser round-trips a flow value as an opaque string and cannot
+  read a block one at all, so emitting block would produce bundles we cannot
+  read back.
+- **Index ordering has one seam, `_index_sort_key`.** A consumer-controlled
+  ordering is then a parameter, not a refactor.
+- **The proposer proposes; a human adjudicates.** Every entry it emits carries
+  `PROPOSED` and the rule that produced it. Research measured the reason: the
+  OKF reference agent's four semantic gates need a model this run path does not
+  have, and heading derivation is inert on most of the K2 corpus — 23 of 33
+  PDFs carry no outline, 95 % of outline entries are CAD metadata, and the
+  scoreable denominator is **one document**.
+
+## Answers to the brief's open questions
+
+- *Where does the plan live?* At a caller-supplied path, parsed by the caller
+  and passed in. The library loads nothing from a fixed location — the same
+  division D5 makes for values.
+- *Is `source_offset` readable?* Not this round. It is write-only until the
+  structured reader (D1b) lands. Nothing built here has to be torn out when it
+  does.
+- *Does a plan cover a whole run?* No — one plan, selected by content hash.
+  Every other dropped file in the same run keeps today's one-concept rule.
+
+## Known gaps, stated rather than left to be discovered
+
+1. **`derived` marks a declared title as inferred.** A segmented concept's
+   `title` is DECLARED by the adjudicator, but structure derivation runs over
+   the segment body and adds `title` to `derived`. The marker is therefore
+   wrong in the one direction this repo cares about — a consumer that distrusts
+   derived fields will distrust a fact the producer stated. The golden pins
+   today's behaviour; the fix belongs with the code that composes the two.
+2. **A plan matching no dropped file is silent.** A typo'd `source_sha256`
+   segments nothing and reports nothing, which is the silent skip this library
+   refuses elsewhere.
+3. **`status: deprecated` has no producer.** `status` is named as a facet key
+   and nothing writes it, so a deprecated parent cannot carry one.
+4. **An unchanged round still rewrites.** `write_bytes` writes
+   unconditionally; the bytes are identical, so no contract is broken, but
+   "an unchanged round writes nothing" is not true today.
+5. **Six new error codes live in `tests/test_segmentation.py`, not
+   `tests/test_error_codes.py`.** The baseline pin forbade editing the
+   registry file. Migrate once the pin is released.
+
+## Relation to the two orders
+
+- `…2527032751` (identity) — executed. Form (c) built, form (a) measured as
+  already present at Door A.
+- `…3790625395` (index ordering) — untouched by design. `_index_sort_key` is
+  the single named seat a consumer-controlled ordering would occupy, so that
+  decision stays a parameter change rather than a refactor.

examples/ingest-golden-segmented/bundle-id.txt

@@ -0,0 +1 @@
+b-golden-segmented

examples/ingest-golden-segmented/expected-bundle/index.md

@@ -0,0 +1,6 @@
+---
+bundle_id: b-golden-segmented
+---
+
+- [Veiledning for eksempelbundel](veiledning.md) — derived: [title]
+- [krav (index)](krav/index.md)

examples/ingest-golden-segmented/expected-bundle/krav/1-1/foerste-krav.md

@@ -0,0 +1,17 @@
+---
+type: reference
+title: Foerste krav
+source_file: veiledning.md
+source_sha256: 6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d
+ingested_at: 2026-08-30T09:00:00Z
+generated: true
+parent: s0
+derived: [title]
+bundle_id: b-golden-segmented
+segment_id: s1
+source_offset: [94, 176]
+---
+
+## 1.1 Foerste krav
+
+Et krav som beskriver den foerste betingelsen i eksempelet.

examples/ingest-golden-segmented/expected-bundle/krav/1-1/index.md

@@ -0,0 +1 @@
+- [Foerste krav](foerste-krav.md) — parent: s0?; derived: [title]

examples/ingest-golden-segmented/expected-bundle/krav/1-2/andre-krav.md

@@ -0,0 +1,17 @@
+---
+type: reference
+title: Andre krav
+source_file: veiledning.md
+source_sha256: 6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d
+ingested_at: 2026-08-30T09:00:00Z
+generated: true
+parent: s0
+derived: [title]
+bundle_id: b-golden-segmented
+segment_id: s2
+source_offset: [176, 253]
+---
+
+## 1.2 Andre krav
+
+Et krav som beskriver den andre betingelsen i eksempelet.

examples/ingest-golden-segmented/expected-bundle/krav/1-2/index.md

@@ -0,0 +1 @@
+- [Andre krav](andre-krav.md) — parent: s0?; derived: [title]

examples/ingest-golden-segmented/expected-bundle/krav/index.md

@@ -0,0 +1,2 @@
+- [1-1 (index)](1-1/index.md)
+- [1-2 (index)](1-2/index.md)

examples/ingest-golden-segmented/expected-bundle/veiledning.md

@@ -0,0 +1,16 @@
+---
+type: reference
+title: Veiledning for eksempelbundel
+source_file: veiledning.md
+source_sha256: 6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d
+ingested_at: 2026-08-30T09:00:00Z
+generated: true
+derived: [title]
+bundle_id: b-golden-segmented
+segment_id: s0
+source_offset: [0, 94]
+---
+
+# Veiledning for eksempelbundel
+
+Denne teksten er syntetisk og finnes kun for a pinne bytes.

examples/ingest-golden-segmented/fixture/inbox/veiledning.md

@@ -0,0 +1,11 @@
+# Veiledning for eksempelbundel
+
+Denne teksten er syntetisk og finnes kun for a pinne bytes.
+
+## 1.1 Foerste krav
+
+Et krav som beskriver den foerste betingelsen i eksempelet.
+
+## 1.2 Andre krav
+
+Et krav som beskriver den andre betingelsen i eksempelet.

examples/ingest-golden-segmented/ingested-at.txt

@@ -0,0 +1 @@
+2026-08-30T09:00:00Z

examples/ingest-golden-segmented/segmentation-plan.json

@@ -0,0 +1,44 @@
+{
+  "version": "1",
+  "source_sha256": "6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d",
+  "extractor_id": "md",
+  "extractor_version": "1",
+  "adjudicated_at": "2026-08-30T08:00:00Z",
+  "entries": [
+    {
+      "segment_id": "s0",
+      "path": "veiledning.md",
+      "title": "Veiledning for eksempelbundel",
+      "okf_type": "reference",
+      "span": [
+        0,
+        94
+      ],
+      "ingested_at": "2026-08-30T09:00:00Z"
+    },
+    {
+      "segment_id": "s1",
+      "path": "krav/1-1/foerste-krav.md",
+      "title": "Foerste krav",
+      "okf_type": "reference",
+      "span": [
+        94,
+        176
+      ],
+      "ingested_at": "2026-08-30T09:00:00Z",
+      "parent_id": "s0"
+    },
+    {
+      "segment_id": "s2",
+      "path": "krav/1-2/andre-krav.md",
+      "title": "Andre krav",
+      "okf_type": "reference",
+      "span": [
+        176,
+        253
+      ],
+      "ingested_at": "2026-08-30T09:00:00Z",
+      "parent_id": "s0"
+    }
+  ]
+}

tests/test_segmented_golden.py

@@ -0,0 +1,154 @@
+"""The SEGMENTED_V1 golden: bytes, pinned.
+
+A NEW golden directory, never an edit to the four that exist. That is how
+`2504011` shipped `OKF_V0_2`'s golden -- zero bytes changed in the three that
+were already there -- and it is what makes "additive" a measured claim rather
+than an intention. The second test below is the measurement.
+
+The fixture is synthetic throughout. No consumer content, no real N500 or
+procurement prose: this repository is public, and a golden is the most durable
+place a leak could land.
+
+Door B shaped, unlike the four Door A goldens: the inputs are an inbox, a
+segmentation plan and a `bundle_id`, because those are exactly what a rebuild
+replays. Re-materializing them into an empty directory must reproduce the
+shipped tree file for file -- which is S7 stated as a fixture instead of as a
+property test.
+"""
+
+from __future__ import annotations
+
+import json
+import subprocess
+from pathlib import Path
+
+from llm_ingestion_okf.inbox import GateDecision, process_inbox
+from llm_ingestion_okf.profiles import SEGMENTED_V1
+from llm_ingestion_okf.segmentation import parse_segmentation_plan
+
+EXAMPLES = Path(__file__).resolve().parents[1] / "examples"
+CASE = EXAMPLES / "ingest-golden-segmented"
+
+EXISTING_GOLDENS = (
+    "ingest-golden-file",
+    "ingest-golden-sql",
+    "ingest-golden-http",
+    "ingest-golden-okf-v0-2",
+)
+
+
+def gate(text: str) -> GateDecision:
+    return GateDecision(sanitized_text=text, disposition="warn")
+
+
+def materialize(out_dir: Path) -> None:
+    plan = parse_segmentation_plan(json.loads((CASE / "segmentation-plan.json").read_text("utf-8")))
+    process_inbox(
+        CASE / "fixture" / "inbox",
+        out_dir,
+        (CASE / "ingested-at.txt").read_text(encoding="utf-8").strip(),
+        okf_type="reference",
+        gate=gate,
+        profile=SEGMENTED_V1,
+        root_frontmatter_values={
+            "bundle_id": (CASE / "bundle-id.txt").read_text(encoding="utf-8").strip()
+        },
+        segmentation=plan,
+    )
+
+
+def tree(root: Path) -> dict[str, bytes]:
+    return {
+        path.relative_to(root).as_posix(): path.read_bytes()
+        for path in sorted(root.rglob("*"))
+        if path.is_file()
+    }
+
+
+def test_the_golden_rematerializes_byte_for_byte(tmp_path: Path) -> None:
+    out_dir = tmp_path / "bundle"
+    materialize(out_dir)
+    expected = tree(CASE / "expected-bundle")
+    actual = tree(out_dir)
+    assert sorted(actual) == sorted(expected)
+    for name in sorted(expected):
+        assert actual[name] == expected[name], f"{name} diverges from the golden bytes"
+
+
+def test_the_golden_covers_the_shape_the_profile_exists_for(tmp_path: Path) -> None:
+    # A golden that happened to be flat would pin nothing this profile adds.
+    expected = tree(CASE / "expected-bundle")
+    concepts = [name for name in expected if not name.endswith(SEGMENTED_V1.index.name)]
+    assert len(concepts) >= 3
+    assert any("/" in name for name in concepts)
+    assert len([name for name in expected if name.endswith(SEGMENTED_V1.index.name)]) >= 3
+    assert expected["index.md"].startswith(b"---\nbundle_id: ")
+
+
+def test_the_four_existing_goldens_are_untouched() -> None:
+    # Measured against the baseline commit, not asserted. `git status` covers
+    # an uncommitted edit; `git diff` against the baseline covers a committed
+    # one -- and this voyage makes commits, so the second is the one that bites.
+    baseline = (
+        Path(__file__).resolve().parents[1]
+        / ".claude/projects/2026-08-30-door-b-concept-granularity/baseline/sha"
+    )
+    if not baseline.is_file():
+        # The baseline is LOCAL-ONLY and gitignored, so a fresh clone has none.
+        # Skipping silently would make this test vacuous where it matters most,
+        # so the weaker check still runs.
+        result = subprocess.run(
+            [
+                "git",
+                "status",
+                "--porcelain",
+                "--",
+                *(f"examples/{name}" for name in EXISTING_GOLDENS),
+            ],
+            cwd=Path(__file__).resolve().parents[1],
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+        assert result.stdout == ""
+        return
+
+    result = subprocess.run(
+        [
+            "git",
+            "diff",
+            "--exit-code",
+            baseline.read_text(encoding="utf-8").strip(),
+            "--",
+            *(f"examples/{name}" for name in EXISTING_GOLDENS),
+        ],
+        cwd=Path(__file__).resolve().parents[1],
+        capture_output=True,
+        text=True,
+        check=False,
+    )
+    assert result.returncode == 0, f"an existing golden moved:\n{result.stdout}"
+
+
+def test_the_new_golden_was_absent_at_the_baseline() -> None:
+    # Without this, the exclusion above could be hiding churn inside an
+    # existing golden that had been renamed into the new directory.
+    baseline = (
+        Path(__file__).resolve().parents[1]
+        / ".claude/projects/2026-08-30-door-b-concept-granularity/baseline/sha"
+    )
+    if not baseline.is_file():
+        return
+    result = subprocess.run(
+        [
+            "git",
+            "cat-file",
+            "-e",
+            f"{baseline.read_text(encoding='utf-8').strip()}:examples/ingest-golden-segmented",
+        ],
+        cwd=Path(__file__).resolve().parents[1],
+        capture_output=True,
+        text=True,
+        check=False,
+    )
+    assert result.returncode != 0, "the new golden already existed at the baseline"