docs(k3): round 24, a reason that became false is struck and the emitter stays flow

`_render_sources` gave three measured reasons for refusing to emit the block
form. Two of them fell with `28f9a4b` and are struck rather than left
standing: a block list no longer round-trips to an empty value (4 609 of
4 609 concept files), and B6's acceptance test can now pass.

Reason 2 STANDS and now carries the emission rule alone, re-measured by
reading `portfolio-optimiser` at `6eb58e5`: `read_provenance` returns
`UnreadableProvenance(reason="block-sequence")` for a block value. It is not
the guard's objection -- guard 1.4.0 reads the block form on 4 609 of 4 609
of the same files -- so the answer to "should our own emitter write block" is
NO, with the number that carries it, and the measurement that would make it
decidable is named in STATE rather than deferred without one.

The README paragraph said this library "cannot read the block form" where one
reader could and one could not. It now separates the two by KEY: `sources` in
either form through every reader, every other block mapping still refused,
with the D1b pointer intact.

Report: 222 lines, S7 form. Five deviations first, including the acceptance
sentence naming THREE flat readers rather than one, a `title` denominator of
7 372 rather than 5 372, and the `okf check` rule answered NO with a
measurement (`contract_check` holds 0 occurrences of `sources`, and
`consume` already writes `sources_unreadable` for the state a rule would
restate; `len(RULES)` is 17). Honesty limits name the one-key-wide set, the
block `verified:` fixture still reading empty, and the rendering not being
byte-identical to the source.

Suite 1807 passed / 1 skipped, rc 0, 98 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 16:50:26 +02:00
commit 2d4f56dc6c
5 changed files with 326 additions and 27 deletions

View file

@ -7,6 +7,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- **A block `sources:` sequence no longer loses the address in the flat
readers (K3-24).** `consume.read_sources` has always read both YAML forms;
the three copies of this library's line-oriented frontmatter grammar read
only the flow one and returned the key with an EMPTY value for the block
form -- not a `KeyError` a consumer can catch, an address that disappears.
Measured 2026-09-12 over four bundles a producer ships, denominator = files
carrying a frontmatter block: 2 756 of 2 757, 446 of 447, 1 133 of 1 134 and
270 of 271 concept files lost it, while PyYAML 6.0.3 and the pinned guard
1.4.0 both read it on 100 % of the same files. After: **0 of each**, and all
three flat readers return what BOTH reference readers return on 4 609 of
4 609 files, plus 2 762 of 2 762 in a flow-form bundle that is unchanged.
`materialize.parse_frontmatter` is public API, so the external consumer is
the one this repairs.
The value type was the choice and it was measured: widening the return type
from `dict[str, str]` costs 15 `mypy --strict` errors across four modules
plus a signature every outside caller follows; rendering the entries back
into the flow form those readers already round-trip costs 0. The rendering
is a READING projection, not a claim that the value is writable.
Narrow on purpose: `profiles.STRUCTURED_BLOCK_KEYS` is `{"sources"}`, the
key `read_sources` already knows how to read, and one grammar now serves all
four call sites (`profiles.read_block_mappings`). A shipped fixture carrying
a block `verified:` still reads as an empty value, pinned by a test so the
next widening is a decision rather than a side effect. The K3-20 guarantee
is unmoved and asserted per reader copy: a nested key never enters the
document's namespace.
### Changed
- **No bundle bytes move.** A five-document folder built before and after is
`diff -r`-identical (52 files), so no ranking measurement is owed. The
emission rule is untouched: this library still writes flow.
- **Two docstrings and one README paragraph corrected rather than left
standing.** `materialize._render_sources` gave three measured reasons for
refusing to emit the block form; reason 1 (a block list round-trips to an
empty value) and reason 3 (B6's acceptance test cannot pass) FELL with this
fix and are struck. Reason 2 STANDS and now carries the rule alone,
re-measured by reading `portfolio-optimiser` at `6eb58e5`: `read_provenance`
returns `UnreadableProvenance(reason="block-sequence")`. It is not the
guard's objection -- guard 1.4.0 reads the block form on 4 609 of 4 609
files. The README said this library "cannot read the block form" where one
reader could and one could not; it now separates the two by KEY.
## [0.8.4] — 2026-09-11
### Added

View file

@ -876,19 +876,36 @@ resource it points at: recovering the resource needs the structured reader.
One limit worth knowing before you write such a concept: §10.2 presents
`executor` and `attester` as nested block mappings, and this library's
frontmatter parser is line-oriented. It reads inline **flow** mappings
(`executor: { resource: …, receipt: [ … ] }`) as opaque values that round-trip
unchanged, but it cannot read the block form — two block mappings that both
carry a `resource` collapse into one namespace and the first is lost. Write the
flow form, and know its limit: it is valid YAML only while every plain value
inside it avoids what ends a flow scalar — `,`, `[`, `]`, `{`, `}`, and for
PyYAML also `?` — as well as `": "`, `" #"`, a trailing `:` and a leading YAML
indicator. Within that limit a YAML consumer recovers the same structure from
either form. Beyond it no flow form works: quoting satisfies PyYAML, but the
guard refuses a quoted value inside a flow mapping, so this library refuses
such a value rather than write frontmatter a reader cannot parse. An earlier
version of this paragraph said "both are valid YAML" without that limit; it was
measured false for an unquoted URL with a query string.
frontmatter parser is line-oriented. Which form it reads depends on the KEY,
and the distinction is worth stating precisely, because this paragraph used to
blur it:
- **`sources` in either form.** `consume.read_sources` has always read both,
and since 2026-09-12 (K3-24) so do all three copies of the flat grammar
(`materialize.parse_frontmatter`, which is public API, and the two internal
ones). A block sequence of mappings is decoded into the flow rendering those
readers round-trip; the entries never enter the document's key namespace.
Measured against PyYAML 6.0.3 and the pinned guard 1.4.0 on 4 609 of 4 609
concept files carrying a block `sources`, all three readers return the same
entries both references do.
- **Every other key, flow only.** `executor`, `attester` and any other block
mapping are still skipped rather than parsed: two block mappings that both
carry a `resource` would collapse into one namespace and the first would be
lost, so they are refused instead. Reading them needs the structured reader
(D1b).
Write the flow form, and know its limit: it is valid YAML only while every
plain value inside it avoids what ends a flow scalar — `,`, `[`, `]`, `{`, `}`,
and for PyYAML also `?` — as well as `": "`, `" #"`, a trailing `:` and a
leading YAML indicator. Within that limit a YAML consumer recovers the same
structure from either form. Beyond it no flow form works: quoting satisfies
PyYAML, but the guard refuses a quoted value inside a flow mapping, so this
library refuses such a value rather than write frontmatter a reader cannot
parse. An earlier version of this paragraph said "both are valid YAML" without
that limit; it was measured false for an unquoted URL with a query string. That
refusal is also why K3-24 is a reading change only: this library still emits
flow, because a named downstream consumer classifies a block sequence as
unreadable provenance even though the guard reads it.
## Non-goals

View file

@ -0,0 +1,222 @@
# K3 round 24 — the block `sources` form reaches every flat reader
Order K3-24, 2026-09-12. Red `eb327bf`, fix `28f9a4b`, base `150c726`
(v0.8.4). PyYAML 6.0.3, guard 1.4.0, Python 3.14.0. Every measurement below
ran through the repository's own interpreter against a frozen `git archive`
export, `__file__` verified under the scratch path and never under
`/Users/ktg/repos`; the four producer bundles and `portfolio-optimiser` were
READ and never written.
## Deviations, first
1. **The order is not this repository's STATE `NESTE`.** STATE's next step is
the K3-21 link-line ranking question; this order is a new finding from a
re-measurement of the producer's D4 work. The link-line question stands as
next after.
2. **Acceptance point 1 says "through EVERY flat reader", and there are
three, not one.** `materialize.parse_frontmatter` (public API),
`structure._split_frontmatter` and `profiles._split_frontmatter` are three
copies of one line-oriented grammar, documented as such at each site. The
order's prose names `parse_frontmatter`; fixing only that one would have
left two readers with the defect and made the acceptance sentence false.
All three are fixed and all three are measured separately below.
3. **The `title` denominator is 7 372, not 5 372.** Counting every file with a
frontmatter block across the five bundles gives 2 757 + 447 + 1 134 + 271 +
2 763 = 7 372. The order's number is not reproduced here and the larger one
is the one measured.
4. **`^sources:` in `tests/` is 4 files, not 3.** PM measured 3 before this
round; this round adds `tests/test_block_sources_flat_readers.py`. Across
`tests/`, `examples/`, `skills/` and `docs/` the count is 9, of which **4
are golden concept files**.
5. **The order's § 2 sub-question about an `okf check` rule is answered NO,
with a measurement rather than a deferral** (§ 6).
## 1. The defect, reproduced
`consume.read_sources` reads both YAML forms. The three copies of the flat
grammar read only the flow one: for a block sequence they return the key with
an **empty value**. The key is present and the value is gone, so no consumer
can catch it as an error.
Denominator = files carrying a frontmatter block. `read_sources` was called
with `_frontmatter_lines(path)`, never a `Path`; the known-positive control
(`test_control_read_sources_reads_the_block_form`) runs before any zero is
believed.
| bundle | files | `read_sources` entries | flat `sources == ""` | key absent | `safe_load` OK |
|---|---|---|---|---|---|
| `r761-2025` | 2 757 | 2 756 | **2 756** | 1 | 2 757 |
| `n100-2023` | 447 | 446 | **446** | 1 | 447 |
| `n200-2024` | 1 134 | 1 133 | **1 133** | 1 | 1 134 |
| `n500-2024` | 271 | 270 | **270** | 1 | 271 |
| `r761-2025-generisk` (flow) | 2 763 | 2 761 | 0 (2 761 non-empty) | 2 | 2 762 |
Every cell reproduces the figure this order was dispatched with.
### Three readers, one file
| reader | `n500-2024` concept, before |
|---|---|
| `yaml.safe_load` (PyYAML 6.0.3) | `[{'resource': 'https://…/859990?languageCode=nb', 'title': 'N500:2024'}]` |
| `llm_ingestion_guard.okf.parse_frontmatter` (1.4.0) | the same mapping |
| `llm_ingestion_okf.materialize.parse_frontmatter` | `''` |
| `llm_ingestion_okf.structure._split_frontmatter` | `''` |
| `llm_ingestion_okf.profiles._split_frontmatter` | `''` |
## 2. The fix, and the number that chose it
**Path (a)**, narrowest: one structure-aware branch for the keys
`profiles.STRUCTURED_BLOCK_KEYS` names — today exactly `{"sources"}`, the key
`read_sources` already knows how to read. "Skip indented lines" stays the rule
for everything else, and the K3-20 refusal it exists for is untouched: a
decoded block lands INSIDE its own value and never in the document's
namespace.
The order asked which TYPE the value takes, because the return type is
`dict[str, str]` and a list of mappings is not a `str`. Both answers measured
on the same tree:
| option | cost |
|---|---|
| re-serialise to the flow form the flat readers already round-trip | **0** call sites moved, 0 `mypy --strict` errors, public signature unchanged |
| widen the return type to `str \| list[dict[str, str]]` | **15** `mypy --strict` errors in **4** of the 5 modules that touch the reader, plus a public-API signature every outside caller follows |
Re-serialisation wins on that number. What it is NOT is stated in the code:
the rendering is a **reading projection**, not a claim that the value is
writable. `yaml_flow_plain` still refuses a `?` and the guard still refuses a
quote inside a flow mapping, so a value rendered here may have no writable
flow form at all — which is the whole reason the producer writes block.
One grammar, four call sites: `read_block_mappings` moved out of
`read_sources`, where it was written and measured, into `profiles` (the module
both the flat readers and `consume` import). Two copies of a block grammar
would be two answers to one question.
## 3. Acceptance
1. **0 files with an empty `sources`** in all four bundles, through all three
flat readers: `0 / 0 / 0` per bundle against 2 756 / 446 / 1 133 / 270.
2. **Dict equality against both reference readers, every file, both
denominators:** PyYAML 2 757/2 757, 447/447, 1 134/1 134, 271/271,
2 762/2 762; guard 1.4.0 identical on the same five denominators. Not a
sample.
3. **`tests/fixtures`: 12 of 12 read.** Eleven byte-identical dicts before and
after; the twelfth differs on exactly one key — the block `sources` that is
the point of the change, `''` → the decoded address. Nothing else moved.
4. **The flow form is unchanged:** `r761-2025-generisk` gives 2 761 non-empty
`sources` before and after.
5. **No document's own `title` moved** on any of the **7 372** files with a
frontmatter block across the five bundles (`diff` on the full title map per
bundle: 5 of 5 identical).
6. **Suite 1807 passed / 1 skipped, rc 0, 94 s** in the working tree. The
baseline on `150c726` is 1782/1 and 1782 + 25 new = 1807; no other test is
red or newly skipped. `ruff` clean, `ruff format --check` 119 files,
`mypy --strict` clean over 21 files.
7. **`okf check`: 17 rules, 0 findings, rc 0** (`len(RULES)` printed as a
literal from the frozen export) over a payload built from a scratch bundle.
8. **`grep "import yaml"` in `src/`: 0 files non-zero.** `dependencies` is
still exactly `["llm-ingestion-guard>=1.2,<2.0"]`; `uv.lock` untouched.
9. **No ranking measurement is owed and it is shown, not asserted:** a
five-document folder built with the pre-fix and post-fix code is `diff -r`
identical, 0 differences over 52 files.
### One pre-existing divergence, measured so it is not attributed here
The three flat readers do not agree with each other on every key, before OR
after: `structure._unquote` strips a `'`-wrapped value where the other two
keep it. On `r761-2025` the three agree on 3 of 2 757 files both before and
after, with the identical key histogram (`prosessnr` 2 727, `hovedprosess`
2 727, `forelder` 2 728, `seksjon` 2 735, `description` 18, `seksjonstittel`
1, `title` 1); on `n500-2024`, 172 of 271 both times. `sources` does not
appear in that histogram after the fix. This round neither caused nor closed
it.
## 4. `_render_sources`' three reasons, one at a time
| reason | verdict | measurement |
|---|---|---|
| 1. a block list round-trips to an empty value, silently | **FELL** | § 1 and § 3: 4 609 of 4 609 files now decode through every flat reader |
| 2. the consumer accepts flow plural and classifies block as unreadable provenance | **STANDS** | `portfolio-optimiser` read at `6eb58e5`: `read_provenance` returns `UnreadableProvenance(reason="block-sequence")` for a block value |
| 3. B6's acceptance test asks for a round trip through this parser | **FELL** | the rewritten `test_the_block_form_round_trips_through_the_flat_reader` passes with both entries |
Reason 2 is **not the guard's objection**. Guard 1.4.0 reads the block form on
4 609 of 4 609 of the same files. The consumer that refuses it is named, and
after this round it is the only thing holding the emission rule.
## 5. Should the emitter write block? **No**, and the number is reason 2
The order's premise is measured and true: `yaml_flow_plain` is `False` for the
producer's address, so this library's own inbox door raises
`inbox_source_file_unaddressable` on exactly the URL the producer ships in
four bundles. No flow form passes both PyYAML (which refuses an unquoted `?`)
and the guard (which refuses a quote inside a flow mapping). That is why the
producer moved to block.
It is still not enough to move the emitter, because reason 2 stands: writing
block would hand `portfolio-optimiser` the state its own decoder reports as
unreadable — replacing a defect in OUR reader with a defect in THEIR reader,
for the same bytes. The byte cost is recorded for whoever revisits it: **4
golden concept files** carry a `^sources:` line (`ingest-golden-okf-v0-2` 1,
`ingest-golden-segmented-okf-v0-2` 3), plus every bundle rebuilt after such a
change, plus the hit@k gate that a byte move in a bundle triggers.
What would make the question decidable is one measurement in another
repository, and it is written into STATE: whether `portfolio-optimiser`'s
`read_provenance` will accept a block sequence. Until that is answered there,
the answer here is no — and the cost of the no is zero, because the producer's
bundles are already readable by all three of okf's readers, by PyYAML and by
the guard.
## 6. Does `okf check` owe a rule? **No**, measured
`contract_check.py` contains **0** occurrences of `sources` and **0** of
`frontmatter`: no rule reads the key today, so nothing regressed. The
candidate was `read_sources`' third state — "the key is there and this reader
cannot decode it". It is already named where it is observable:
`consume.build_payload` writes `sources_unreadable: true` into the excerpt for
exactly that state. A checker rule would restate an existing mechanism, and
`okf check` compares a DECLARED identity against a DECLARED identity without
opening the bundle, so it could not verify the claim independently anyway. No
rule was added and `len(RULES)` is **17**.
The nine `parse_frontmatter` call sites in `src/` were re-counted
(`importer.py:557`, `:581`; `inbox.py:663`, `:687`, `:1473`, `:1641`;
`consume.py:353`, `:1952`; `materialize.py:166`) and none of them reads
`sources` out of the flat dict — so the consumer this repairs is the EXTERNAL
one, reaching `okf.parse_frontmatter` as public API. The reprojection path was
confirmed rather than assumed: `structure.structure_frontmatter` writes
`declared[key]` back for every facet key, and `sources` is a facet key on
**none of the seven profiles** (four carry a `FacetPolicy`, three carry
`None`). Had anyone added it, the pre-fix reader would have written the empty
value back into the concept file — destroying the address on disk and not only
in memory. That is the distance between "harmless today" and "safe", and it is
now closed by the reader rather than by the facet list.
## 7. Honesty limits
- **`STRUCTURED_BLOCK_KEYS` is one key wide.** A fixture in this tree carries
a block `verified:` sequence and it still reads as an empty value. That is
the same defect family for a different key, pinned by
`test_a_block_key_outside_the_named_set_is_still_empty` so the next widening
is a decision rather than a side effect. It is a LIMIT, not a closed case.
- **The rendering is not byte-identical to the source.** A consumer comparing
`parse_frontmatter`'s value against the file's own bytes will see a
difference; a consumer comparing ENTRIES will not. The structured reader
(D1b) is still the answer for anyone who needs the original bytes.
- **A rendered value may have no writable form.** Writing it back into a
frontmatter can produce a file PyYAML or the guard refuses. This is stated
in the function's own docstring; nothing in the code prevents it, because
preventing it would mean refusing the addresses the fix exists to carry.
- **Quoted leaves diverge by design and the divergence is named:** the guard
keeps a leaf's quotes verbatim, PyYAML decodes them, and this library
follows `read_sources`' K3-22 rule (a `"`-wrapped leaf decoded, a
`'`-wrapped one left standing). The producer's four bundles carry **0**
quoted leaves, so the divergence is unreached there — measured, not assumed.
- **The three flat readers still disagree on `'`-quoted scalars** (§ 3). Not
caused here, not closed here.
- **One bundle was built to prove no bytes move**, five documents. A larger
corpus was not rebuilt, and the claim is exactly as wide as that
measurement.
Conformance is the floor, never the proof: every acceptance number above was
run against artefacts, not against the shape of the code.

View file

@ -596,9 +596,12 @@ def read_sources(lines: Sequence[str]) -> tuple[tuple[Mapping[str, str], ...], b
handling one form delivers the other bundle with no address at all -- and
for N500 there is nothing else, because it carries zero locator keys.
Reading the block form is not a licence to WRITE it: this library's
line-oriented parser still cannot round-trip block lists, so the emission
rule (flow only) is untouched.
Reading the block form is not a licence to WRITE it, and since K3-24 the
reason is a different one: the line-oriented parser DOES round-trip a
block `sources` list now (it shares this function's grammar), so what
holds the emission rule at flow is the downstream consumer that classifies
a block sequence as unreadable provenance -- measured, not remembered, in
`materialize._render_sources`.
"""
for position, line in enumerate(lines):
if line[:1] in (" ", "\t") or not line.startswith("sources:"):

View file

@ -217,19 +217,29 @@ def _render_sources(sources: Sequence[Source]) -> str:
emitted form. The list is here; the block form is not, and the reason is
measured rather than argued:
- `parse_frontmatter` is line-oriented and skips indented lines, so a block
list round-trips to an EMPTY value with every entry gone, silently. We
would be writing provenance we cannot read back, and `_is_ingest_owned`
reads through that same parser.
- The consumer B6 was written for accepts `[{ k: v }, { k: v }]` plural
and classifies a block sequence as unreadable provenance. Block would hand
it exactly the state it reports as unreadable.
- B6's own acceptance test asks for a round trip through this parser. No
block form can pass it.
- **FELL 2026-09-12 (K3-24).** `parse_frontmatter` used to skip indented
lines unconditionally, so a block list round-tripped to an EMPTY value
with every entry gone, silently. It now decodes a block `sources`
sequence, measured against PyYAML 6.0.3 and the pinned guard on 4 609 of
4 609 concept files carrying one. A reason that has become false is worse
than no reason, so it is struck here rather than left standing.
- **STANDS.** The consumer B6 was written for accepts `[{ k: v }, { k: v }]`
plural and classifies a block sequence as unreadable provenance.
Re-measured 2026-09-12 by reading `portfolio-optimiser` at `6eb58e5`:
`read_provenance` returns `UnreadableProvenance(reason="block-sequence")`
for a block value. Block would hand the consumer that asked for
multi-source exactly the state it reports as unreadable. This reason
alone carries the emission rule, and it is NOT the guard's: guard 1.4.0
reads the block form on 4 609 of 4 609 of the same files.
- **FELL with the first.** B6's own acceptance test asks for a round trip
through this parser, and a block form now passes it
(`test_the_block_form_round_trips_through_the_flat_reader`).
The flow form also satisfies commons' §5 "all values MUST be single-line",
and §11 requires parseable YAML rather than block YAML. Reading block needs
the structured reader (D1b); until then the constraint binds what we write.
and §11 requires parseable YAML rather than block YAML. The flat readers
now read a block `sources`; that is a READING change and not a licence to
write one, because the standing reason above is about a consumer, not
about us.
A single source renders byte-identically to the one-entry form that shipped
before this took a list, which is what keeps all six goldens unmoved.