feat(frontmatter): --frontmatter KEY=VALUE stamps a key on every concept of a run

K3-19 b. `okf build --frontmatter KEY=VALUE`, repeatable, split on the FIRST
'=' (`cli.frontmatter_from_flags`) because a publisher's address carries '='
itself. The value is written verbatim on ONE line: the block form of a
`sources` list is invisible to this package's line-oriented readers, so the
flow form is the only one that survives them. Also reachable as
`build(frontmatter=...)`, `measure(concept_frontmatter_values=...)`,
`process_inbox(concept_frontmatter_values=...)` and
`render_inbox_concept(concept_frontmatter_values=...)`, keyword-only with
defaults, so every existing call site is source-compatible.

Precedence: a stated value beats what the document declares, which beats the
file name. A run may ADD any key and REPLACE only `sources` and
`description` -- the two with a derived layer below the flag. Every other key
the door writes is refused by `inbox.validate_concept_frontmatter` before a
proposal is written (`run_frontmatter_invalid`): measured from the bytes, owned
by another argument, the ownership stamp a later run reads back (including
Door A's `ingest_manifest`, which would make that door claim a Door B file),
or a derived facet whose `derived` marker would go on naming a replaced value.
A value that would not read back as stated -- empty, multi-line, or padded,
since `parse_frontmatter` strips -- is refused too.

SPEC SS 4.1 "Extensions" lets a producer add any key and SS 11 forbids a
consumer to reject one. Without the flag nothing moves: a test holds the
flagged tree to the plain one minus exactly the stated line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 03:14:46 +02:00
commit 912b85026b
5 changed files with 171 additions and 2 deletions

View file

@ -565,6 +565,17 @@ and fixtures, never code.
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`, `docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and `docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`. `docs/2026-09-08-k3-runde3-per-filtype.md`.
- **`--frontmatter KEY=VALUE` (K3-19, repeatable) is not a segmentation flag**
and moves no byte unless given: it stamps a key on every concept of the run,
split on the FIRST `=` and written verbatim on ONE line -- a block-form
`sources` is invisible to `parse_frontmatter`, so the flow form is the only
one that survives our own readers. It adds any key and REPLACES only
`sources` and `description`, the two with a derived layer below them:
precedence flag > what the document declares > file name. Every other key
the door writes (`inbox._door_keys`, including Door A's `ingest_manifest`,
which would make that door claim a Door B file) is refused before anything
is read. `okf project` does not take it -- it owns no flag that moves a
bundle's bytes.
- **A TWELFTH flag, `--pdf-outline`, is OFF** (round 12, 2026-09-10) and it is - **A TWELFTH flag, `--pdf-outline`, is OFF** (round 12, 2026-09-10) and it is
the only one here that does not read the extracted text at all: it cuts a PDF the only one here that does not read the extracted text at all: it cuts a PDF
at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D

View file

@ -145,6 +145,14 @@ than the clock, so two builds of the same folder are byte-identical — a
wall-clock default would break rebuild-equals-incremental for every caller who wall-clock default would break rebuild-equals-incremental for every caller who
did not pass them. did not pass them.
`--frontmatter KEY=VALUE` (repeatable) stamps a key on every concept of the
run, for what the operator knows and the document does not say — an edition,
a publisher's address. It splits on the first `=` and writes the value
verbatim on one line, so `'sources=[{ resource: <url>, title: <t> }]'` survives
whole. It adds any key and replaces only `sources` and `description`, the two
with a layer below them (what the document declares, else the file name);
every other key the door writes itself is refused before anything is read.
### The segmentation flags ### The segmentation flags
Nine rules are reachable from `okf build`, and since 2026-09-11 **all nine Nine rules are reachable from `okf build`, and since 2026-09-11 **all nine

View file

@ -70,14 +70,14 @@ from __future__ import annotations
import argparse import argparse
import sys import sys
import tempfile import tempfile
from collections.abc import Sequence from collections.abc import Mapping, Sequence
from functools import partial from functools import partial
from pathlib import Path from pathlib import Path
from .corpus import LOG_NAME, CorpusReport, load_plans, measure from .corpus import LOG_NAME, CorpusReport, load_plans, measure
from .errors import IngestError from .errors import IngestError
from .extract import declared_identity from .extract import declared_identity
from .inbox import walk_inbox from .inbox import validate_concept_frontmatter, walk_inbox
from .materialize import reduce_to_id_grammar from .materialize import reduce_to_id_grammar
from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile
from .propose import ProposerError, heading_reserve_applies from .propose import ProposerError, heading_reserve_applies
@ -251,6 +251,32 @@ DEFAULT_PDF_OUTLINE = False
DEFAULT_STAMP = "1970-01-01T00:00:00Z" DEFAULT_STAMP = "1970-01-01T00:00:00Z"
def frontmatter_from_flags(pairs: Sequence[str]) -> dict[str, str]:
"""`--frontmatter KEY=VALUE`, split on the FIRST `=` and only there.
The first `=` because the value is the one that needs the rest: a
publisher's address carries `?languageCode=nb`, and a split on every `=`
would cut the `sources` flow mapping in half. Key and value are validated
by the door (`validate_concept_frontmatter`); this only refuses what is not
a pair, and a key named twice -- which of two values was meant is a guess.
"""
values: dict[str, str] = {}
for pair in pairs:
key, sep, value = pair.partition("=")
if not sep:
raise IngestError(
f"--frontmatter {pair!r} has no '='; the form is KEY=VALUE",
code="run_frontmatter_invalid",
)
if key in values:
raise IngestError(
f"--frontmatter names {key!r} twice; refusing to pick one of the two values",
code="run_frontmatter_invalid",
)
values[key] = value
return values
def _document_prefixes(inbox: Path, walked: Sequence[Path]) -> dict[Path, str]: def _document_prefixes(inbox: Path, walked: Sequence[Path]) -> dict[Path, str]:
"""Each document's directory: the name it declares, else its file name. """Each document's directory: the name it declares, else its file name.
@ -405,6 +431,7 @@ def build(
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE, pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR, ocr: bool = DEFAULT_OCR,
pdf_outline: bool = DEFAULT_PDF_OUTLINE, pdf_outline: bool = DEFAULT_PDF_OUTLINE,
frontmatter: Mapping[str, str] | None = None,
) -> CorpusReport: ) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing. """Folder in, bundle out. The whole command, minus argument parsing.
@ -422,6 +449,11 @@ def build(
""" """
if proposed_at is None: if proposed_at is None:
proposed_at = ingested_at proposed_at = ingested_at
# Refused HERE, before one proposal is written: a stated key that is wrong
# is wrong for every document, and the proposal pass is the long half.
concept_values = validate_concept_frontmatter(
frontmatter or {}, profile=SEGMENTED_OKF_V0_2 if segments else STRUCTURED_V1
)
# Bound to THIS run's outline minimum, once, so the proposer and the door # Bound to THIS run's outline minimum, once, so the proposer and the door
# cannot be handed two different thresholds for the same question. # cannot be handed two different thresholds for the same question.
reserve = ( reserve = (
@ -438,6 +470,7 @@ def build(
pdf_headings=pdf_headings, pdf_headings=pdf_headings,
heading_reserve=reserve, heading_reserve=reserve,
ocr=ocr, ocr=ocr,
concept_frontmatter_values=concept_values,
) )
_write_log(bundle, report, profile=STRUCTURED_V1) _write_log(bundle, report, profile=STRUCTURED_V1)
return report return report
@ -495,6 +528,7 @@ def build(
pdf_headings=pdf_headings, pdf_headings=pdf_headings,
heading_reserve=reserve, heading_reserve=reserve,
ocr=ocr, ocr=ocr,
concept_frontmatter_values=concept_values,
) )
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2) _write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
return report return report
@ -584,6 +618,20 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument( build_parser.add_argument(
"--okf-type", default="reference", help="okf_type for every concept and proposal" "--okf-type", default="reference", help="okf_type for every concept and proposal"
) )
build_parser.add_argument(
"--frontmatter",
action="append",
default=None,
metavar="KEY=VALUE",
help=(
"stamp KEY: VALUE on every concept of this run; repeatable. Split on "
"the FIRST '=' and written verbatim on ONE line, so a flow mapping such "
"as 'sources=[{ resource: <url>, title: <t> }]' survives whole. Adds "
"any key, and REPLACES only sources and description -- the two with "
"a layer below them (the document's own identity, else the file "
"name). Every other key the door writes itself is refused"
),
)
build_parser.add_argument( build_parser.add_argument(
"--plans-dir", "--plans-dir",
type=Path, type=Path,
@ -934,6 +982,7 @@ def main(argv: list[str] | None = None) -> int:
pdf_headings_reserve=args.pdf_headings == "font-reserve", pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr, ocr=args.ocr,
pdf_outline=args.pdf_outline, pdf_outline=args.pdf_outline,
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
) )
except (IngestError, OSError, ValueError) as exc: except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr) print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

View file

@ -283,6 +283,7 @@ def measure(
pdf_headings: bool = False, pdf_headings: bool = False,
heading_reserve: Callable[[str], bool] | None = None, heading_reserve: Callable[[str], bool] | None = None,
ocr: bool = False, ocr: bool = False,
concept_frontmatter_values: Mapping[str, str] | None = None,
) -> CorpusReport: ) -> CorpusReport:
"""Run the corpus through the door and count what happened. """Run the corpus through the door and count what happened.
@ -307,6 +308,7 @@ def measure(
pdf_headings=pdf_headings, pdf_headings=pdf_headings,
heading_reserve=heading_reserve, heading_reserve=heading_reserve,
ocr=ocr, ocr=ocr,
concept_frontmatter_values=concept_frontmatter_values,
) )
elapsed = time.monotonic() - started elapsed = time.monotonic() - started

View file

@ -21,6 +21,7 @@ from __future__ import annotations
import hashlib import hashlib
import os import os
import re
import unicodedata import unicodedata
from collections.abc import Callable, Mapping, Sequence from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, replace from dataclasses import dataclass, replace
@ -114,6 +115,7 @@ def render_inbox_concept(
units: SourceUnits | None = None, units: SourceUnits | None = None,
span: tuple[int, int] | None = None, span: tuple[int, int] | None = None,
source_title: str | None = None, source_title: str | None = None,
concept_frontmatter_values: Mapping[str, str] | None = None,
) -> str: ) -> str:
"""Frame extracted text as an inbox concept file with its provenance layer. """Frame extracted text as an inbox concept file with its provenance layer.
@ -131,6 +133,10 @@ def render_inbox_concept(
`source_title` is what the document calls itself, for the `sources` `source_title` is what the document calls itself, for the `sources`
entry's `title`; `None` keeps the file name there, as before it existed. entry's `title`; `None` keeps the file name there, as before it existed.
`concept_frontmatter_values` are keys the CALLER states for every concept
of a run, validated by :func:`validate_concept_frontmatter` and applied
LAST, so a stated `sources` or `description` replaces the derived one.
`segment` and `bundle_id` carry the 1-to-N identity layer and are read ONLY `segment` and `bundle_id` carry the 1-to-N identity layer and are read ONLY
when the profile declares the segmentation capability. A concept the plan when the profile declares the segmentation capability. A concept the plan
does not cover keeps today's rule verbatim, and the four shipped profiles does not cover keeps today's rule verbatim, and the four shipped profiles
@ -238,9 +244,95 @@ def render_inbox_concept(
title=source_title, title=source_title,
) )
) )
if concept_frontmatter_values:
# LAST, and the position is the precedence: a value the caller states
# for the run beats what the document declares, which beats the file
# name. Validation keeps it to the keys that have a derived layer to
# replace -- everything else this door writes is refused.
frontmatter.update(
validate_concept_frontmatter(concept_frontmatter_values, profile=profile)
)
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{_normalize_body(text)}" return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{_normalize_body(text)}"
#: The two keys a run may REPLACE rather than only add. Each has a layer below
#: the flag -- the document's own title or the file name for `sources`, the
#: document's own first spec point for `description` -- so stating one for the
#: run is choosing a layer, which is what the precedence exists for.
RUN_FRONTMATTER_OVERRIDES = frozenset({"sources", "description"})
# A key the line-oriented readers here recover exactly: no `:`, no space, no
# leading `-` that a YAML reader would take for a sequence entry.
_RUN_KEY = re.compile(r"^[A-Za-z_][A-Za-z0-9_-]*$")
def _door_keys(profile: BundleProfile) -> frozenset[str]:
"""Every key this door writes itself under `profile`, less the overrides.
Refused as run values because each is something a run cannot restate
without lying: measured from the bytes (the hash, the offsets, the
locators), owned by another argument (`type`, `ingested_at`, the bundle
id), the ownership stamp a later run reads back (`generated`, and Door A's
`ingest_manifest`, which would make that door claim this one's file), or a
derived facet whose `derived` marker would go on naming a value the run had
replaced.
"""
keys = set(DEFAULT.frontmatter.order)
keys.update(("parent", "adjudicated_by", "adjudicated_at", "adjudication_dwell_s"))
if profile.index.facets is not None:
keys.update(profile.index.facets.keys)
if profile.segmentation is not None:
policy = profile.segmentation
keys.update((policy.bundle_id_key, policy.segment_id_key, policy.offset_key))
if policy.adjudication_key is not None:
keys.add(policy.adjudication_key)
if profile.provenance is not None:
address = profile.provenance
keys.update(
(
address.sources_key,
address.pages_key,
address.sheet_key,
address.rows_key,
address.lines_key,
)
)
return frozenset(keys - RUN_FRONTMATTER_OVERRIDES)
def validate_concept_frontmatter(
values: Mapping[str, str], *, profile: BundleProfile
) -> dict[str, str]:
"""Refuse a run-stated key or value that would not read back as stated.
SPEC SS 4.1 lets a producer add any key and SS 11 forbids a consumer to
reject one, so the limits here are this package's own and each is a
reader it has to survive: the value is written verbatim on ONE line
because every reader here is line-oriented, which rules out a line break
and -- since `parse_frontmatter` strips -- surrounding whitespace.
"""
written = _door_keys(profile)
for key, value in values.items():
if not _RUN_KEY.match(key):
raise MaterializationError(
f"frontmatter key {key!r} is not a plain key ([A-Za-z_][A-Za-z0-9_-]*)",
code="run_frontmatter_invalid",
)
if key in written:
raise MaterializationError(
f"frontmatter key {key!r} is written by this door itself; a run may "
f"add keys and replace only {sorted(RUN_FRONTMATTER_OVERRIDES)}",
code="run_frontmatter_invalid",
)
if not value or value != value.strip() or "\n" in value or "\r" in value:
raise MaterializationError(
f"frontmatter value for {key!r} must be one non-empty line with no "
f"surrounding whitespace, got {value!r}",
code="run_frontmatter_invalid",
)
return dict(values)
# The characters that would end a YAML flow mapping early, so a path carrying # The characters that would end a YAML flow mapping early, so a path carrying
# one would produce a `sources` list that parses as something other than what # one would produce a `sources` list that parses as something other than what
# was written. The guard refuses a quoted scalar inside a flow mapping (1.3.0, # was written. The guard refuses a quoted scalar inside a flow mapping (1.3.0,
@ -678,6 +770,7 @@ def _render_segments(
source_file: str, source_file: str,
units: SourceUnits | None, units: SourceUnits | None,
source_title: str | None = None, source_title: str | None = None,
concept_frontmatter_values: Mapping[str, str] | None = None,
) -> BlockedFile | None: ) -> BlockedFile | None:
"""Render every segment, or refuse the WHOLE document. """Render every segment, or refuse the WHOLE document.
@ -745,6 +838,7 @@ def _render_segments(
bundle_id=bundle_id, bundle_id=bundle_id,
units=units, units=units,
source_title=source_title, source_title=source_title,
concept_frontmatter_values=concept_frontmatter_values,
), ),
decision.reasons, decision.reasons,
) )
@ -804,6 +898,7 @@ def process_inbox(
pdf_headings: bool = False, pdf_headings: bool = False,
heading_reserve: Callable[[str], bool] | None = None, heading_reserve: Callable[[str], bool] | None = None,
ocr: bool = False, ocr: bool = False,
concept_frontmatter_values: Mapping[str, str] | None = None,
) -> InboxResult: ) -> InboxResult:
"""Convert every file dropped in `inbox_dir` into an OKF concept. """Convert every file dropped in `inbox_dir` into an OKF concept.
@ -821,6 +916,8 @@ def process_inbox(
a reserved `okf_type`, and a missing inbox directory. a reserved `okf_type`, and a missing inbox directory.
""" """
validate_ingested_at(ingested_at) validate_ingested_at(ingested_at)
# Wrong for every file at once, so refused for the run before any is read.
run_values = validate_concept_frontmatter(concept_frontmatter_values or {}, profile=profile)
# Rendered HERE, before anything is read or written, and the result carried # Rendered HERE, before anything is read or written, and the result carried
# to the index write at the bottom. `_render_root_frontmatter` refuses a key # to the index write at the bottom. `_render_root_frontmatter` refuses a key
# the policy does not name, and a refusal must leave no bundle behind -- # the policy does not name, and a refusal must leave no bundle behind --
@ -1077,6 +1174,7 @@ def process_inbox(
source_file=source_name(path), source_file=source_name(path),
units=units, units=units,
source_title=source_title, source_title=source_title,
concept_frontmatter_values=run_values,
) )
if blocked is not None: if blocked is not None:
if blocked.disposition == _DISPOSITION_QUARANTINE: if blocked.disposition == _DISPOSITION_QUARANTINE:
@ -1130,6 +1228,7 @@ def process_inbox(
# unit boundary after it. # unit boundary after it.
span=(0, len(text)), span=(0, len(text)),
source_title=source_title, source_title=source_title,
concept_frontmatter_values=run_values,
), ),
decision.reasons, decision.reasons,
) )