4 releases 16 tags

RSS feed

• v0.10.0 bc39e8091f

  Compare

  v0.10.0 Stable

  ktg released this 2026-09-17 08:01:38 +00:00 | 133 commits to main since this release

  Signed by ktg

  SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q

  Added

  • A bundle carries the images its sources declare (0.10.0). Until now no
    reader in this package fetched, named, described or copied a single image:
    <img>'s attributes were never read, a NISO-STS <graphic> was walked past,
    a PDF was opened for its text alone, the converter's markdown writer dropped
    every picture, and the only writer into a bundle took content: str. The two
    lossiness warnings said so on every run, which made the loss honest and did
    not make it smaller. Measured on R761 Prosesskoden:2025: the process text is
    carried in full while 12 Tabell N-N and 9 Figur N-N captions stand over
    nothing, so process 84's "toleranseklasse ... er gitt i tabell 84-2" points
    at empty space.
    Five readers place, one module decides. pdf (embedded image XObjects),
    docx/pptx/odt/rtf (the converter's media, through --extract-media),
    html/htm (<img src alt>, local paths and inline data URIs) and xml
    (<graphic xlink:href>, resolved against the href and then against a sibling
    graphics/). llm_ingestion_okf.assets decides what an image IS, what it is
    called and how it is pointed at, so "carried N of M" means one thing across
    all five. .xlsx is deliberately excluded: a two-line block inside its pipe
    tables would break the row locator read back out of them.
    The bytes go to assets/ at the bundle root under
    <sha256[:12]>-<the source's own base name>, and the concept carries a
    two-line pointer where the picture stood -- a markdown image, then the
    source's own file name and the size in pixels. A PDF stream that is already a
    file (DCTDecode, JPXDecode) is carried VERBATIM; raw samples are encoded
    to PNG with zlib from the stdlib, so no new dependency and no rasteriser
    version enters an asset's bytes or its content-addressed name. What this
    encoder cannot express exactly -- a stencil mask, a Decode array, CMYK,
    anything but 8-bit samples -- is refused with a code and counted, never
    approximated.
    ON by default, with --no-assets reproducing the pre-0.10.0 bytes.
    Measured over the 43-document reference corpus, two builds of one commit:
    453 -> 454 concepts, 865 -> 867 markdown files, 0 -> 2 964 assets (2 964
    carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
    3 088 s, peak RSS 6.26 -> 8.74 GB, and 422 of 865 markdown files differ. The
    one new concept has a measured cause: the pointers are body text, so a
    section holding 146 of that document's images grew from 19.0 % to 30.6 % of
    the extracted text and crossed --outline-gate's 0.20 share clause.
    log.md states it either way -- "N carried of M found", or NOT CARRIED
    under --no-assets, so a bundle nobody looked for figures in cannot be
    mistaken for a bundle of documents that had none. A concept on this
    repository's own profiles also carries images: N, conditional, counted out
    of the concept's own text.
    The image bytes are NOT screened, and the log says so: the guard is
    text-only, the pointer block passes the gate as body text, and the picture
    beside it passes nothing.
    Door C carries them too. Measured before the repair: importing a bundle
    built with --assets merged 6 of 6 concepts and wrote no assets/ at all,
    so every pointer in the imported bundle named a missing file. Only the assets
    a MERGED concept points at are carried -- an asset belonging to a refused
    concept must not ride in on the back of a cleared one.
    A proposed SPEC section 6.4 for the layout is in
    docs/plan/okf-assets-section-6-4.md; _okf-canonical is not edited from
    here.

  Fixed

  • A markdown image is no longer read as a cross-reference.
    structure._LINK reads [...](target) and never looked at the character in
    front of the bracket, so an asset pointer would have arrived in the index as
    a references edge to a concept that cannot exist -- and the digits in an
    asset's file name would have been read as a document number. The link's span
    is still masked, so the number scan cannot see it either.

  • okf build now runs a real guard, and the bundle says which one (F1).
    From the day the command was packaged until 2026-09-15, corpus.measure
    wired an unconditional approve-everything stub into process_inbox and no
    add_argument call anywhere in the package named a gate -- so the only path
    most people use screened nothing, while pyproject.toml made the guard a
    MANDATORY runtime dependency and the README recommended a composition the
    command line could not reach. Reported from outside by claude-code-llm-wiki
    and reproduced here before anything moved.
    --gate takes guard-trusted-source (the new default),
    guard-user-upload or none, and the name is written into the bundle's
    section 9 log.md either way, so a consumer holding a bundle can tell a
    screened one from an unscreened one. An unknown name is refused rather than
    resolved to the stub: falling back would reproduce the defect with an extra
    step. okf project owns no flag that moves a bundle's bytes and takes the
    default; the corpus harness carries the same flag and the same default,
    because a test holds the two paths byte-equal.
    The default was chosen on a measurement, not on caution. Over the 453
    concept bodies of the pinned reference bundle, PRESET_TRUSTED_SOURCE
    returns the persist disposition on 453 of 453 while PRESET_USER_UPLOAD
    holds 1, taking one of the 39 source documents out. Neither tier waves
    anything through: an invisible carrier and a CRITICAL finding fail secure at
    both, measured against guard 1.4.0. Door B's own library default is
    unchanged at PRESET_USER_UPLOAD -- an inbox drop is an untrusted upload,
    an operator pointing okf build at their own folder is not. The second tier
    ships as guard_adapter.inbox_gate_trusted_source, the three-line adapter
    that module's own docstring describes, rather than as a preset parameter.
    The composition the README recommends is now tested. Before this change
    grep -rl inbox_gate tests/ gave ONE file with 0 occurrences of segment,
    while the nine files passing segmentation= all injected a local warn-stub:
    no test ran a real guard verdict and a segmentation plan in the same call.

  • A fenced code block no longer declares structure (F2). The proposer read
    every line of the extracted text with the same grammars, so # Use the opus[1m] alias inside a ```bash fence became a level-1 ATX heading. Two
    effects, and the smaller one was the visible one: the document was REFUSED
    entirely when the line carried [ or ] (Door B validates a title fail-fast
    and never repairs one) -- 5 of 191 pages of the reporter's corpus -- and the
    concept TITLE was silently taken from somebody's shell session everywhere
    else, on 62 of 191 pages (32.5 %).
    No rule in find_candidates reads a fenced line now: not ATX, not the
    numbered grammar, not a table row, not a bold title, and not Arm D's outline
    run, which selects from the whole line list and would otherwise let a fenced
    install listing decide which run wins. Backtick and tilde fences, up to three
    leading spaces, a closing fence at least as long as its opener, and
    CommonMark's rule that a backtick fence's info string may not contain a
    backtick -- that last one is what keeps a line holding only `okf build`
    from silencing the rest of a document.
    It lands unconditionally rather than behind a flag, and the exposure is
    measured on the bytes: 0 of 865 concept files in the pinned default bundle
    and 0 of the shipped fixtures and goldens that reach the proposer carry a
    fence of either kind, so a rule that can only fire INSIDE a fence cannot have
    moved anything this repository has measured. It is a defect, not a default
    move.

  Added

  • okf quality <bundle> --fasit <json> -- boundary recall against the
    structure the source itself declares (G37b). The bundle-only gate returned
    UNMEASURED and exit 3 on the very arm it was built for, because no
    bundle-only metric reaches it; boundary_share -- declared boundaries that
    became a concept, over declared boundaries -- is the one metric measured that
    orders the arms correctly, and it needs the publisher's own structure, so it
    arrives as an input rather than as a constant. The fasit is a JSON list whose
    rows carry title and norm, validated at the door: a file that is not a
    list, a row missing either key, or anything that is not JSON exits 2 with
    the reason, never a quiet UNMEASURED.
    The normalisation was measured before the metric was built: stripping all
    whitespace and lowercasing reproduces the fasit's own norm from its own
    title on 2 761 of 2 761 rows (keeping only alphanumerics scores 58).
    A boundary is recovered in either of two forms, and both are load-bearing:
    a concept whose normalised title equals norm, or the pair of the concept's
    own directory and its residual title -- because the numbering token a
    publisher glues into a heading lands in the concept id on one route and in the
    title on another. Measured on the known-good arm, the literal form alone
    reaches 22 of 2 761 where the two together reach 2 759; on another
    build of the same product the split is the exact opposite (2 727 literal, 0
    paired). One bar, at the value measured on the pinned artifact: 2 759/2 761,
    corpora = 1. It separates the known-bad arm at 1 148 of 2 761 (41.6 %),
    which is now a FAIL and exit 1 instead of exit 3. --fasit is an
    assertion, like okf consume --ref: a bundle of another product scores 0 of
    2 761 (measured on two of them) and reads FAIL -- the assertion being wrong,
    not the bundle. The bar rests on one product, and the run says so on every
    boundary row. docs/2026-09-12-g37-terskler.md SS 7 carries the premises
    re-measured, the seven bundles, the interval any bar could sit in, and the two
    R761 builds this one fails.

  Unchanged

  • Without --fasit the command is exactly what it was, held by a test: no
    boundary row, and 860019-mdb-100 still exits 3. No version bump, no tag,
    okf check untouched.

  Downloads

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads