open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(accounting): a document refused whole is never clean, and the run says how many (H1)

Browse source 
The gate gains a FIFTH column, `Unit.refused`, and it is the only one
that is not a defect in the report: the elements of a document the
build read and persisted nothing of. Their fate is declared honestly,
so `unaccounted` and `double_booked` both stay 0 -- which is exactly
why nothing else could see the loss. `refused_whole` asks its question
only for a corpus that persisted NOTHING, so one refused source beside
an accepted one, the ordinary case on a heterogeneous corpus, reached
row 3 as clean.

Row 3's reason now carries `N element(s) lost with R of D document(s)
refused whole`, and each unclean unit's detail line carries
`refused=` beside u, d, unverified and invalid, with the document's
own rejection code in the note.

On the build side `Accounting.refused` is written into the JSON and
into the `**Accounting**` bullet of `log.md` as `R of D document(s)
refused whole`. The exit code is NOT moved: it belongs to the whole
run, and a corpus holding one unreadable file among many is ordinary,
so the order's other half -- state it in the accounting -- is the one
taken. `okf build` still exits 1 when it persisted nothing at all.

`test_a_corpus_refused_whole_under_the_default_gate_is_red` kept its
point and lost its premise: the numbers still balance, and that is now
asserted as u = 0 and d = 0 rather than as a clean unit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 05:48:57 +02:00
commit d27ca503c8
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 51 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

17
src/llm_ingestion_okf/accounting.py
View file

@ -793,6 +793,19 @@ class Accounting:
def double_booked(self) -> int:
return sum(sum(d.double_booked.values()) for d in self.documents)
@property
def refused(self) -> int:
"""Documents the build read and persisted NOTHING of.
Their elements are booked honestly, as coded rejections, so neither
`unaccounted` nor `double_booked` moves and the loss is invisible in
the two numbers a reader looks at. A run that persisted at least one
document exits 0 -- the exit code belongs to the whole run, and a
corpus holding one unreadable file is the ordinary case -- so this
count is what keeps a partial refusal from being silent (H1).
"""
return sum(1 for d in self.documents if d.status == REJECTED)
@property
def images_found(self) -> int:
return sum(d.counts.get("image", 0) for d in self.documents)
@ -806,6 +819,7 @@ class Accounting:
"accounting_version": ACCOUNTING_VERSION,
"unaccounted": self.unaccounted,
"double_booked": self.double_booked,
"refused": self.refused,
"documents": [d.to_json() for d in self.documents],
"files": [f.to_json() for f in self.files],
}
@ -814,7 +828,8 @@ class Accounting:
lines = [
f"* **Accounting**: {len(self.documents)} document(s) and {len(self.files)} "
f"other file(s); {self.elements} elements found in the sources; "
f"{self.unaccounted} unaccounted, {self.double_booked} double-booked."
f"{self.unaccounted} unaccounted, {self.double_booked} double-booked; "
f"{self.refused} of {len(self.documents)} document(s) refused whole."
]
for doc in self.documents:
if doc.status == REJECTED: