test(consume): hit@8 over six questions against a random-ranker baseline
hit@8 = 5 of 6, every hit at rank 1, against a chance baseline of 1.35 of 6 over a denominator of 629 concepts per question. Wall time 0.51-0.56 s per question; spent 17 970 - 74 838 bytes against a 120 000 limit. Two things this measurement did NOT establish, both in the report: - BOTH known-negative controls FAILED. A question the bundle has no answer to still returns eight excerpts, because no natural Norwegian question is lexically disjoint from a 629-concept corpus under a four-character shared-prefix rule -- measured per token, the interrogative `hvor` reaches 40 concepts, `brukes` 83. So `no_lexical_match` works per concept and not as a whole-question gate: an empty excerpt list is evidence of absence, a full one is not evidence of presence. The fix is named (rarity weighting) and NOT built, because this step's fence freezes the instrument before it is measured. - The question texts were written during execution, after the ranker existed. The plan recorded the gold documents' SIZE profile -- its per-row baselines sum to 1.35 and the sizes used here reproduce that exactly, which is an independent check that this is the set the plan profiled -- but it recorded no question texts, and three of six gold documents could not be pinned uniquely from the sizes. Not a blind evaluation, and the report says so. The scorer is a tool, not a script in a document: `tools/okf_consume_measure.py` takes the gold set as an INPUT because it is tracked in a public repository and an answer key names a consumer's documents. hit_rank, both chance baselines and the document-size census are unit-tested; the corpus run is a measurement. Public-file rule, checked with a pattern DERIVED from the corpus's own 39 document names rather than hand-picked, and shown able to find first (67 hits on the bundle's own index): zero corpus document names in any tracked file in this repository. One leak was found and removed on the way -- a corpus concept name in a code comment and a hardcoded corpus path in a test. Suite run after git add: 1230 passed, mypy --strict clean on 27 files, ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
51735fa7a8
commit
d7751c0b9a
6 changed files with 598 additions and 39 deletions
185
docs/2026-09-07-okf-konsumskill-maaling.md
Normal file
185
docs/2026-09-07-okf-konsumskill-maaling.md
Normal file
|
|
@ -0,0 +1,185 @@
|
||||||
|
# The consumption pre-pass, measured — hit@8 on a 629-concept bundle
|
||||||
|
|
||||||
|
**Date:** 2026-09-07 · **Instrument:** `tools/okf_consume.py` at `51735fa`,
|
||||||
|
scored by `tools/okf_consume_measure.py` · **Corpus:** a 629-concept, 39-document
|
||||||
|
OKF v0.2 segmented bundle built by this library's own doors on 2026-09-03.
|
||||||
|
|
||||||
|
This is the first time `docs/consumption-contract.md` has been run against a
|
||||||
|
bundle. The contract has existed since S5 with no instrument, and a rule with no
|
||||||
|
instrument is a rule nobody can be shown to have broken. What follows is that
|
||||||
|
instrument's first numbers, with the two things a first number needs: the
|
||||||
|
denominator it was measured over, and the chance baseline it should be read
|
||||||
|
against.
|
||||||
|
|
||||||
|
**Read the headline with its granularity attached.** The order this work came
|
||||||
|
from asked for "≥ 5 spørsmål der **fasit-konseptet** er kjent" — gold at
|
||||||
|
**concept** granularity. The operator answered the open question on it
|
||||||
|
(2026-09-07 04:33Z) with option (a): gold is the top-level **document** the
|
||||||
|
question names, taken from the corpus's own document titles and never from
|
||||||
|
anything the ranker computes. **Five of the six rows below are therefore
|
||||||
|
DOCUMENT-granularity, which is a weaker claim than the order asked for.** One row
|
||||||
|
— the price form — has a gold document holding exactly one concept, so document
|
||||||
|
and concept granularity coincide there, and it is the only concept-level result
|
||||||
|
in this report.
|
||||||
|
|
||||||
|
## The result
|
||||||
|
|
||||||
|
| figure | value |
|
||||||
|
|---|---|
|
||||||
|
| questions | 6 positive + 2 known-negative controls |
|
||||||
|
| **hit@8** | **5 of 6**, every hit at **rank 1** |
|
||||||
|
| expected by chance | **1.35 of 6** |
|
||||||
|
| denominator | **629 concepts** per question, every one accounted for |
|
||||||
|
| `k` | 8 |
|
||||||
|
| known-negative controls | **2 of 2 FAILED** — see below |
|
||||||
|
| `spent` per question | 17 970 – 74 838 bytes (median 20 182), limit 120 000 |
|
||||||
|
| whole payload per question | 109 951 – 165 109 bytes |
|
||||||
|
| wall time per question | 0.51 – 0.56 s (629 concepts, 1.95 MB of concept text) |
|
||||||
|
|
||||||
|
Five hits at rank 1 against an expectation of 1.35 is a real signal and a small
|
||||||
|
one: **n = 6**. Nothing here supports a claim about the ranker's behaviour on a
|
||||||
|
seventh question, and this report makes none.
|
||||||
|
|
||||||
|
### Per-row, with the baseline each row must be read against
|
||||||
|
|
||||||
|
The per-question gold ids are **not in this file**. They name documents in a
|
||||||
|
consumer's corpus, and this repository keeps corpus paths and document titles out
|
||||||
|
of tracked files; they live in the local project directory beside the run, and go
|
||||||
|
to `portfolio-optimiser` by coord. What is publishable is the shape:
|
||||||
|
|
||||||
|
| row | gold concepts | hit@8 | rank | chance, analytic | chance, empirical |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| 1 — **the concept-level row** | 1 | ✅ | 1 | 0.013 | 0.013 |
|
||||||
|
| 2 | 49 | ✅ | 1 | 0.479 | 0.483 |
|
||||||
|
| 3 | 20 | ✅ | 1 | 0.229 | 0.231 |
|
||||||
|
| 4 | 43 | ✅ | 1 | 0.434 | 0.438 |
|
||||||
|
| 5 | 11 | ✅ | 1 | 0.132 | 0.134 |
|
||||||
|
| 6 | 5 | ❌ | — | 0.062 | 0.063 |
|
||||||
|
|
||||||
|
**Why every row carries its own baseline.** A document-prefix hit is *easier for
|
||||||
|
a large gold document*: with gold sets spanning 1 to 49 concepts, a raw count of
|
||||||
|
4/6 could be produced by a ranker that had learned nothing. Row 2 alone is 48 %
|
||||||
|
likely by chance; row 1 is 1.3 %. The two hardest rows (1 and 6, at 0.013 and
|
||||||
|
0.062) are one hit and one miss.
|
||||||
|
|
||||||
|
The empirical baseline is 20 000 draws at seed 20260907. It agrees with the
|
||||||
|
analytic form to within about half a percentage point — **not** to three decimal
|
||||||
|
places, and this report does not claim it does.
|
||||||
|
|
||||||
|
**The miss, diagnosed rather than absorbed.** The gold's best concept ranked
|
||||||
|
**11 overall**, just outside `k = 8`, and comes back at rank 11 for `k = 12` and
|
||||||
|
`k = 16`. Cause: the profile matches two tokens when they share four leading
|
||||||
|
characters, so a compound question term matches two much larger sibling
|
||||||
|
documents in the same subject area, and those two fill all eight slots. The
|
||||||
|
document prior does not rescue it (0.333 against 0.433 and 0.415). This is a
|
||||||
|
near-miss at the reported `k`, not an absence — and it is the shape of failure to
|
||||||
|
expect from this ranker: crowding by large same-prefix neighbours.
|
||||||
|
|
||||||
|
## The controls that failed, stated first rather than last
|
||||||
|
|
||||||
|
The order required a known-negative: a question with no answer in the bundle
|
||||||
|
must produce an empty or "not found" payload, never a confident guess. **Both
|
||||||
|
known-negative questions produced eight excerpts.** The control failed.
|
||||||
|
|
||||||
|
| control | tokens reaching nothing | delivered |
|
||||||
|
|---|---|---|
|
||||||
|
| the question chosen as planned | 2 of 6 | 8 |
|
||||||
|
| the most lexically disjoint question found | 4 of 6 | 8 |
|
||||||
|
|
||||||
|
**Why, measured rather than guessed.** The pre-pass has a per-concept relevance
|
||||||
|
rule (`no_lexical_match`) that withholds a concept the question does not reach,
|
||||||
|
and it works: it fires on 600-odd concepts per run. It does **not** work as a
|
||||||
|
whole-question gate, because no natural Norwegian question is lexically disjoint
|
||||||
|
from a 629-concept corpus under a four-character shared-prefix rule. Measured
|
||||||
|
per token: the interrogative `hvor` reaches 40 concepts, `mange` 17, `brukes` 83,
|
||||||
|
`etablert` 38. Four candidate questions were probed and none reached zero on
|
||||||
|
every token. The first control's content words did genuinely match — one of them
|
||||||
|
reached 17 concepts describing welded seams in floor covering, which is a correct
|
||||||
|
lexical answer and a useless topical one.
|
||||||
|
|
||||||
|
**Consequence for anyone using this, stated in the skill as well as here: an
|
||||||
|
empty `excerpts` list is evidence of absence; a full one is not evidence of
|
||||||
|
presence.** When the delivered excerpts do not answer the question, the right
|
||||||
|
output is `[sourced-not-sufficient]` plus a statement that the cut found nothing
|
||||||
|
responsive — not eight answers.
|
||||||
|
|
||||||
|
**What would fix it, named and not built.** Rarity weighting: a token matching a
|
||||||
|
large fraction of the corpus carries little information, and requiring a
|
||||||
|
delivered concept to match at least one *informative* token would make the gate
|
||||||
|
work. Smoothed IDF is on the plan's own list of mechanisms worth lifting and
|
||||||
|
this build does not implement it. It is not attempted here because this step's
|
||||||
|
scope fence freezes the instrument before it is measured — tuning a ranker
|
||||||
|
against the questions it is being scored on produces a number about the tuning.
|
||||||
|
|
||||||
|
## Honesty limits
|
||||||
|
|
||||||
|
Six, and the first three matter most.
|
||||||
|
|
||||||
|
1. **n = 6.** Six questions is a small sample and the table says so on every
|
||||||
|
reading. The chance baseline is what keeps it from being meaningless, not
|
||||||
|
what makes it large.
|
||||||
|
2. **The question texts were written after the ranker existed.** The plan fixed
|
||||||
|
the gold documents' *size profile* before any ranker existed — its recorded
|
||||||
|
per-row baselines sum to 1.35, and the sizes used here reproduce that total
|
||||||
|
exactly, which is an independent check that this is the gold set the plan
|
||||||
|
profiled. But the plan recorded no question **texts**, so those were written
|
||||||
|
during execution. Three of the six gold documents also could not be pinned
|
||||||
|
uniquely from the recorded sizes and were chosen among equal-sized
|
||||||
|
candidates. The gold documents themselves came from the corpus's own titles,
|
||||||
|
per the operator's answer, and not from anything the ranker computes — but
|
||||||
|
this is not a blind evaluation and must not be quoted as one.
|
||||||
|
3. **One ranking change was made with a gold answer visible.** The stage-one
|
||||||
|
document prior originally *summed* lexical overlap across a document's units,
|
||||||
|
which measures document size: measured on this corpus, a 79-concept document
|
||||||
|
scored 6.0 (0.076 each) against the price document's 2.0 over 1 concept, and
|
||||||
|
the price gold fell outside the top 8. The prior is now a density. The fix is
|
||||||
|
justified by the scoring function's own arithmetic rather than by the answer
|
||||||
|
— a prior that grows with size is measuring size — but it was found by running
|
||||||
|
row 1, and the ranker is therefore not blind to that row.
|
||||||
|
4. **Five of six rows are DOCUMENT granularity**, not the concept granularity
|
||||||
|
the order asked for. Row 1 is the exception and is labelled throughout.
|
||||||
|
5. **Two of the three states this contract makes consumers distinguish have
|
||||||
|
denominator zero in this corpus.** `adjudicated` never occurs (all 618 present
|
||||||
|
values are `proposed`; 11 concepts carry no key and read as `unknown`), and
|
||||||
|
`verified` is absent on all 629 — anchored and unanchored — so every excerpt
|
||||||
|
is `unverified`. `type: verdict` is likewise absent on all 629, which makes
|
||||||
|
the § 9.1 exclusion **vacuous here**. All three are exercised only against a
|
||||||
|
synthetic fixture, and the skill states that rather than implying coverage.
|
||||||
|
6. **The contract checker is a lint, not the acceptance.** Measured: it returns
|
||||||
|
0 findings on a payload declaring `considered=0, withheld=0, delivered=0`
|
||||||
|
paired with the *unfilled* template. A green checker means the payload carries
|
||||||
|
what a claim would have to rest on — not that the claim is sound. The real
|
||||||
|
gates are the negative controls, the partition assertion, the placeholder scan
|
||||||
|
and the digest test, and none of those is something the checker performs.
|
||||||
|
|
||||||
|
## What the contract's own gates reported
|
||||||
|
|
||||||
|
| gate | result |
|
||||||
|
|---|---|
|
||||||
|
| SS 5.2, `considered == withheld + delivered` | closes on every run, by construction |
|
||||||
|
| SS 5.3, every withheld concept names its rule | 621 of 621 on the row-1 run, from a closed set of six |
|
||||||
|
| SS 7.4, the instrument reproduced a known-positive before reporting | 10 349 expected, 10 349 measured, plus a second route (`wc -c` = 10 060, delta 289) |
|
||||||
|
| SS 7.2, `spent` is the **delivered set** | 74 838 spent against a 120 000 limit on row 1 — where a whole-payload reading would have put 165 109 against the same limit and refused |
|
||||||
|
| SS 3.3, the ref is a content identity, not a declared version | `sha256-tree:` over the reachable tree; `--ref` asserts and never overrides |
|
||||||
|
| SS 9.2, no directory enumerated | the index walk reaches 629 concepts, exactly the set the forbidden method finds |
|
||||||
|
| contract check on a real payload | `conformant: 14 rules over 8 excerpts and 621 withheld entries, 0 findings`, exit 0 |
|
||||||
|
| negative control: broken identity | `NOT conformant`, 2 findings, exit 1 |
|
||||||
|
| negative control: missing payload | exit 2 |
|
||||||
|
|
||||||
|
## Reproduce
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# One payload, and the checker on it.
|
||||||
|
.venv/bin/python tools/okf_consume.py <bundle> \
|
||||||
|
--question "Hvordan skal prisene fylles ut?" --out payload.json
|
||||||
|
.venv/bin/python tools/okf_contract_check.py \
|
||||||
|
--skill skills/okf-consume/SKILL.md --payload payload.json
|
||||||
|
|
||||||
|
# The whole table. The gold set is an INPUT and is not in this repository.
|
||||||
|
.venv/bin/python tools/okf_consume_measure.py <bundle> --questions <gold-set.json>
|
||||||
|
```
|
||||||
|
|
||||||
|
The gold-set file's shape is in `tools/okf_consume_measure.py`'s module
|
||||||
|
docstring. Determinism is asserted by the suite, not by inspection: two runs of
|
||||||
|
the same `(bundle, question, k, limit)` produce byte-identical output, and the
|
||||||
|
module reaches no clock.
|
||||||
2
tests/fixtures/consume-bundle/krav/index.md
vendored
2
tests/fixtures/consume-bundle/krav/index.md
vendored
|
|
@ -1,4 +1,4 @@
|
||||||
- [Prissammenstilling](prissammenstilling.md) — adjudication: proposed
|
- [Pristabell](pristabell.md) — adjudication: proposed
|
||||||
- [Årlig kontroll av anlegget](aarlig-kontroll.md) — adjudication: adjudicated
|
- [Årlig kontroll av anlegget](aarlig-kontroll.md) — adjudication: adjudicated
|
||||||
- [verdict-lookalike](verdict-lookalike.md) — adjudication: proposed
|
- [verdict-lookalike](verdict-lookalike.md) — adjudication: proposed
|
||||||
- [Loggnotat](loggnotat.md) — adjudication: proposed
|
- [Loggnotat](loggnotat.md) — adjudication: proposed
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
---
|
---
|
||||||
type: reference
|
type: reference
|
||||||
title: Prissammenstilling
|
title: Pristabell
|
||||||
source_file: Prisskjema.xlsx
|
source_file: Pristabell.xlsx
|
||||||
source_sha256: 1111111111111111111111111111111111111111111111111111111111111111
|
source_sha256: 1111111111111111111111111111111111111111111111111111111111111111
|
||||||
ingested_at: 2026-09-01T00:00:00Z
|
ingested_at: 2026-09-01T00:00:00Z
|
||||||
adjudication: proposed
|
adjudication: proposed
|
||||||
|
|
@ -9,7 +9,7 @@ bundle_id: consume-fixture
|
||||||
verified: [{ by: process:okf-check, at: 2026-09-01T00:00:00Z }]
|
verified: [{ by: process:okf-check, at: 2026-09-01T00:00:00Z }]
|
||||||
---
|
---
|
||||||
|
|
||||||
## Prissammenstilling
|
## Pristabell
|
||||||
|
|
||||||
Prisene fylles ut i dette skjemaet. Summen av alle poster overfoeres til
|
Prisene fylles ut i dette skjemaet. Summen av alle poster overfoeres til
|
||||||
tilbudsbrevet.
|
tilbudsbrevet.
|
||||||
|
|
@ -37,6 +37,7 @@ PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||||
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
|
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
|
||||||
|
|
||||||
import okf_consume # noqa: E402
|
import okf_consume # noqa: E402
|
||||||
|
import okf_consume_measure # noqa: E402
|
||||||
import okf_contract_check # noqa: E402
|
import okf_contract_check # noqa: E402
|
||||||
|
|
||||||
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
|
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
|
||||||
|
|
@ -425,8 +426,8 @@ def test_a_concept_in_a_high_scoring_document_outranks_an_equally_lexical_one()
|
||||||
question = "Hvordan skal prisene fylles ut?"
|
question = "Hvordan skal prisene fylles ut?"
|
||||||
lifted = okf_consume.concept_scores(concepts, question, {"krav": 10.0, "dyp": 0.0})
|
lifted = okf_consume.concept_scores(concepts, question, {"krav": 10.0, "dyp": 0.0})
|
||||||
dropped = okf_consume.concept_scores(concepts, question, {"krav": 0.0, "dyp": 10.0})
|
dropped = okf_consume.concept_scores(concepts, question, {"krav": 0.0, "dyp": 10.0})
|
||||||
krav_first = [c.concept_id for c, _, _ in lifted].index("krav/prissammenstilling")
|
krav_first = [c.concept_id for c, _, _ in lifted].index("krav/pristabell")
|
||||||
krav_later = [c.concept_id for c, _, _ in dropped].index("krav/prissammenstilling")
|
krav_later = [c.concept_id for c, _, _ in dropped].index("krav/pristabell")
|
||||||
assert krav_first < krav_later
|
assert krav_first < krav_later
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -442,7 +443,7 @@ def test_the_price_concept_leads_on_the_price_question_in_the_fixture() -> None:
|
||||||
concepts = _fixture_concepts()
|
concepts = _fixture_concepts()
|
||||||
scores = okf_consume.document_scores(FIXTURE, "Hvordan skal prisene fylles ut?")
|
scores = okf_consume.document_scores(FIXTURE, "Hvordan skal prisene fylles ut?")
|
||||||
ranked = okf_consume.concept_scores(concepts, "Hvordan skal prisene fylles ut?", scores)
|
ranked = okf_consume.concept_scores(concepts, "Hvordan skal prisene fylles ut?", scores)
|
||||||
assert ranked[0][0].concept_id == "krav/prissammenstilling"
|
assert ranked[0][0].concept_id == "krav/pristabell"
|
||||||
|
|
||||||
|
|
||||||
# --- Step 7: the cut ----------------------------------------------------------
|
# --- Step 7: the cut ----------------------------------------------------------
|
||||||
|
|
@ -562,7 +563,7 @@ def test_excerpts_come_back_in_rank_order_and_carry_that_rank() -> None:
|
||||||
# second one.
|
# second one.
|
||||||
delivered, _, _ = _cut_fixture()
|
delivered, _, _ = _cut_fixture()
|
||||||
assert [excerpt["rank"] for excerpt in delivered] == list(range(1, len(delivered) + 1))
|
assert [excerpt["rank"] for excerpt in delivered] == list(range(1, len(delivered) + 1))
|
||||||
assert delivered[0]["concept_id"] == "krav/prissammenstilling"
|
assert delivered[0]["concept_id"] == "krav/pristabell"
|
||||||
|
|
||||||
|
|
||||||
# --- Step 8: the payload ------------------------------------------------------
|
# --- Step 8: the payload ------------------------------------------------------
|
||||||
|
|
@ -765,16 +766,33 @@ def test_spent_is_the_delivered_set_where_the_whole_payload_reading_would_refuse
|
||||||
assert int(budget["spent"]) <= int(budget["limit"])
|
assert int(budget["spent"]) <= int(budget["limit"])
|
||||||
|
|
||||||
|
|
||||||
|
#: The gold set is LOCAL-ONLY: it names corpus documents, which never reach a
|
||||||
|
#: tracked file here. The test reads it rather than restating it, so this file
|
||||||
|
#: carries the assertion and not the answer key.
|
||||||
|
GOLD_SET = PROJECT_ROOT / ".claude/projects/2026-09-07-okf-consume-prepass/hit-at-k-questions.json"
|
||||||
|
|
||||||
|
|
||||||
@requires_k2
|
@requires_k2
|
||||||
def test_the_price_form_gold_is_delivered_for_the_price_question() -> None:
|
@pytest.mark.skipif(not GOLD_SET.is_file(), reason=f"the local gold set is absent ({GOLD_SET})")
|
||||||
# SC6. The one question whose gold is confirmed by a signal from outside
|
def test_every_gold_document_in_the_local_set_is_reached_or_named_as_a_miss() -> None:
|
||||||
# this repository: a live model reached that directory unprompted in three
|
# SC5 and SC6 together, run against the answer key rather than a literal.
|
||||||
# navigation steps on 2026-09-06.
|
# Row 1's gold is the one confirmed by a signal from outside this
|
||||||
payload = okf_consume.build_payload(K2_BUNDLE, question="Hvordan skal prisene fylles ut?")
|
# repository -- a live model reached that document unprompted in three
|
||||||
excerpts = payload["excerpts"]
|
# navigation steps on 2026-09-06 -- and its gold document holds exactly one
|
||||||
assert isinstance(excerpts, list)
|
# concept, so it is also the one concept-granularity row.
|
||||||
ids = [str(excerpt["concept_id"]) for excerpt in excerpts]
|
spec = json.loads(GOLD_SET.read_text(encoding="utf-8"))
|
||||||
assert "del-ii-bilag-7-prisskjema/prissammenstilling-sheet-1" in ids
|
questions = spec["questions"]
|
||||||
|
assert len(questions) >= 5, "fewer than five questions is not the measurement"
|
||||||
|
hits = 0
|
||||||
|
for entry in questions:
|
||||||
|
payload = okf_consume.build_payload(K2_BUNDLE, question=entry["question"])
|
||||||
|
excerpts = payload["excerpts"]
|
||||||
|
assert isinstance(excerpts, list)
|
||||||
|
if okf_consume_measure.hit_rank(excerpts, entry["gold_document"]) is not None:
|
||||||
|
hits += 1
|
||||||
|
# The published bar, and the published number. A regression that drops a
|
||||||
|
# row goes red here rather than in a document nobody re-runs.
|
||||||
|
assert hits == 5, f"hit@8 moved: {hits} of {len(questions)}"
|
||||||
|
|
||||||
|
|
||||||
# --- Step 9: the CLI ----------------------------------------------------------
|
# --- Step 9: the CLI ----------------------------------------------------------
|
||||||
|
|
@ -970,22 +988,34 @@ def test_the_shipped_example_payload_is_current_and_regenerates_byte_for_byte()
|
||||||
assert shipped == regenerated
|
assert shipped == regenerated
|
||||||
|
|
||||||
|
|
||||||
def test_no_k2_concept_path_or_document_title_reaches_the_tracked_skill() -> None:
|
@requires_k2
|
||||||
# CLAUDE.md's public-file rule, with the pattern widened to the bare
|
def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
|
||||||
# basenames a report is most likely to leak, and shown capable of finding
|
# CLAUDE.md's public-file rule. The pattern is DERIVED from the corpus's own
|
||||||
# against the bundle's own index before its zero here is believed.
|
# top-level document names at run time rather than hand-picked, so it covers
|
||||||
leak = re.compile(
|
# every document rather than the six someone thought of -- and so this
|
||||||
r"del-ii-bilag|del-i-vedlegg|del-i-konkurranse|prisskjema|prissammenstilling|stange",
|
# tracked file carries no corpus name of its own.
|
||||||
re.IGNORECASE,
|
documents = sorted(
|
||||||
|
{concept_id.split("/", 1)[0] for concept_id in okf_consume.enumerate_concepts(K2_BUNDLE)}
|
||||||
)
|
)
|
||||||
|
assert len(documents) > 30, "too few documents to be the real corpus"
|
||||||
|
leak = re.compile("|".join(re.escape(name) for name in documents), re.IGNORECASE)
|
||||||
|
|
||||||
|
# The known-positive, first: the pattern must be shown able to find before
|
||||||
|
# its zero counts as a measurement.
|
||||||
|
control = (K2_BUNDLE / "index.md").read_text(encoding="utf-8")
|
||||||
|
assert leak.findall(control), "the pattern cannot find; the zeros below would mean nothing"
|
||||||
|
|
||||||
tracked = [
|
tracked = [
|
||||||
SKILL,
|
SKILL,
|
||||||
SKILL.parent / "references" / "README.md",
|
SKILL.parent / "references" / "README.md",
|
||||||
SKILL.parent / "references" / "example-payload.json",
|
SKILL.parent / "references" / "example-payload.json",
|
||||||
|
PROJECT_ROOT / "tools" / "okf_consume.py",
|
||||||
|
PROJECT_ROOT / "tools" / "okf_consume_measure.py",
|
||||||
|
PROJECT_ROOT / "tests" / "test_okf_consume.py",
|
||||||
|
PROJECT_ROOT / "docs" / "2026-09-07-okf-konsumskill-maaling.md",
|
||||||
|
PROJECT_ROOT / "README.md",
|
||||||
|
PROJECT_ROOT / "CLAUDE.md",
|
||||||
]
|
]
|
||||||
if K2_BUNDLE.is_dir():
|
|
||||||
control = (K2_BUNDLE / "index.md").read_text(encoding="utf-8")
|
|
||||||
assert leak.findall(control), "the pattern cannot find; its zero below would mean nothing"
|
|
||||||
for path in tracked:
|
for path in tracked:
|
||||||
assert leak.findall(path.read_text(encoding="utf-8")) == [], path
|
assert leak.findall(path.read_text(encoding="utf-8")) == [], path
|
||||||
|
|
||||||
|
|
@ -997,3 +1027,56 @@ def test_the_readme_consume_section_states_the_rule_count_the_code_emits() -> No
|
||||||
assert len(okf_consume.WITHHOLDING_RULES) == 6
|
assert len(okf_consume.WITHHOLDING_RULES) == 6
|
||||||
assert "closed set of six" in readme
|
assert "closed set of six" in readme
|
||||||
assert "tools/okf_consume.py" in readme
|
assert "tools/okf_consume.py" in readme
|
||||||
|
|
||||||
|
|
||||||
|
# --- Step 11: the measurement scorer -----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_gold_at_rank_three_is_a_hit_at_eight_and_a_miss_at_two() -> None:
|
||||||
|
excerpts = [
|
||||||
|
{"concept_id": "other-doc/a", "rank": 1},
|
||||||
|
{"concept_id": "other-doc/b", "rank": 2},
|
||||||
|
{"concept_id": "gold-doc/c", "rank": 3},
|
||||||
|
]
|
||||||
|
assert okf_consume_measure.hit_rank(excerpts, "gold-doc") == 3
|
||||||
|
assert okf_consume_measure.hit_rank(excerpts[:2], "gold-doc") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_document_prefix_test_does_not_match_a_longer_document_name() -> None:
|
||||||
|
# Without the trailing slash, a gold document `bilag-7` would count a
|
||||||
|
# concept in `bilag-70` as a hit -- a false positive that inflates the
|
||||||
|
# headline number and looks like a correct answer.
|
||||||
|
excerpts = [{"concept_id": "bilag-70/a", "rank": 1}]
|
||||||
|
assert okf_consume_measure.hit_rank(excerpts, "bilag-7") is None
|
||||||
|
assert okf_consume_measure.hit_rank([{"concept_id": "bilag-7", "rank": 1}], "bilag-7") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_analytic_chance_baseline_is_exact_for_a_single_gold_concept() -> None:
|
||||||
|
# One gold concept in a 629-concept corpus, drawn 8: exactly 8/629, which
|
||||||
|
# is derivable by hand and pins the closed form.
|
||||||
|
assert okf_consume_measure.chance_analytic(1, 629, 8) == pytest.approx(8 / 629)
|
||||||
|
assert okf_consume_measure.chance_analytic(629, 629, 8) == 1.0
|
||||||
|
assert okf_consume_measure.chance_analytic(0, 629, 8) == 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_empirical_baseline_reproduces_the_analytic_one_on_a_fixed_seed() -> None:
|
||||||
|
# Two routes to one number. The agreement is about half a percentage point
|
||||||
|
# at 20 000 trials -- asserted at that tolerance rather than at the three
|
||||||
|
# decimal places the plan claimed, because the looser figure is the one
|
||||||
|
# that is true.
|
||||||
|
for gold in (1, 5, 11, 20, 43, 49):
|
||||||
|
analytic = okf_consume_measure.chance_analytic(gold, 629, 8)
|
||||||
|
empirical = okf_consume_measure.chance_empirical(gold, 629, 8)
|
||||||
|
assert abs(analytic - empirical) < 0.01, gold
|
||||||
|
|
||||||
|
|
||||||
|
def test_document_sizes_counts_root_level_concepts_as_their_own_document() -> None:
|
||||||
|
sizes = okf_consume_measure.document_sizes(("a/1", "a/2", "b/1", "root-concept"))
|
||||||
|
assert sizes == {"a": 2, "b": 1, "root-concept": 1}
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_measurement_instrument_names_no_corpus_document() -> None:
|
||||||
|
# It is tracked and public; the answer key is an input, never a constant.
|
||||||
|
source = (PROJECT_ROOT / "tools" / "okf_consume_measure.py").read_text(encoding="utf-8")
|
||||||
|
leak = re.compile(r"del-ii-bilag|del-i-vedlegg|del-i-konkurranse|prisskjema|stange", re.I)
|
||||||
|
assert leak.findall(source) == []
|
||||||
|
|
|
||||||
|
|
@ -491,19 +491,19 @@ MIN_TOKEN_LENGTH = 3
|
||||||
#: How many leading characters two tokens must share to count as a match.
|
#: How many leading characters two tokens must share to count as a match.
|
||||||
#:
|
#:
|
||||||
#: THIS INSTRUMENT'S OWN CONSTANT, and a measurement rather than a preference.
|
#: THIS INSTRUMENT'S OWN CONSTANT, and a measurement rather than a preference.
|
||||||
#: Token equality fails on Norwegian compounds: the question word `prisene`
|
#: Token equality fails on Norwegian compounds: a question's inflected noun
|
||||||
#: equals none of a price-form concept's `title`, `source_file` or path tokens.
|
#: equals none of the tokens in a concept's `title`, `source_file` or path when
|
||||||
#: Plain substring containment does not save it either -- neither `prisene` nor
|
#: the concept spells the same subject as a compound. Plain substring
|
||||||
#: `prissammenstilling` contains the other. A shared prefix does: `pris|ene` and
|
#: containment does not save it either -- of `varene` and `varemottak`, neither
|
||||||
#: `pris|sammenstilling` share 4.
|
#: contains the other. A shared prefix does: `vare|ne` and `vare|mottak` share 4.
|
||||||
#:
|
#:
|
||||||
#: MEASURED HERE, 2026-09-07, over the 629-concept K2 corpus for the question
|
#: MEASURED 2026-09-07 over a 629-concept corpus, for one question's subject
|
||||||
#: token `prisene`: a 4-character floor matches **3** concepts -- over the
|
#: token: a 4-character floor matches **3** concepts -- over the concept id
|
||||||
#: concept id alone AND over title + `source_file` + id together, the same 3 --
|
#: alone AND over title + `source_file` + id together, the same 3 -- and the
|
||||||
#: and the price-form gold is among them. The plan this implements recorded 6
|
#: gold concept is among them. The plan this implements recorded 6 for the same
|
||||||
#: for the same measurement; 6 is not reproducible with this rule, and the
|
#: measurement; 6 is not reproducible with this rule, and the number that is
|
||||||
#: number that is reproducible is the one carried here. A 3-character floor
|
#: reproducible is the one carried here. A 3-character floor over-matches
|
||||||
#: over-matches Norwegian function words.
|
#: Norwegian function words.
|
||||||
MIN_SHARED_PREFIX = 4
|
MIN_SHARED_PREFIX = 4
|
||||||
|
|
||||||
_TOKEN_SPLIT_RE = re.compile(r"[^0-9a-zà-öø-ÿ]+")
|
_TOKEN_SPLIT_RE = re.compile(r"[^0-9a-zà-öø-ÿ]+")
|
||||||
|
|
|
||||||
291
tools/okf_consume_measure.py
Normal file
291
tools/okf_consume_measure.py
Normal file
|
|
@ -0,0 +1,291 @@
|
||||||
|
"""Measure a consumption pre-pass: hit@k per question, against a chance baseline.
|
||||||
|
|
||||||
|
Measure, don't build. This instrument calls `tools/okf_consume.py` exactly as a
|
||||||
|
consumer would and scores what comes back; it contains no ranking of its own,
|
||||||
|
because a scorer sharing code with the thing it scores measures its own
|
||||||
|
agreement rather than the ranker's performance.
|
||||||
|
|
||||||
|
**The gold set is an INPUT, never a constant here.** It arrives as a JSON file
|
||||||
|
naming, per question, the gold document. Two reasons, and the second is the
|
||||||
|
binding one:
|
||||||
|
|
||||||
|
- An instrument with a corpus baked in measures one corpus.
|
||||||
|
- This file is tracked in a PUBLIC repository. A gold set names documents in a
|
||||||
|
consumer's corpus, and this repository's standing rule keeps corpus paths and
|
||||||
|
document titles out of tracked files. The instrument is publishable; the
|
||||||
|
answer key is not, so it lives beside the run and not beside the code.
|
||||||
|
|
||||||
|
**hit@k is scored at DOCUMENT granularity by default**, which is a weaker claim
|
||||||
|
than concept granularity and is labelled as such wherever the number is used: a
|
||||||
|
gold document counts as hit at k when any of the top-k excerpts has that
|
||||||
|
document as its `concept_id` path prefix, and the reported rank is the position
|
||||||
|
of the first such excerpt. A gold document holding one concept is the only case
|
||||||
|
where document and concept granularity coincide.
|
||||||
|
|
||||||
|
**Every row carries its own chance baseline**, computed two ways, because a
|
||||||
|
document-prefix hit is easier for a large gold document: a gold set spanning one
|
||||||
|
to fifty concepts makes a raw hit count uninterpretable without the baseline it
|
||||||
|
should be read against.
|
||||||
|
|
||||||
|
The questions file:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"questions": [{"question": "...", "gold_document": "...", "note": "optional"}],
|
||||||
|
"negatives": [{"question": "...", "label": "..."}]}
|
||||||
|
```
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import random
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
||||||
|
|
||||||
|
import okf_consume # noqa: E402
|
||||||
|
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
|
||||||
|
|
||||||
|
#: The seed every empirical baseline in a published table is drawn with. Fixed
|
||||||
|
#: so a reader re-running this reproduces the same figure; the analytic form
|
||||||
|
#: beside it is what makes the empirical one checkable at all.
|
||||||
|
DEFAULT_SEED = 20260907
|
||||||
|
|
||||||
|
#: Trials for the empirical baseline. Enough that it lands within about half a
|
||||||
|
#: percentage point of the analytic value, which is the agreement actually
|
||||||
|
#: observed -- NOT three decimal places, and this instrument does not claim it.
|
||||||
|
DEFAULT_TRIALS = 20_000
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Row:
|
||||||
|
"""One question, measured."""
|
||||||
|
|
||||||
|
question: str
|
||||||
|
gold_document: str
|
||||||
|
gold_concepts: int
|
||||||
|
hit: bool
|
||||||
|
rank: int | None
|
||||||
|
considered: int
|
||||||
|
withheld: int
|
||||||
|
delivered: int
|
||||||
|
spent: int
|
||||||
|
payload_bytes: int
|
||||||
|
seconds: float
|
||||||
|
chance_analytic: float
|
||||||
|
chance_empirical: float
|
||||||
|
|
||||||
|
|
||||||
|
def hit_rank(excerpts: Sequence[Mapping[str, object]], gold_document: str) -> int | None:
|
||||||
|
"""The rank of the first excerpt inside `gold_document`, or `None`.
|
||||||
|
|
||||||
|
Prefix on the `concept_id`, which in this library is the concept's
|
||||||
|
bundle-relative path with the suffix removed -- so the test is a string
|
||||||
|
operation on a field the payload already carries. The `+ "/"` is not
|
||||||
|
cosmetic: without it, a gold document `bilag-7` would match a concept in
|
||||||
|
`bilag-70`.
|
||||||
|
"""
|
||||||
|
for excerpt in excerpts:
|
||||||
|
concept_id = str(excerpt.get("concept_id", ""))
|
||||||
|
if concept_id == gold_document or concept_id.startswith(f"{gold_document}/"):
|
||||||
|
rank = excerpt.get("rank")
|
||||||
|
return int(rank) if isinstance(rank, int) else None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def chance_analytic(gold_concepts: int, corpus: int, k: int) -> float:
|
||||||
|
"""P(a uniform random k-of-corpus draw touches any of `gold_concepts`).
|
||||||
|
|
||||||
|
The complement of drawing k items that all miss, computed as a product of
|
||||||
|
hypergeometric terms rather than with factorials, so it stays exact for the
|
||||||
|
sizes a corpus actually reaches.
|
||||||
|
"""
|
||||||
|
if gold_concepts <= 0 or k <= 0 or corpus <= 0:
|
||||||
|
return 0.0
|
||||||
|
miss = 1.0
|
||||||
|
for drawn in range(min(k, corpus)):
|
||||||
|
remaining = corpus - drawn
|
||||||
|
available = corpus - gold_concepts - drawn
|
||||||
|
if available <= 0:
|
||||||
|
return 1.0
|
||||||
|
miss *= available / remaining
|
||||||
|
return 1.0 - miss
|
||||||
|
|
||||||
|
|
||||||
|
def chance_empirical(
|
||||||
|
gold_concepts: int,
|
||||||
|
corpus: int,
|
||||||
|
k: int,
|
||||||
|
*,
|
||||||
|
trials: int = DEFAULT_TRIALS,
|
||||||
|
seed: int = DEFAULT_SEED,
|
||||||
|
) -> float:
|
||||||
|
"""The same probability, drawn rather than derived.
|
||||||
|
|
||||||
|
Kept beside the analytic form because two routes to one number is what makes
|
||||||
|
either believable: an off-by-one in the closed form and a bug in the sampler
|
||||||
|
do not agree by accident.
|
||||||
|
"""
|
||||||
|
rng = random.Random(seed)
|
||||||
|
pool = range(corpus)
|
||||||
|
gold = set(range(gold_concepts))
|
||||||
|
hits = sum(1 for _ in range(trials) if gold & set(rng.sample(pool, k)))
|
||||||
|
return hits / trials
|
||||||
|
|
||||||
|
|
||||||
|
def document_sizes(concept_ids: Sequence[str]) -> dict[str, int]:
|
||||||
|
"""How many concepts each top-level document holds."""
|
||||||
|
sizes: dict[str, int] = {}
|
||||||
|
for concept_id in concept_ids:
|
||||||
|
document = concept_id.split("/", 1)[0]
|
||||||
|
sizes[document] = sizes.get(document, 0) + 1
|
||||||
|
return sizes
|
||||||
|
|
||||||
|
|
||||||
|
def measure_question(
|
||||||
|
bundle: Path, question: str, gold_document: str, *, k: int, sizes: Mapping[str, int]
|
||||||
|
) -> Row:
|
||||||
|
started = time.perf_counter()
|
||||||
|
payload = okf_consume.build_payload(bundle, question=question, k=k)
|
||||||
|
elapsed = time.perf_counter() - started
|
||||||
|
text = okf_consume.serialise(payload)
|
||||||
|
excerpts = payload["excerpts"]
|
||||||
|
counts = payload["denominators"]
|
||||||
|
budget = payload["budget"]
|
||||||
|
assert isinstance(excerpts, list) and isinstance(counts, dict) and isinstance(budget, dict)
|
||||||
|
corpus = int(counts["considered"])
|
||||||
|
gold_concepts = sizes.get(gold_document, 0)
|
||||||
|
rank = hit_rank(excerpts, gold_document)
|
||||||
|
return Row(
|
||||||
|
question=question,
|
||||||
|
gold_document=gold_document,
|
||||||
|
gold_concepts=gold_concepts,
|
||||||
|
hit=rank is not None,
|
||||||
|
rank=rank,
|
||||||
|
considered=corpus,
|
||||||
|
withheld=int(counts["withheld"]),
|
||||||
|
delivered=int(counts["delivered"]),
|
||||||
|
spent=int(budget["spent"]),
|
||||||
|
payload_bytes=len(text.encode("utf-8")),
|
||||||
|
seconds=round(elapsed, 3),
|
||||||
|
chance_analytic=round(chance_analytic(gold_concepts, corpus, k), 3),
|
||||||
|
chance_empirical=round(chance_empirical(gold_concepts, corpus, k), 3),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def token_reach(bundle: Path, question: str) -> dict[str, int]:
|
||||||
|
"""How many concepts each of the question's tokens reaches.
|
||||||
|
|
||||||
|
The known-negative control's own PRECONDITION, measured before any payload
|
||||||
|
is built: a question whose tokens all reach real corpus text is not a
|
||||||
|
question the bundle has no answer to, and scoring it as one would report a
|
||||||
|
property of the question as a property of the ranker.
|
||||||
|
"""
|
||||||
|
root_index = bundle / okf_consume.DEFAULT_PROFILE.index.name
|
||||||
|
root_bundle_id = parse_frontmatter(root_index).get("bundle_id", "")
|
||||||
|
corpus = [
|
||||||
|
set(
|
||||||
|
okf_consume.normalise(
|
||||||
|
f"{concept.title} {concept.concept_id.replace('/', ' ')} {concept.body}"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for concept in (
|
||||||
|
okf_consume.read_concept(
|
||||||
|
bundle / f"{concept_id}{okf_consume.CONCEPT_SUFFIX}",
|
||||||
|
bundle_root=bundle,
|
||||||
|
root_bundle_id=root_bundle_id,
|
||||||
|
)
|
||||||
|
for concept_id in okf_consume.enumerate_concepts(bundle)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
return {
|
||||||
|
token: sum(
|
||||||
|
1
|
||||||
|
for tokens in corpus
|
||||||
|
if any(okf_consume.tokens_match(token, other) for other in tokens)
|
||||||
|
)
|
||||||
|
for token in okf_consume.normalise(question)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
||||||
|
)
|
||||||
|
parser.add_argument("bundle", type=Path, help="the OKF bundle to measure against")
|
||||||
|
parser.add_argument(
|
||||||
|
"--questions", type=Path, required=True, help="JSON gold set (see the module docstring)"
|
||||||
|
)
|
||||||
|
parser.add_argument("--k", type=int, default=okf_consume.DEFAULT_K)
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
args = parse_args(argv)
|
||||||
|
try:
|
||||||
|
spec = json.loads(args.questions.read_text(encoding="utf-8"))
|
||||||
|
except OSError as error:
|
||||||
|
print(f"okf_consume_measure: FAILED - {error}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
sizes = document_sizes(okf_consume.enumerate_concepts(args.bundle))
|
||||||
|
rows = [
|
||||||
|
measure_question(
|
||||||
|
args.bundle, entry["question"], entry["gold_document"], k=args.k, sizes=sizes
|
||||||
|
)
|
||||||
|
for entry in spec.get("questions", [])
|
||||||
|
]
|
||||||
|
negatives = []
|
||||||
|
for entry in spec.get("negatives", []):
|
||||||
|
reach = token_reach(args.bundle, entry["question"])
|
||||||
|
started = time.perf_counter()
|
||||||
|
payload = okf_consume.build_payload(args.bundle, question=entry["question"], k=args.k)
|
||||||
|
elapsed = time.perf_counter() - started
|
||||||
|
counts, budget, withheld = (
|
||||||
|
payload["denominators"],
|
||||||
|
payload["budget"],
|
||||||
|
payload["withheld"],
|
||||||
|
)
|
||||||
|
assert isinstance(counts, dict) and isinstance(budget, dict) and isinstance(withheld, list)
|
||||||
|
negatives.append(
|
||||||
|
{
|
||||||
|
"question": entry["question"],
|
||||||
|
"label": entry.get("label", ""),
|
||||||
|
"token_reach": reach,
|
||||||
|
"tokens_reaching_nothing": sum(1 for value in reach.values() if value == 0),
|
||||||
|
"tokens": len(reach),
|
||||||
|
"delivered": int(counts["delivered"]),
|
||||||
|
"withheld": int(counts["withheld"]),
|
||||||
|
"considered": int(counts["considered"]),
|
||||||
|
"spent": int(budget["spent"]),
|
||||||
|
"payload_bytes": len(okf_consume.serialise(payload).encode("utf-8")),
|
||||||
|
"seconds": round(elapsed, 3),
|
||||||
|
"rules": sorted({str(entry["rule"]) for entry in withheld}),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"k": args.k,
|
||||||
|
"seed": DEFAULT_SEED,
|
||||||
|
"trials": DEFAULT_TRIALS,
|
||||||
|
"documents": len(sizes),
|
||||||
|
"rows": [row.__dict__ for row in rows],
|
||||||
|
"hits": sum(1 for row in rows if row.hit),
|
||||||
|
"expected_by_chance": round(sum(row.chance_analytic for row in rows), 2),
|
||||||
|
"negatives": negatives,
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
Loading…
Add table
Add a link
Reference in a new issue