feat(gate): two presentation checks the evidence actually supports
Adversarial deep research (25 sources, 124 claims extracted, 25 verified: 11 confirmed / 14 refuted) plus direct measurement of all 18 cloned open/ repos. The useful half of the result is what it REFUSED to support, so the research is recorded in docs/presentation-research-2026-08-03.md rather than being spent and forgotten. - BADGE-COUNT (WARN) — past five badges. Trockman et al., ICSE 2018 (n=294,941 npm packages) measured a non-linear relationship with popularity inflecting at five, motivated by surveyed maintainers calling over-badged READMEs cluttered and "trying too hard". WARN and never ERROR: the coefficient sits in an appendix with no CI or p-value. Counting deliberately uses a NARROWER rule than the existing claim check, so a screenshot or an architecture diagram is never counted as clutter. Fires on 8 of 18. - README-LANGUAGE (WARN) — prose not in the language this repo's readers were declared to speak, via a new `locales` axis in the register. Class is structural, a trait is what the code DOES, a locale is who it is FOR — the standard's own "who the reader is decides what is required". English is the default; ms-ai-architect and okr are declared nb, named by the operator as Norway-only in audience. Stopword-frequency comparison over prose with code stripped: a Norwegian flag name in a shell example cannot decide the document. Fires on exactly those two, silent on all sixteen English repos. One design correction found mid-implementation: the first version returned SKIP when a README had too little prose to judge, which broke a passing fixture and would have stopped any terse repo from ever reaching OK. SKIP is for a check that could not RUN; this one ran, saw everything and found no prose to be in the wrong language — the same shape as "no licence claim to back". Insufficient prose is now OK, and evenly bilingual prose is the SKIP, because there the question is live and unanswered. Deliberately NOT built, because the evidence does not reach: any rule about images, diagrams or terminal recordings (every such claim refuted 0-3); a README length bound (no evidence-based target exists); a section count (would fire on 9 of 18 — textbook "suspect the CHECK"); Mermaid source length and #gh-dark-mode-only (zero occurrences, and the instance limit is not readable via the API, so any threshold would be a guess). Verified live against the operator's own forge (15.0.6+gitea-1.22.0): Mermaid DOES render in README.md — two div.mermaid-block iframes carrying real SVG — while #gh-dark-mode-only landed only in Gitea 1.26.0 and is unavailable here. 103 tests green, up from 92. No version bump: the catalog ref still trails at v0.1.1 against 0.1.3, and starting a second release chain over that is the operator's call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TCAGKZT8h9F46ygzSkhEee
This commit is contained in:
parent
5eba10acdd
commit
dc386d4471
7 changed files with 474 additions and 2 deletions
186
docs/presentation-research-2026-08-03.md
Normal file
186
docs/presentation-research-2026-08-03.md
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
# Presentation research — what "professional" can and cannot be measured as
|
||||
|
||||
**Date:** 2026-08-03 · **Status:** research complete, architecture decision open
|
||||
**Method:** adversarial deep research (107 agents, 25 sources fetched, 124 claims
|
||||
extracted, 25 verified, **11 confirmed / 14 refuted**) + direct measurement of
|
||||
the 18 locally-cloned `open/` repos + live verification against the operator's
|
||||
own Forgejo instance.
|
||||
|
||||
The headline is uncomfortable and load-bearing: **most of what people call
|
||||
"professional presentation" has no evidence base as a requirement.** Half the
|
||||
research question returned nothing that survived adversarial verification. That
|
||||
result constrains what this plugin may honestly check.
|
||||
|
||||
---
|
||||
|
||||
## 1. Measured: the org as it stands
|
||||
|
||||
18 of the 19 registered `open/` repos (`.profile` is not cloned locally).
|
||||
|
||||
| Dimension | Measurement |
|
||||
| --- | --- |
|
||||
| Real images | **0.** Every `![...]` in all 18 READMEs resolves to a badge. |
|
||||
| Mermaid diagrams | 3 blocks total — `voyage` 2, `llm-security` 1, `graceful-handoff` 1. |
|
||||
| README length | 1,239 → 8,725 words. 7× spread. |
|
||||
| H2 sections | 5 → 26. Nine repos sit above the empirical IQR ceiling of 12. |
|
||||
| Badges | 0 → 8. **Six repos carry 7–8.** |
|
||||
| Language | `ms-ai-architect` ships Norwegian body prose *and* a Norwegian changelog table; `okr` ships a Norwegian changelog table. Both are public. |
|
||||
| Heading casing | Mixed *within one file* in 5/18 — "What Is This?" beside "What it does". |
|
||||
| First H2 | `Install` in 15/18. The org's strongest existing coherence signal. |
|
||||
|
||||
`linkedin-studio` was flagged by the crude language heuristic and cleared on
|
||||
inspection — a false positive, discarded before it reached this table.
|
||||
|
||||
## 2. Confirmed by the evidence
|
||||
|
||||
Peer-reviewed, adversarially survived. **Every number here is descriptive of
|
||||
what typical repos do, not prescriptive of what an excellent repo should do.**
|
||||
Inverting them into requirements contradicts the authors' own conclusions — the
|
||||
research flagged this "descriptive-to-normative trap" as its single biggest
|
||||
methodological hazard.
|
||||
|
||||
1. **Structure: median 7 sections, middle 50% between 5 and 12**
|
||||
(Prana et al., *Empirical Software Engineering* 2019, doi
|
||||
10.1007/s10664-018-9660-3; n=393 repos / 4,226 annotated sections). The
|
||||
metric is **sections, not words** — the paper reports no length measurement.
|
||||
Population filtered to ≥2 KB English `.md` software repos.
|
||||
|
||||
2. **Content skew, and the one normative finding in the whole set.** What 97.0%,
|
||||
How 88.5%, References 60.8%, Who 52.9%, Contribution 27.8%, **Why 25.7%,
|
||||
When/status 21.4%**. The authors argue in their own voice that supplying
|
||||
*Why* and *When* is how a repository differentiates itself and signals
|
||||
maturity. This is the only recommendation sourced from authors rather than
|
||||
from inverting a statistic.
|
||||
|
||||
3. **No evidence-based README length target exists.** The only measured
|
||||
baselines (Gaughan et al., CHASE 2025) are of *first versions*, which are
|
||||
stubs — median reading time 14.79 s, 39% readable in under 10 seconds. The
|
||||
authors frame this as a deficiency, not a norm. Any length rule is taste.
|
||||
|
||||
4. **Badges: a measured inflection at 5.** Trockman et al., ICSE 2018 (doi
|
||||
10.1145/3180155.3180209; n=294,941 npm packages): a non-linear relationship
|
||||
with popularity inflecting at 5 badges, motivated by survey respondents
|
||||
calling over-badged READMEs cluttered and "trying too hard". The paper also
|
||||
splits badges into **assessment signals** (a service actually ran an
|
||||
analysis) versus **conventional signals** (a lookup or a static statement —
|
||||
"cheap to produce, therefore easy to fake"), and advises limiting the latter.
|
||||
*This is the only presentation anti-pattern in the entire evidence set with a
|
||||
measured basis.* It aligns exactly with this plugin's existing
|
||||
`BADGE-STATIC-CLAIM` invariant. The coefficient sits in an appendix without
|
||||
CI or p-value, so it supports "badge count is not monotonically beneficial"
|
||||
and **cannot** support a hard "never exceed 5".
|
||||
|
||||
5. **Prose readability is machine-measurable.** Flesch Reading Ease, McAlpine
|
||||
EFLAW, and Linsear Write via Python `textstat`; published medians 50.86 /
|
||||
16.50 / 7.14. Linsear Write is the one designed for technical documents.
|
||||
EFLAW scores accessibility to non-native English readers — it does **not**
|
||||
measure language choice, and confidence here is medium (single source, and
|
||||
the medians describe sparse first drafts).
|
||||
|
||||
## 3. Refuted — evidence you might expect to have, and do not
|
||||
|
||||
Fourteen claims were killed 0–3 or 1–2. The consequential ones:
|
||||
|
||||
- **Every claim about visual content failed.** That images are rare and
|
||||
therefore differentiating; that static images correlate with popularity while
|
||||
GIFs and video do not; that outbound links are the strongest structural
|
||||
feature. **There is currently no evidential basis for any rule about
|
||||
screenshots, diagrams, GIFs, or terminal recordings.** Adding visuals may
|
||||
still be right — but it is taste, and must be argued as taste.
|
||||
- **Nothing survived on cross-repo coherence mechanisms.** No evidence on shared
|
||||
skeletons, org profile pages, catalog hubs, naming taxonomies, or which
|
||||
mechanisms are GitHub-specific. This is the central half of the question and
|
||||
it is unresearched, not settled.
|
||||
- **Nothing survived on named exemplar organisations.**
|
||||
- **Nothing survived on English-only language policy.** One incidental
|
||||
datapoint: Prana et al. excluded 48 of 1,193 inspected repos as non-English
|
||||
(~4%).
|
||||
- **Nothing survived on prose/markdown linting tooling.** Vale, markdownlint,
|
||||
textlint and write-good have no evidenced effect on perceived quality. Only
|
||||
the three `textstat` metrics are evidenced, and only as instruments.
|
||||
- **Nothing survived on AI-generated-prose tells** beyond badge overload —
|
||||
arguably the most consequential 2026 anti-pattern, and entirely unevidenced.
|
||||
|
||||
## 4. Platform ground truth — verified live, not inferred
|
||||
|
||||
Measured against `git.fromaitochitta.com`, **Forgejo `15.0.6+gitea-1.22.0`**:
|
||||
|
||||
- **Mermaid renders in `README.md`. Confirmed on the real instance.** The
|
||||
published `voyage` README yields two `div.mermaid-block` containers, each
|
||||
holding an `iframe.markup-render` whose `srcdoc` contains real SVG (26,634 and
|
||||
24,713 bytes), laid out at 191 px and 1,459 px. The server marks the fences
|
||||
`class="chroma language-mermaid display"`; Forgejo's own frontend then
|
||||
replaces them. Diagrams therefore need no committed image — and no
|
||||
user-authored `iframe`, since Forgejo injects its own.
|
||||
- **GitHub's `#gh-dark-mode-only` / `#gh-light-mode-only` fragments will NOT
|
||||
work here.** That support landed upstream in **Gitea 1.26.0** (PR #36922,
|
||||
merged 2026-03-21, released 2026-04-18). This instance is built on
|
||||
**gitea-1.22.0** — four minor versions below. Both variants would stack.
|
||||
Any dark-mode image strategy is unavailable today.
|
||||
- **A rendered caution:** the second `voyage` diagram occupies 1,459 px of
|
||||
vertical space and wraps node text mid-word. Mermaid rendering *working* and
|
||||
Mermaid rendering *well* are separate questions.
|
||||
- `MERMAID_MAX_SOURCE_CHARACTERS` is a per-instance limit; both current diagrams
|
||||
are under it. A longer diagram degrades silently to a plain code block.
|
||||
|
||||
## 5. What this permits, and what it forbids
|
||||
|
||||
Cross-referencing the evidence against this plugin's own invariants
|
||||
(`No hook until the rule is precise`; `When a check fires many times in one
|
||||
repo, suspect the CHECK`; `a gate that fails every repo teaches people to switch
|
||||
the gate off`; `the gate sees ONE repo`):
|
||||
|
||||
**Defensible as engine checks** — precise, measurable, low false-positive risk:
|
||||
|
||||
- Badge count above the measured inflection (as `WARN`, never `ERROR` — the
|
||||
source cannot carry a hard threshold).
|
||||
- Non-English prose on a public README. Machine-detectable, and the two hits
|
||||
found are unambiguous. Needs the operator's policy decision first: this is a
|
||||
coherence choice, not a research finding.
|
||||
- Section count outside 5–12, as information rather than a failure.
|
||||
|
||||
**Not defensible as engine checks:**
|
||||
|
||||
- Anything about images, diagrams, screenshots — no evidence base.
|
||||
- README length — no evidence base.
|
||||
- Tone, voice, "reads as AI-generated" — no evidence base, and irreducibly
|
||||
judgement.
|
||||
- Readability thresholds — the instruments exist, the *thresholds* do not, and
|
||||
a wrong threshold is precisely the false-positive class that gets gates
|
||||
switched off.
|
||||
|
||||
**Out of scope for this plugin by its own charter:** everything about
|
||||
recognisability *across* repos. The gate sees one repo. A shared skeleton, an
|
||||
org profile page, a consistent description register, a catalog hub — these are
|
||||
org-level questions, and the research found no evidence to guide them anyway.
|
||||
|
||||
## 6. Open decisions
|
||||
|
||||
1. **Language policy.** Is English-only the standard for public `open/` repos?
|
||||
Research offers nothing; this is the operator's call. Two repos fail it today
|
||||
if the answer is yes.
|
||||
2. **Where does cross-repo coherence live?** Not here. Deciding *where* is not
|
||||
the same as deciding *who builds it* — that requires a session in the owning
|
||||
repo.
|
||||
3. **Is a visual standard wanted at all**, given that it can only ever be taste?
|
||||
Mermaid is confirmed available; that is capability, not justification.
|
||||
|
||||
## Verification log
|
||||
|
||||
| Claim | Source |
|
||||
| --- | --- |
|
||||
| Median 7 sections, IQR 5–12 | Prana et al., EMSE 2019, doi 10.1007/s10664-018-9660-3 |
|
||||
| Why 25.7% / When 21.4%; authors' differentiation argument | ibid., Table 3 + §7 Implications |
|
||||
| No evidence-based length target; first-version stubs | Gaughan et al., CHASE 2025, arXiv:2502.18440 |
|
||||
| Badge inflection at 5; assessment vs conventional signals | Trockman et al., ICSE 2018, doi 10.1145/3180155.3180209 |
|
||||
| Flesch / EFLAW / Linsear via `textstat` | Gaughan et al., CHASE 2025 |
|
||||
| `#gh-dark-mode-only` landed in Gitea 1.26.0 | go-gitea/gitea PR #36922, issue #35545, release history |
|
||||
| Mermaid renders in this instance's README | Live DOM inspection, `git.fromaitochitta.com/open/voyage` |
|
||||
| Instance version `15.0.6+gitea-1.22.0` | `GET /api/v1/version` |
|
||||
| All §1 org measurements | Direct measurement of 18 cloned repos |
|
||||
|
||||
**Not verified:** whether Forgejo carries Gitea PR #36922 in any later release
|
||||
(only that this instance predates it); whether the markdown sanitizer strips
|
||||
user-authored `iframe` (two claims failed verification, so embedded terminal
|
||||
recordings remain unestablished); any claim about exemplar organisations,
|
||||
linting tools, or English-only policy.
|
||||
Loading…
Add table
Add a link
Reference in a new issue