Series: Putting My Confidence Up Against Measurement
After the redesign, a record of putting plausible-looking confidence up against actual measurement: the trap of few-shot counterexamples, information-density loss, wikilink false positives, a RAG claim made without evidence, a UTF-8 parsing bug, tag and link experiments, reasoning effort and isolation conditions, shoring up judgment with embedding search, filling in a missing prompt rule only to get the same results, and using control groups to isolate when a counterintuitive claim mentioned only once disappears from a summary.
-
In LLM summaries, a claim made once disappeared and one made twice survived
Chasing an LLM summary that dropped a source's causal claim, I ruled out grammar and counterintuitiveness with controls. A claim stated once vanished.