Series
Multi-part series, in reading order.
- [Complete] Building an LLM Wiki in Obsidian (1 of 6 in English)
A record of building Andrej Karpathy's 'LLM Wiki' idea as an Obsidian plugin myself: why I built another one when ten plugins already existed, a date bug, UX details like dynamic badges and a dirty check, how it evolved into an autonomous ingest engine, and finding where button text was doing the job of a state value while moving the UI to English.
- Fixing LLM Note Cleanup by Measurement (4 of 4 in English)
A record of fixing the dedup of LLM-generated notes in an Obsidian plugin and measuring the fixed results again on real notes: an information loss the judge cannot see passing through, cutting the ingest time spent waiting on dedup and proving the results stayed the same, treating a correct merge suggestion as a bug and trying to fix it, switching from asking an LLM to name the merged note to a rule that compares the note names, and more.
- [Complete] What AI Verification Misses (10 of 10 in English)
Putting the checks that verify AI-generated output under measurement to see what they actually miss: a merge audit that mistook a parent concept and its subtype for the same thing, a subagent-driven full audit that skimmed list-heavy files instead of checking item by item, re-verifying only against other cases while never rechecking the original false positive, an ingest step where sibling notes copied paragraphs verbatim and nothing caught it at first, and even the embedding filter built to catch that duplication flipping its verdicts depending on model size (small/large), once that filter was removed, duplication turning up in groups that had been recorded as clean, a merge rule whose wrong example was deleted while the rule that produced it stayed, a real-note test that passed cleanly while the fixed code never ran once, and sentences left pointing at nothing after duplicates were removed, blocked shape by shape in code until I switched to having an LLM reread each edited note and judge it, and that judge then blocking every broken note while rolling back nearly half of the duplicate cleanup.
- Could a Decision-Only Model Take Over? (4 of 4 in English)
A measured record of whether, instead of calling a full LLM every time an Obsidian plugin asks for a judgment such as whether to merge two notes or whether an edited note now has a broken sentence, that judgment could be handed to models built only for decisions (Laya, TypeSafe Jev, OpenAI Decisions API). Beyond accuracy, it follows the confidence attached to wrong answers, the pull from the order of the options, and how the results change depending on how the question is split up.
- Putting My Confidence Up Against Measurement (2 of 11 in English)
After the redesign, a record of putting plausible-looking confidence up against actual measurement: the trap of few-shot counterexamples, information-density loss, wikilink false positives, a RAG claim made without evidence, a UTF-8 parsing bug, tag and link experiments, reasoning effort and isolation conditions, shoring up judgment with embedding search, filling in a missing prompt rule only to get the same results, and using control groups to isolate when a counterintuitive claim mentioned only once disappears from a summary.
- Before Fixing It, I Had to Recheck the Instruction (3 of 8 in English)
— I was designing a guard against LLM note over-splitting and didn't see the one that already existed
After the redesign, a record of times when the "fix it" instruction or the document itself was wrong: fixing a dead comment and getting it wrong again, a hypothesis that was only half right, a mistake avoided by asking once more before fixing, "delete it" and "how to delete it" turning out to be different questions, fixing a prompt rule twice when the place that needed fixing was the stage that splits notes, carefully built detection code that, once measured, turned out to have done nothing, and designing a safeguard against over-splitting without recognizing the same safeguard built the day before.
- [Complete] I Almost Mistook Two Variables for One (7 of 7 in English)
Same sources, same plugin, but switching machines made the results differ 3x. Tracking down the cause, what I thought was a single culprit turned out to be two different settings, and separating them took rerunning the experiment eight times; along the way I found and fixed three completely unrelated bugs.
- Reviewing MCP Servers and Skills (1 of 9 in English)
A record of reviewing, one by one, tools worth plugging into or swapping in for Claude Code or Hermes. Beyond MCP servers (such as codebase knowledge graphs), it covers workflow skills (Superpowers), token-saving tools (Headroom, OmniRoute), a memory plugin (claude-mem), a browser automation tool (agent-browser), and an open-source coding agent that can take Claude Code's place (OpenCode), all judged by the same question: is it actually worth adding to, or switching from, the setup I use now?