Tag: Empirical
All the articles with the tag "Empirical".
-
Asking an LLM to name the merged note shook its verdicts, and a few lines of name comparison did the job
An Obsidian plugin named merged notes after a subtopic. Asking the LLM for a title shook its verdicts or added calls; a name-containment rule fixed it.
-
I treated an LLM's note merge suggestion as a bug and fixed it three times, but the suggestion was right
I took an Obsidian plugin's LLM note merge suggestion for a bug and rewrote the note-creation prompt three times. The suggestion was right; my audit was wrong.
-
I cut LLM note ingest time by 44%, and proved the results were the same by replaying responses
An Obsidian plugin's ingest went from 554 to 313 seconds once LLM dedup stopped waiting in line. Replaying saved LLM responses proved the notes were unchanged.
-
I had an LLM fix the notes its judge flagged, and a loss the judge couldn't see got through
An Obsidian plugin had an LLM fix dedup edits its judge flagged, not roll them back. Over 96 runs it mostly restored the original, and an unseen loss passed.
-
An LLM judge blocked every broken note, but half of the dedup got rolled back
An LLM judge rechecked notes edited by LLM dedup and rolled back flagged ones. No broken note survived five test runs, but 13 of 28 rewrites were undone.
-
Judging broken sentences with Decisions API and Jev gave four times the false positives
I had OpenAI Decisions API and Jev judge LLM-edited notes for broken sentences. They flagged ~4x more normal notes than gpt-6-luna; split questions backfired.
-
When an LLM removed duplicates from my notes, the remaining sentences lost what they pointed to
Removing note duplicates with an LLM left 'this technology stack' and 'also' pointing at nothing. I stopped adding code rules and had an LLM judge each edit.
-
My LLM note-merge test passed, but the fixed code never ran once
I fixed an LLM note-merge bug in my Obsidian plugin and reran it on real notes. It passed, but the fixed code never ran until I restored the pre-merge state.
-
I deleted the wrong example from my LLM note-merge rule, but the rule stayed
My Obsidian plugin's LLM rule for merging duplicate notes blocked even same-product notes. I traced it, tested real merges, and adopted Karpathy's LLM Wiki.
-
In LLM summaries, a claim made once disappeared and one made twice survived
Chasing an LLM summary that dropped a source's causal claim, I ruled out grammar and counterintuitiveness with controls. A claim stated once vanished.