Series: Fixing LLM Note Cleanup by Measurement
A record of fixing the dedup of LLM-generated notes in an Obsidian plugin and measuring the fixed results again on real notes: an information loss the judge cannot see passing through, cutting the ingest time spent waiting on dedup and proving the results stayed the same, treating a correct merge suggestion as a bug and trying to fix it, switching from asking an LLM to name the merged note to a rule that compares the note names, and more.
-
I had an LLM fix the notes its judge flagged, and a loss the judge couldn't see got through
An Obsidian plugin had an LLM fix dedup edits its judge flagged, not roll them back. Over 96 runs it mostly restored the original, and an unseen loss passed.
-
I cut LLM note ingest time by 44%, and proved the results were the same by replaying responses
An Obsidian plugin's ingest went from 554 to 313 seconds once LLM dedup stopped waiting in line. Replaying saved LLM responses proved the notes were unchanged.
-
I treated an LLM's note merge suggestion as a bug and fixed it three times, but the suggestion was right
I took an Obsidian plugin's LLM note merge suggestion for a bug and rewrote the note-creation prompt three times. The suggestion was right; my audit was wrong.
-
Asking an LLM to name the merged note shook its verdicts, and a few lines of name comparison did the job
An Obsidian plugin named merged notes after a subtopic. Asking the LLM for a title shook its verdicts or added calls; a name-containment rule fixed it.