The Obsidian plugin I’m building takes articles and has an LLM turn each one into notes on its concepts, people and companies. Right after the notes are created, an automatic cleanup step runs: it finds similar pairs of notes, and an LLM compares their full text and shows a card suggesting “merge these.” The merge itself only happens when a person presses the card’s run button. This post is about the half day I spent with an AI coding agent treating those merge suggestions as a bug. To give away the ending: the suggestions were right, and what was wrong was the audit that blocked them.
The step that split the notes and the step that wanted them merged said opposite things (LLM note merge suggestions)
During a test run, the automatic cleanup suggested two merges. One paired a general-definition note from an article about agent harnesses with a note split off from the same article for its example, “the process of fixing one coding bug.” The other covered three notes from an article about reward hacking: the parent-concept note, the reinforcement learning one and the LLM agent one.
The agent auditing the run recorded that it hadn’t executed either, calling them “notes with different scopes.” When I later asked to look at these suggestions again, the agent framed the question like this: why does the very next step want to merge what the note-creation step deliberately split? Going back through two weeks of records, there had been five suggestions to merge sibling notes from the same article, and two of them had actually been merged at the time.
I changed the prompt three times, and each time the other side broke (LLM prompt testing)
The cause the agent found sounded plausible. The automatic cleanup that decides whether to merge notes has a rule that notes describing the same subject in a different context are the same concept (a standard set in an earlier post), but the splitting rules in the note-creation step had no such definition. Two steps using different standards would reach opposite conclusions.
So I changed the note-creation prompt and measured it. For one article at a time, the plugin’s real code built the LLM request, and I sent each article three times with the prompt before and after the change. Besides the 5 articles that had caused the problem, I also ran articles whose notes should be split, as a regression check.
| Round | What changed | Times the problem sibling notes appeared | Side effect |
|---|---|---|---|
| Current prompt | — | 6 of 15 | — |
| 1st | Added the same-concept definition to the note-creation rules | 3 of 15 | What remained took a different shape |
| 2nd | + relaxed the rule “make a separate note that represents the whole source” | 0 of 15 | Notes got coarser (4 → 1-2, 5-6 → 2-3) |
| 3rd | + “split it if another article would come along and update it” | 4 of 15 | More notes only a sentence or two long |
The rule I relaxed in the 2nd round had originally been added to stop the anecdote at the top of an article from landing in no note and disappearing. That rule made a separate note representing the whole source, so when an article was really about one subject, that note overlapped with the subject note. Relaxing it made the problem shape go away, but this time even named subtypes like “anxious leader” and “avoidant leader” got buried as sections of the parent note. Karpathy’s LLM Wiki, which this plugin follows, says that when a new source arrives the LLM “integrates it into the existing wiki — updating entity pages,” and that “A single source might touch 10-15 wiki pages.” A wiki only builds up if there are concept pages for the next article to update. So in the 3rd round I added that question, and this time the whole splitting side pushed back. Fixing one side pushed the other over the threshold.
“Feels like you keep fiddling around the edges instead of doing an RCA” (root cause analysis)
After the third result, the agent proposed excluding notes from the same article from merge candidates in code. After hearing what that proposal would not fix, I told the agent that. I meant that I had asked for a root cause analysis (RCA), and it was only fixing things around the symptom. The agent came back with a deeper cause. The note-creation step never sees the existing wiki notes, because in July the path that updated existing notes was removed and merging was handed entirely to the cleanup step. So, it explained, two steps were drawing note boundaries separately.
In the code, that was true. But when I asked it to check again, the agent reversed itself. All three measurements had been run with an empty vault and a single article. There wasn’t a single existing note, and the splitting within one article still happened. Showing it the existing notes wouldn’t change how one article gets cut. That cause fits duplication between different articles, not this problem.
Looking at the design again, it wasn’t a bug (note merge criteria)
The agent went back to describing the cause as the difference in rules and suggested running more measurements. Hearing that, I described the design as I remembered it. The note-creation step only splits, the automatic cleanup that follows suggests merges, and a person confirms and runs them. I built it that way on purpose in July. In that case, “the cleanup step wants to merge what the creation step split” is not a contradiction but the designed path.
There was one question I should have asked: was the suggestion right? The rule contains this sentence (translated from the plugin’s Korean prompt): “A note that only explains a concept as applied to a particular situation is not a subconcept of that concept.” By this rule, the general definition and its applied example are the same concept, and so are the reinforcement learning and LLM agent versions of reward hacking. Both suggestions were right. “Different scope” wasn’t the rule; it was the auditing agent’s hunch, and I hadn’t checked that verdict against the rule either.
What actually got fixed
I didn’t change the note-creation prompt at all. The three rounds of measurement (123 LLM calls, about $0.48) were shelved with only the record and the conclusion kept. Instead:
- I corrected the two verdicts in the audit records to “the suggestion was right.” From now on, when auditing merge suggestions, I check them against the rule’s wording instead of judging scope by feel.
- I narrowed the real remaining problems down to two. The note that merged the three reward hacking notes took the name “Reward Hacking in LLM Agents” instead of the parent-concept note’s name, and a merge suggestion that isn’t executed shows up again, unchanged, on every scan. For the first, the code turned out to use the name of the first note in the list of notes to merge, and that list had no fixed order. That one was fixed in the next part. I confirmed the cause of the second later. The verdict that two notes are the same concept stays in a cache and is reused on the next scan, but the fact that the user passed over the suggestion without running it is recorded nowhere. So as long as the two notes stay the same, every scan produces the same suggestion again. This was a known, accepted behavior from the original design, and I haven’t decided yet whether to add a way to hide suggestions the user chose not to merge.
What stays with me
The measurement itself was thorough. It built requests with the real code, set the criteria in advance, and ran regression articles alongside. Still, if the question is wrong, careful measurement just measures the wrong place precisely. What was missing this time was the very first step: does this behavior go against the design? Had I asked that first, it would have ended before the first measurement. Checking each candidate cause against the conditions of the evidence already collected (this time, “an empty vault”) is the same lesson.