The Obsidian plugin I’m building takes articles and has an LLM write notes for each concept, person and company in them. After that, an automatic cleanup step looks for groups of notes covering the same concept and suggests merging them. In the previous post I concluded that a suggestion to merge three reward hacking notes, a parent concept note plus a reinforcement learning one and an LLM agents one, was the right call. What was left was the merged note’s name. It should have been the parent concept, “Reward Hacking”, but it came out as “Reward Hacking in LLM Agents”. The code turned out to use the name of the first note in the group as-is, and that order is whatever order Obsidian returns its file list in, which isn’t guaranteed.
Asking the LLM for the title too shook its verdicts — adding a question to the prompt
The first idea was to ask the LLM. A call that judges whether two notes are the same concept already existed, so I had it also answer “if these were merged, whose title should be used?” Running it twice over 172 real note pairs, the title choice mostly went to the note covering the whole concept. But the verdict itself flipped on both runs for 2 pairs. Separate things with their own names, “Hannibal” and “Battle of Cannae”, AlpaSim and AlpaGym, were judged to be the same concept, and the reason sentences cited the newly added title question. Adding one question had shaken the answer to the original question.
Asking for the title separately worked, but added calls — LLM API calls
So I moved the title question into its own call. Asking twice for each of the 88 pairs judged to be the same concept, 73 pairs got the same title both times, and both the agent harness pair (“Agent Harness” and its applied-example note) and the reward hacking pair picked the parent note. It worked, but this approach adds one LLM call for every pair to merge. A design where calls grow every time a new problem appears didn’t seem like a good one.
On top of that, the reward hacking group was shaped like two pairs chained together: “LLM agents–reinforcement learning” and “reinforcement learning–parent”. The parent note and the LLM agents note weren’t judged to be the same concept, so they had no pair. Counting the titles picked per pair gave a 1–1 tie, and a tie falls back to the first note’s name again.
The parent concept’s name is inside the subtopic’s name — a rule with no LLM call
The rule I finally chose doesn’t call the LLM. A parent concept’s name usually sits verbatim inside the subtopic notes’ names. “Reward Hacking” is inside “Reward Hacking in LLM Agents”, and “Agent Harness” is inside “Coding Agent Harness Example”. So within a group, it picks the name contained in the most other note names (ignoring case and symbols). If no such name exists, or there’s a tie, it uses the first note’s name as before. I kept the existing rule that a group mixing Korean and English names picks only among the Korean names.
While I was there I fixed a neighboring problem too. Merge results were always written to this week’s folder. So if the target name matched one of the notes being merged and that note lived in last week’s folder, the plugin treated the original as a different file with the same name and appended “_dup2” to the result. Both originals got deleted and only “Reward Hacking_dup2” was left. Even with the right name, this problem would remain, so I fixed it as well. Now, when the target name matches an original, the result is written in that original’s place. I also changed the merge suggestion prompt to not invent a new name and to use the name of the note that covers the whole concept, and if a name outside the originals still comes back, the rule above picks one of the originals.
Checking on real notes — every group whose name changed moved to the parent concept
First I replayed past test records. The test vault is committed to git after every run, and I took out 43 commits holding the state right after a real cleanup, one at a time, and rebuilt the merge groups with the plugin code. No LLM calls, only the stored verdicts. Of 17 groups, 4 changed names, and all 4 moved to the parent concept. The reward hacking group became “Reward Hacking”, and three groups about leadership practices got short concept names like “Setting the Stage” instead of “Setting the Stage Leadership Practice”. The other 13 kept their names.
Then I ran the real app twice, with 6 articles each time. In the first run, automatic cleanup grouped the two harness notes, and the target was “Agent Harness”. In the file order the app returned, the example note came first, so under the old rule “Coding Agent Harness Example” would have survived. All 5 merges the LLM suggested folded an English note into a Korean note on the same concept, and the LLM used an original’s name as the target, so all five were merged in the original note’s place. For each merged note I also checked that nothing from the two notes before the merge had gone missing. That said, all the notes in these runs lived in this week’s folder, so the “_dup2” fix wasn’t confirmed in the real app. I checked that part by turning the case with an original in last week’s folder into a test.
There are cases this rule can’t catch. With “현대자동차 Alpamayo 자율주행 전략” and “현대자동차 자율주행 전략” (Hyundai Motor’s autonomous driving strategy, with and without “Alpamayo”), a word sits in the middle, so it isn’t containment, and the first note’s name comes out as before the fix. The same goes for names where neither contains the other, like “Fearless Organization” and “Fearless Workplace”, but pairs like that usually mean almost the same thing, so either name surviving was fine.
Closing thoughts
Asking the LLM did solve it. But one approach shook the original verdict, and the other added calls. The single observation that a parent concept’s name sits inside its subtopic notes’ names was enough, and that took a few lines of comparison. Since a judging call already existed, adding one more question looked free, but that question changed the answer to the original one. When adding a question to a call, measure first whether the answer to the original question stays the same, not the answer to the added one.