The Obsidian plugin I’m building takes new source documents, has an AI read them, and turns them into wiki notes
(/ingest). A separate command (/refactor) then scans the whole wiki and proposes merging notes that overlap or
splitting notes that have grown too large. A merge proposal isn’t executed right away: it first goes through one
more LLM call, an audit that asks, “Should these two really be merged?”
I had already reworked this audit once, in an earlier post (Korean only). Shown only the two candidate notes, it was often wrong. When the vault contained a third note that overlapped with both candidates, putting that note into the audit prompt as evidence fixed it: 32 out of 32 checks came out right.
A few weeks passed. I wiped the wiki, ran /ingest again from scratch, and was auditing every result of the
/refactor scan that followed when I found a case the audit had missed.
One merge proposal looked wrong — a parent concept and its subtype
One proposal in the new scan wanted to merge two notes, “anxious leader” and “anxious-type leader,” into one. Its stated reason (translated from Korean) said both documents covered the anxious-type leader, had the same classification and tags, and that the same leadership pattern was scattered under the names “anxious-type leader” and “anxious leader.”
I opened both notes. “Anxious leader” was a parent-concept note covering two subtypes, the anxious-type leader
and the avoidant-type leader. The body said so in as many words: anxious leaders broadly divide into anxious-type
and avoidant-type leaders. “Anxious-type leader” was a separate note covering only one of those subtypes. Their
## Related pages sections already linked parent to child and child to parent: a well-structured hierarchy. Merging
them would have deleted the parent-concept note, and with it the context that contrasted the two subtypes.
It went 32 for 32 before, so why did it miss this? — embedding-similarity gating
This was the same merge audit, the one that went 32 for 32 once the third note was supplied as evidence. I opened the code to check. The cause wasn’t its judgment. The audit never ran at all.
The audit was built as “find a third note by embedding similarity; if none is found, approve right away.” Every case I had tested back then was about whether the candidates’ sub-items already existed elsewhere in the vault as separate notes, and in those cases the third-note evidence really exists. “Anxious leader” and “anxious-type leader” are different. What overlaps isn’t a third note but the two candidates themselves. One being the other’s parent concept is a signal that the question “does it overlap with a third note?” can never catch. So the evidence search came back empty, and the audit quietly approved the merge without calling the LLM.
The right rule already existed elsewhere in the codebase
Digging further, I found that the exact rule, “containment is not identity,” was already in the codebase. It lived in the prompt of a separate verification step that finds duplicate candidates whose titles are too different for clustering to group them. That prompt said explicitly that when one is the other’s parent concept, subcomponent, or specific case, they are different concepts. But the rule was used only in that candidate-detection step. It had never been wired into the gate that checks the merge proposals actually generated.
Removing the gating entirely — always run the LLM audit
My first idea was to add an exception: when there’s no evidence, also check for containment. But that would split the decision into two branches and leave room for another hole like this one later. Instead I removed the evidence gating itself, so every merge candidate always goes through the audit LLM. If there’s evidence, it’s added to the prompt; if not, the audit judges with just the two documents. I also added “containment is not identity” to the prompt as a new criterion.
The trade-off was clear: one more LLM call per merge candidate, even without evidence. It gives back the savings I had designed in earlier, when the point was not to call the LLM at all when it wasn’t needed. But this misjudgment would have broken a real hierarchy, so I accepted the cost.
What stays with me
The conclusion of that earlier post was that a judgment that failed even with the full text of both notes became 32 out of 32 once one missing piece of evidence was supplied. This story doesn’t overturn that conclusion. It narrows the scope of the problem it covered. On the failure type the audit was aimed at, overlap with a third note, it was still accurate. The problem was that it had never been designed to target a different failure type: containment between the candidates themselves. “Does this audit work?” and “Is the problem it was built to catch the whole problem that actually happens?” are different questions, and the 32 successes only answered the first.