The Obsidian plugin I’m building has an LLM read source material and organize it into notes, one per concept,
and its /refactor command merges notes that cover the same subject into one. When this merge feature combined
two notes, there was a bug where content found in only one of the sources would sometimes vanish entirely. The
first case found was simple: after merging two notes, a named tool mentioned on only one side was nowhere in the
result. Collecting a few more reproductions, the same pattern showed up: SpaceX’s segment-level financial figures,
a list of use cases that disappeared when three articles were merged, alternative figures in a note about Korea’s
National Pension. The merge decision itself (“these two are the same subject”) was right every time, but during
the actual combining, content unique to one side was leaking out.
Laying the prompts side by side: only the information-density rule was missing
To find the cause, I opened the merge execution prompt side by side with the other prompts in the same pipeline (ingest and split). All three used common rules like “match the language of the source” and “don’t make up anything that isn’t in the source,” but exactly one rule, the information-density rule (“even figures, statistics, and named examples mentioned in just a sentence or two of the source must be kept, not dropped”), was in the ingest and split prompts and not in the merge prompt. That rule had originally been written separately to catch a similar omission bug in the ingest step. The merge prompt had only the loose phrase “so that no information from each document is lost,” and the specific rule had never been carried over.
It looked like a plausible cause. There was precedent for fixing other prompts for the same ailment twice, and the merge prompt alone had never gotten the treatment.
I decided to verify it on real notes (prompt A/B test)
Instead of fixing it straight from reading the prompt, I decided to actually compare a version with the rule added to the merge prompt against the original. Instead of made-up examples, I dug through the commit history of the source repository that had actually been used for past bug reproductions and found two sets of real notes in their pre-merge state.
The first was the output of splitting one document into three pieces (business structure, IPO, capital raising): notes that don’t overlap, each holding its own figures. The second was notes derived from two different source articles on psychological safety: a far more overlapping structure, explaining the same concept with different research and examples. Both sets were real, unmanipulated content of the kind that would actually get merged.
Eight runs, no difference
I ran each set through merge execution four times with the original prompt without the rule (OLD) and the prompt with the rule added (NEW), eight runs in total. The model was gpt-5.6-luna, the same as the plugin, with reasoning effort set to medium. To score the results, I prepared in advance a list of “specific information that must survive” for each set: numbers like revenue share and operating margin, researchers’ names and publication years, figures for specific events.
The SpaceX set (out of 26 items) scored 26, 26, 25, 25 with OLD and 26, 25, 25, 25 with NEW. The psychological safety set (out of 15 items) scored 15, 15, 14, 15 with OLD and 15, 14, 15, 15 with NEW. They were so alike you couldn’t even tell which was better. The occasional missing item was a difference in section-title wording, not an actual figure or proper noun.
Plausible and proven were different things
What looked like an obvious defect when comparing prompt texts made no difference at all in actual runs. There are two ways to read this: either this model already preserves density well at that scale (2 to 3 notes, under 200 lines) with or without the rule, or the severe cases originally reproduced (a merge entangling four notes, a separate in-depth report that wasn’t even in the merge’s source list) were never about prompt wording in the first place, but about much larger scale or a different step that picks the wrong merge targets.
I reverted the rule I had added. I decided not to leave something in the code whose effect was unproven. Instead, the decision went like this: keep checking every merge run against the sources, but only attempt the next code fix after pinning down the exact reproduction conditions again (which notes, which settings, what went missing).
A new case came up that night, and I ran nine more
Past midnight on the night I made that decision, a new case came up in real notes. Two notes about a tool called
Graphify, made from two different articles, were merged, and the uv tool install method that one article
presented as the “recommended” installation disappeared entirely, leaving only the other side’s pip install
method. The same-subject judgment was right, and what went missing was information unique to one side. It was
exactly the shape of this bug.
This time I reproduced it closer to the real situation. I made a fresh note from each of the same two articles and
merged the pair with OLD and with NEW. Only after five runs at medium did I realize the plugin setting I actually
use is high, not medium. The earlier eight runs had also used a different setting from the real one. I ran four
more at high. In all nine, both OLD and NEW kept the uv tool install recommendation intact. Even the original
prompt without the rule never once reproduced the omission that had actually happened.
A third set of material, the same result. Two notes pulled out on their own couldn’t seem to produce the conditions under which this omission happens. I reverted the rule again, and decided that next time I’d dig into the cause right in the run where the omission actually happened, before cleaning up the notes.
Final thoughts
The moment you look at a code diff and think “that’s it,” and the moment you confirm that diff actually changes the outcome, were two different things. Both feel like logical reasoning, but the first only means the explanation is smooth; only the second is evidence. Change the code first to fix a bug you can’t reproduce, and all you’re left with is the conviction that you fixed it, while nobody knows what actually changed. For now, I chose to wait for a reproduction instead of that conviction.