The Obsidian plugin I’m building takes in an article and has an LLM split the concepts, people, and companies in it into separate notes. In this post, I call notes that came out of the same article together “sibling notes.” As shown in Part 4, sibling notes often copied the same paragraph from each other verbatim, so I added a dedup step: right after the notes are created, it gathers all the sibling notes, finds the overlapping facts, keeps each one in a single note, and replaces that part of the other notes with a link sentence like “covered in detail in [[that note]].” This post is a record of chasing a problem where, after that dedup step deleted sentences, the sentences left behind broke.
I stopped fact loss, and sentences started breaking — a side effect of LLM dedup
The first problem was fact loss. When the dedup step found one overlapping fact, it replaced the entire paragraph containing that fact with a link sentence, so other content in the same paragraph disappeared too. So I changed the prompt to replace only the overlapping sentences and leave the rest alone. Rerunning with the same inputs, the runs that lost facts went from 15 out of 19 to 0 out of 20.
But the next real test turned up a different problem. All the facts were still there, but the sentences were off.
- “The X-ble Shoulder is introduced as a product developed on top of this technology stack.” — the earlier sentence that explained “this technology stack” had been deleted.
- “Related content is organized in the [[Sim-to-Real]] note.” — what it was related to had gone missing.
- Fragments with nothing but a link left dangling at the end of a sentence.
When I made it delete less, the remaining sentences that leaned on the deleted ones lost something to lean on.
It broke in a different shape every time I blocked one — demonstratives and connectives
From here on came a run of blocking one shape at a time.
First, words that point back to earlier content, like “this ~” or “that ~.” I added a rule to the prompt: “if deleting a sentence leaves a pointing word in another sentence without its referent, replace that word with the referent’s name.” With the same input, breakage went from 5 out of 5 runs to 1 out of 5. I blocked the remaining one in code: a safeguard that skips the rewrite if the sentence right after the deleted paragraph starts with a demonstrative (이, 그, 해당 in Korean, roughly “this,” “that,” “the said”).
The next real test produced sentences whose meaning had changed. “Missions like somersaults or handstands” became “missions like somersaults,” and “stopping, collision, cost, and safety problems” became “cost problems.” I had told it to delete only the overlapping items, so it deleted some items from a list and made a different claim with the leftover pieces. Comparing on the same inputs, the old prompt that rewrote whole paragraphs did this in 0 out of 9 runs, and the prompt that had it replace only phrases did it in 8 out of 15. So I changed the unit to the sentence. A sentence containing an overlapping fact is replaced whole with a link sentence, and a sentence mixed with other content is left untouched. Runs with changed meaning dropped to 1 out of 16, and I blocked that one too with a code safeguard that “skips the rewrite if it is a shortened version of the original sentence.”
In the test after that, it broke within a single paragraph. It deleted a definition sentence and left the “This definition and …” right after it. The earlier safeguard only looked at the sentence following the paragraph, so it missed this. I widened the safeguard to cover the inside of the paragraph.
And the next test brought yet another new shape. When the first sentence of a section was deleted, the remaining sentence started with “Also, it runs on battery ….” “Also” is a connective, not a pointing word, so it wasn’t covered by the safeguard, and the remaining sentence was the original sentence unchanged, so the “shortened sentence” safeguard didn’t catch it either. The section heading was “The Practical Gap Between Training and Execution,” but what was left in the section was about on-device AI. When I fed that run’s 8 saved rewrites back through both safeguards, not one of them was caught.
Breakage showed up in all four real tests. The count went down, to one each in the last two, but the shape was different every time. The safeguards kept multiplying, and the next shape was always outside them.
One cause, but the cure was too strong — paragraph-level dedup
The shapes differed, but the cause was the same. When you delete only some sentences in a paragraph, any remaining sentence that leaned on a deleted one is left on its own. It doesn’t matter whether it leans by a pointing word, a connective, or a section heading.
Then the fix is to not split paragraphs. I changed the prompt so that a whole paragraph is replaced with a single link sentence only when every sentence in it already exists in the receiving note, and the paragraph is left alone if even one doesn’t, and ran it 27 times.
Breakage went to 0. But so did dedup, nearly. On the same input, rewrites that had been 1 to 8 per run became 0 to 1, and the four real duplicates that had been cleaned up every time until then were not cleaned up even once in 8 runs. It was rare for a whole paragraph of a sibling note to exist in another note. Since almost nothing overlapped at the paragraph level, this rule was the same as turning the dedup step off.
Instead of blocking shapes in advance, I had it reread the result — LLM-as-a-judge
So I changed direction. Instead of predicting which shapes would break before deleting and blocking them, I showed the note after deletion back to an LLM and had it judge “did this edit create a new problem?” The criteria were the four that had come up so far: did a pointing word lose its referent, did a sentence with no subject or with nothing but a link appear, did the meaning change because only part of a list remained or a condition dropped out, and did the section heading stop matching what was left. The plan was that if a problem is found, that note gets rolled back to how it was before the edit.
I rebuilt the 38 rewrites from the last four real tests into before/after note pairs, read them myself and sorted them into 9 broken, 6 ambiguous, and 23 normal, and then ran the judge 3 times each. The first judge wording caught 20 of 27 broken cases. It missed sentences with nothing but a link and sentences where only part of a list remained, so once I spelled those two cases out in the criteria, it caught 27 of 27 and wrongly flagged 2 of 69 normal ones.
I couldn’t trust these numbers. I had tuned the judge wording while looking at the same 38 cases, and the example I put in resembled one of them.
I checked again with 124 cases not used in tuning — prompt overfitting
I built the other 124 rewrites that had accumulated from earlier repro experiments the same way, ran the judge once on each, and read all 124 myself.
Of the 20 the judge flagged, 15 were actually broken: the “this technology stack” seen earlier, an “also” that had been receiving the deleted content, “the shortlist” whose referent had vanished, a case where sentences disappeared without a link and only the section heading remained, and a case where “rivals or regulators” was left as just “regulators.” 3 were ambiguous cases where the meaning had narrowed slightly, and 1 or 2 were wrongly flagged. Among the 104 it didn’t flag, it missed 1 clear case and 3 minor ones. It isn’t perfect, but it even caught shapes like “a section’s content disappearing entirely,” which the code safeguards never look at because there’s no pointing word and no shortened sentence. That’s because it doesn’t need each shape defined one by one.
While reading, I learned one more thing. The dedup step sometimes edits a note twice. The first rewrite orphans “this technology stack,” and the second rewrite fixes that sentence to “Robotics Lab’s technology stack.” Judging each rewrite on its own flags this intermediate state as broken. So I decided to judge only once, on the final state of the note after all its rewrites are applied. Regrouped by note into 134 notes, the judge caught 18 of 20 broken ones and flagged 3 of 103 normal ones. It takes about 3 seconds per note.
It isn’t in the plugin yet. How to roll back notes the judge flags, and whether to keep the existing code safeguards, will be decided after I write the design doc first.
Closing thoughts
Defining and blocking each shape of breakage as it appeared did reliably fix one case every time. So it looked like progress. But the fact that four real tests each ended with a different shape of breakage meant that what I was blocking wasn’t the cause, but a list of shapes the cause produced. The cure that removed the cause outright removed the feature along with it. The remaining path was to reread the result like a person and judge it. And because the judge wording couldn’t be trusted on its tuning data alone, I had to read data it had never seen one more time.