The Obsidian plugin I’m building has an LLM turn a single article into several notes, and then runs a dedup step: a fact that appears in more than one note is kept in just one of them, and in the other notes that sentence is replaced with a link sentence. This step broke sentences often enough that, in the previous post, I had another LLM reread each edited note and judge it, and roll the note back to its pre-edit state if there was a problem. No broken note survived, but 13 of the 28 rewrites in the judged notes were rolled back, so their duplication stayed. This post is the record of trying to save those 13, and finding a gap in the judge instead.
The first fix that came to mind had already been tried — adding prompt rules
The rolled-back cases came in three shapes: a deleted sentence took with it the subject or the referent of the next sentence; a section was left with nothing but a link sentence; or a newly added pointer sentence changed the meaning while summarizing the facts. So I proposed adding rules to the dedup prompt to block those three.
When I opened the prompt to write the design doc, those rules were already there. There was a rule saying not to write a sentence that contains only a wikilink, and a rule to fix referring words that lost their target because a sentence was deleted. They broke anyway, for reasons. The referring-word rule listed only the Korean words for “this,” “that,” and “the said,” plus “this” and “these,” but what actually broke was the Korean word for “also,” “however,” “the same question,” and omitted subjects. The link-only sections were cases where an original “See” sentence was all that remained, not a newly written sentence, so the rule didn’t catch them. And the pointer sentence that changed the meaning was, if anything, the result of the instruction to write each link mention as “a complete sentence that says what it covers,” which made the model restate the facts.
And this was a road I had already been down a few times. In the post before that, every time I blocked one shape, the next real test produced a new one. Adding three more items to the list would have been close to doing the same thing once more.
Fix it once before rolling back — LLM self-correction
So I picked an approach that doesn’t define the shapes in advance. When the judge finds a problem, the note isn’t rolled back right away. Instead, an LLM gets that note’s rewrites and the problems the judge found, and fixes them once. I limited the fix to two kinds of change: restoring sentences from the original paragraph as they were, and fixing only connectives, subjects, and referring words. No new facts are allowed, and if it can’t fix something, it puts the original paragraph back. The fixed result goes through the existing code checks again (rules that check things like whether the next sentence pointed to a deleted one), then gets the same judge. If it passes, it’s kept; if not, it’s rolled back as before.
Before measuring, I set the pass criteria. First, among results that passed the re-judge, there must be no breakage, no change in meaning, and no loss of information. The biggest risk was that the judge might go easy on fixed results, so I put this first. Second, at least half of the flagged notes must be kept after fixing, with duplication actually reduced.
The data was 32 notes the judge had flagged: the 8 rolled back in the previous post’s real tests, and 24 the judge flagged while I was validating the judge’s wording in the post before that. Each note got a fix and a re-judge three times, 96 runs in total.
In 69 of 96 runs there was no rewrite left to keep — the fix mostly restored the original
The results: kept after fixing 23 times, rolled back at re-judge 4 times, and no rewrite left at all 69 times. Counted by rewrite, the fix put 86 of 107 back to the original paragraph. The easiest way to remove a problem was to restore the deleted sentence, and that brings the duplication back too. In the end it was almost the same as the current rollback.
I read each of the 23 passing runs myself. Six of them still had problems.
- Information loss, 3 times. In one strategy note, the statement giving “limited planning time” as a reason traditional strategy frameworks were simplified was deleted all three times and passed. That content isn’t in any other note.
- Section heading and content no longer matching, 2 times. The fix removed only the “also” that had lost its antecedent and left the deletion of the previous sentence in place, so a section titled (in Korean) “the realistic gap between training and execution” was left with only an on-device AI story. That’s exactly the shape the code checks failed to catch in the post before that.
- Change in meaning, once. A rewrite passed that dropped the order — have AI generate a broad set of alternatives first, then have executives discuss the candidates — and kept only the part about discussing the candidates.
The heading mismatch and the change in meaning were each flagged by the re-judge in a different run on the same note. The judge gave different answers on the same note from run to run. Notes where duplication went down with no problem were only 2 of the 8 from real tests and 2 of the 24 validation notes. Both criteria failed.
The judge could never see loss in the first place — LLM-as-a-judge limits
Following the 3 loss cases led to something more important. The judge can’t see information loss. Its input is only the edited note before and after the edit, and not the note that took over the content. So the judge has no way of knowing whether a deleted fact really remains in another note.
This strategy note was one that had been rolled back in the previous post’s real tests. What the judge flagged then wasn’t the loss but an “It” that had lost its referent because the sentence before it was deleted. The loss disappeared only because the whole note was rolled back over the “It.” All three times, the fix changed the “It” the judge pointed out to “The article,” but left the other rewrite that caused the loss as it was. The judge was satisfied with that. No one had been watching the loss from the start.
That means the current approach has the same gap. A rewrite whose sentences are fine but which loses information passes the judge and is kept. That said, when I read the 14 kept notes from the previous post’s real tests myself, there was no loss. Perhaps because the dedup prompt has a rule not to delete details missing from the receiving note, loss itself was rare. The sample is too small to say for sure.
So the fix is shelved, and a person checks for loss
I didn’t add the fix. The judge and the rollback stay as they are. Getting the judge to see loss would mean showing it the receiving note too, which amounts to building a new judge, and since loss is rare right now I decided not to chase it separately. Instead, on every real test, for each rewrite that passed the judge and was kept, I open the receiving note to check that the deleted fact is really there, and record the loss count.
Final thoughts
Part of why rollback was safe was luck. When the judge flagged some other problem, the note’s loss disappeared along with it; the judge never saw the loss. What exposed that was an improvement: “fix only the problems the judge found.” Once the flagged problems were fixed precisely, the unflagged problem went straight through. If I hadn’t set the criteria before measuring, I might have looked at the number 23 passes and added the fix. Only after reading the passing results one by one did I see what those 23 had let through.