The Obsidian plugin I’m building takes in an article and has an LLM split the concepts, people, and companies in it into separate notes. Notes from the same article tend to repeat the same facts, so right after the notes are created there is a dedup step: an LLM finds the overlapping facts, keeps each one in a single note, and replaces that sentence in the other notes with a link sentence like “covered in detail in [[that note]].” In Part 9 I chased a problem where, after this step deleted sentences, the sentences left behind lost what they pointed to. Instead of blocking each broken shape in advance, I ended up deciding to have another LLM reread each edited note and judge it. This post is the record of actually putting that judge into the plugin and running it.
If the judge flags a note, the whole note is rolled back — LLM-as-a-judge
The approach is simple. When the dedup step edits some notes, one judge call per edited note goes out, all at once. The judge sees the note before the edit, the note after the edit, and the changed paragraphs, and answers one question: did this edit create a new problem? The criteria are the four set in Part 9. Did a referring word lose its target? Did a link-only sentence or a cut-off sentence appear? Did a condition or part of a list go missing and change the meaning? Does a section heading no longer match what’s left under it?
If the judge says there is a problem, every rewrite in that note is rolled back. It is also rolled back if the judge call fails or the answer doesn’t match the expected format. Either way, a wrong verdict only leaves one duplicate behind; a broken note never gets saved. That asymmetry is why I chose this approach.
Five runs of six sources each — how accurate the LLM judge was
In a test environment where I remote-control Obsidian on another PC, I fed in six articles and read every resulting note myself, five times. Each run used a different set of six articles.
The judge ran on 22 notes. 14 passed and were kept, and 8 were rolled back. Of the 8 rolled back, 7 really were broken. Some examples:
- A sentence describing employee inquiries was deleted, so in the next sentence, “the same questions kept coming” (from a Korean note), you could no longer tell what the questions were the same as. On top of that, the note that took over the content pointed back the other way: “this background and the HR team’s other automation cases are covered in [[the original note]].” Left as is, the two notes would only have pointed at each other, and the background would not have survived intact anywhere.
- The preceding sentence turned into a link sentence, so the “however” in the next sentence lost what it contrasted with.
- The subject of the first sentence turned into a link sentence, so the next “also … set as a goal” (from a Korean note) lost its subject.
- The pre-edit note’s “helped her begin to overcome” was carried into a link sentence and inflated to “helping her overcome.”
- All of one section’s explanation was deleted, leaving only the “See [[…]].” sentence that had originally been at its end.
The remaining one was too strict. The judge said the meaning had changed, but the changed sentence was a pointer to another note, and the deleted content was fully present in that note. The cost was just one duplicate left behind.
Time wasn’t a problem either. The judge took about 2–12 seconds per note, while turning six articles into notes took between four and a half and six and a half minutes each run.
A verdict I thought was wrong was right — confirmed by rerunning it
In the first run, the reason given for one rolled-back note looked off. A “See [[…]].” sentence had been at the end of a numbered list item from the start, and the judge flagged that sentence as the problem. I assumed it had misread a sentence that was already there as a new one, and wrote “the judge’s reason is inaccurate” in the report.
To check, I rebuilt the same input the plugin had sent and ran the judge five times for each variant. Fed in as is, it flagged the note four times out of five. With every See sentence removed, it never flagged it. With the See sentence kept and only that item’s rewrite removed, it flagged it once out of five (for a different reason). So the problem wasn’t the See sentence that had always been there. The rewrite had deleted that item’s explanation, leaving only a bold heading and one See sentence, so the link sentence became the item’s only content. The verdict matched the criteria, and the “inaccurate” in my report was my own wrong guess, so I corrected it.
What the judge sees weakly — dropped subjects in Korean
Of the 14 that passed, 3 were borderline. All three had the same shape: a deleted sentence took the subject or context of the next sentence with it.
For example, one note replaced two sentences, saying the person had built a Chrome translation extension by describing the features they wanted in plain words, with a single link sentence saying the build process is covered in that extension’s note. That left the next sentence, “when a problem came up during installation, they described the situation and got a fix,” with nothing to say what was being installed. Korean drops subjects often, so it still reads fine, and the deleted content was all present in another note, so no information was lost. The judge caught visible connectives like “also” and “however” well, but saw weakly the context that a dropped subject had been holding.
The price paid — LLM dedup rolled back by nearly half
Safety was secured. Across the five runs, a broken note never made it into the final result. In exchange, 13 of the 28 rewrites in the notes the judge checked, 46%, were rolled back. Whatever was rolled back leaves the duplication it was meant to remove.
The rolled-back cases grouped into three shapes again: the deleted sentence took the next sentence’s subject or target with it, a section or list item was left with only a link sentence, or a newly added pointer sentence changed the meaning while summarizing the original fact. All three are problems made by the dedup step before the judge, not by the judge. So the next question became not how to fix the judge, but how to make the rewrites the judge is blocking break less.
Remaining thoughts
Before adding the judge, the goal was to leave no broken note behind. That was achieved. But once the blocking mechanism clearly worked, what it was blocking, and how much, came into view. The 13 rollbacks were the judge’s results, and at the same time a measurement that over 40% of the rewrites in the notes the judge checked had been broken (12 even excluding the one too-strict case). And the one verdict I thought was wrong turned out to be my guess being wrong. When a reason looks off, rebuilding the same input and running it again was faster than guessing.