The Obsidian plugin I’m building takes several articles and has an LLM turn each one into notes on its concepts, people and companies. Notes from the same article tend to repeat the same facts, so before the notes are written there is a step where an LLM cleans up overlapping sentences and another LLM rereads each edited note and judges it. The previous post and the ones before it were about making that cleanup and judging trustworthy. This one is about making the same step fast.
Where did the time go — a sequential bottleneck
Feeding in six articles took four and a half to six and a half minutes every time. I had just added the judge, so I checked it first, but it took only 2 to 12 seconds per note. Instead of guessing, I split the run into stages using the timestamps in the run logs, and the big chunk was somewhere else.
In the test settings, the plugin sends four articles to the LLM at once, and once all the answers are back, it writes the notes one article at a time, in order. The dedup step sat inside that writing stage, right before each file was written. So dedup calls of 3 to 62 seconds each ran back to back, one per article. Across five real runs, writing the first four articles took 77 to 168 seconds, and more than 90% of that was dedup calls. The last two articles didn’t send their requests until all of that writing was done.
There was a reason not to just run them at once — parallel processing
It looked like I could just run the dedup calls for different articles at the same time, but the write order existed
for a reason. The next article’s notes look at the files the previous article actually wrote to decide whether a name
collides (if it does, _dup2 is appended) and whether a link points to a real note. Running dedup concurrently before
writing would change the titles and links that dedup sees.
So I kept the order and moved only the dedup to after the write. Notes are written first and the plugin moves straight on to the next article, while dedup and judging run in the background, per article and concurrently, and patch the written files as they finish. Right before patching, if a file changed in the meantime, it is skipped. The final step that merges notes whose names collided, and the automatic cleanup suggestions after it, run only once every dedup is done.
Faster was obvious, but “the same” had to be proven separately — LLM response replay testing
Running the same articles on the new build, the writing stage dropped from 164 seconds to 4. Faster showed up as a single number. The hard part was “are the results the same as before?” An LLM produces different notes from the same article every time, so if you put the notes from an old run and a new run side by side, you can’t tell whether a difference came from the code or from the LLM.
So I took the LLM out of the comparison. I fed the LLM responses saved from real runs (note generation, dedup, judging) back in unchanged, and built the notes with the old code and the new code separately. Both sets, 56 and 49 files, were identical down to the character. The replayed results also matched the notes from the real runs (only one file differed, where I had substituted a merge response that hadn’t been saved), which confirmed that the replay reproduces the real thing.
Comparing only the notes would have missed something, though. The replayer looked up recorded responses by note title,
so even if the input sent to the LLM differed, the same answer came back. So I also compared the input of every call,
and exactly one place differed: a link to a note that had _dup2 appended because of a name collision. The old code
fixed that link after dedup; the new code passes dedup a body where it is already fixed. The final files were the same.
Was the 125-second call slow because of concurrency — response time and output tokens
In the new build’s first run, one dedup call took 125 seconds. The same group of articles had taken 22 seconds in the earlier run. If sending several calls at once had slowed each one down, I would need to cap the number of concurrent calls.
I measured instead of deciding by guess. I took the same six dedup inputs and alternated, three times each, between sending them one at a time and sending all six at once. The median call time was 14.1 versus 14.3 seconds, and output speed was 108 versus 111 tokens per second, the same. Call time was proportional to the length of the answer, and even for the same input, the answer length varied severalfold from round to round. One group ranged from 3,700 to 7,300 tokens. The 125 seconds looks like that round’s answer simply being unusually long, so I set no cap.
I measured them alternately in the same time window — A/B testing
The first comparison was several hours apart from the old runs, so differences in API response speed were mixed in. So I ran the old build and the new build back to back, alternating. With four articles sent at a time, 338 seconds became 272, and 303 became 171. At the plugin’s default of two at a time, the articles were split into three rounds, so there were more stretches of waiting, and 554 seconds became 313. That’s 44%. I also read the real runs’ dedup results one by one. Cases where a deleted fact wasn’t still present in another note: zero on the new build.
A cancel test lost log lines — a file write race condition
Last, I pressed the stop button in the middle of an ingest. It was right after the first four articles were written and the dedup had started running in the background. The ingest ended cleanly a second later and the notes were intact. But the debug log had four dedup-start lines and only one dedup-complete line.
If three dedup calls had failed to finish and were hanging somewhere, that would be a big problem. Tracing it, the dedup calls had all stopped and finished along with the cancel; only the log lines had vanished. Logging worked by reading the whole file, appending a line and writing it back, with no lock. Called at the same time, each call reads the same contents and writes its own version, overwriting the others’ lines. Writing four times concurrently with the real code left one line in the file; doing it in sequence left four.
This bug predates the change. In every run log so far, of the start records for the four articles sent together, only one had survived, and this was why. In fact it had caught my eye about ten days before this measurement. I noted in that run’s report that the start records of the articles sent together had never been written to the file at all, and even wrote that concurrent writes seemed to be colliding. But I didn’t follow it up, and after that I computed start times by subtracting the elapsed time from the completion time. Once this change increased concurrent logging, it surfaced in a way I could no longer wave off. I made logging write one line at a time, in order, and shipped the fix in the same release.
What stays with me
Faster showed up right away as a single number. “The same” didn’t show up on its own. With an LLM in the loop that gives a different answer every time, putting two results side by side proves nothing. Only after pinning the LLM’s answers and feeding them in could I compare just the code. And the logs I was using for verification could be wrong too. My audits were counting rewrites and judgments from those logs, so when the log lost lines, the audits could have been quietly undercounting.