What this plugin does — organizing Obsidian notes automatically with an LLM
Some background first. I’m building a plugin of my own (not released yet). It runs on top of a note-taking app called Obsidian and uses an LLM to organize notes, and it has two core features.
The first, the note-creation feature (ingest from here on), reads source text such as news articles or blog posts (“raw” documents) and splits each concept, person, and event in it into its own standalone note (“wiki” notes). For example, feed it a single article titled “8 habits of leaders who survive the AI era,” and each of the 8 habits it mentions, along with the people who appear in the article, becomes a separate note. In this plugin I call the principle of having each note cover one clear entity or concept “atomicity.” When a single note lumps several topics together, both searching and linking it to other notes get murky later.
The second, the wiki-cleanup feature (refactor from here on), goes back over the accumulated notes later: if two notes actually cover the same thing, it proposes merging them, and if one note bundles several concepts, it proposes splitting it. If ingest is the entrance that breaks incoming writing into small pieces, refactor is what tidies up the warehouse those pieces pile into.
This series is a record of an investigation that ran over several weeks into the quality of ingest (= how accurately it splits things). What set it off was one very simple observation.
I expected the same sources to give the same results — LLM output reproducibility
There’s a test vault (a vault is what Obsidian calls its unit of “working folder”) where this plugin actually runs. The vault is synced in real time across two laptops (I’ll call them “Laptop 1” and “Laptop 2” here) by a separate plugin called Obsidian LiveSync, and separately, the contents of its raw/wiki folders are also kept in a git repository, so I can compare and audit the note contents at different points in time.
This test vault holds 30 news articles and blog posts as raw sources. On Laptop 1, I ran all 30 through ingest from scratch and got 189 wiki notes. I ran the same 30 sources, with the same version of the plugin, again on Laptop 2. It produced 63 notes.
That was nearly a 3x difference. The sources were the same and the plugin version was the same; there was no reason the results should diverge this much just because I switched machines. Opening a few individual cases made the difference even clearer. For one source, “The 10 Best Open-Source Coding LLMs Right Now,” Laptop 1’s output split every model the source mentioned (GLM 5.2, Kimi K2, DeepSeek V4, Qwen3 Coder, MiniMax M3, and so on) into its own note, 15 in all. Laptop 2’s output for the same article was just 3 notes, and most of the individual model names had vanished from the note titles and bodies. The article “AI 시대, 대체 불가능한 리더가 되는 8가지 방법” (“8 ways to become an irreplaceable leader in the AI era”) was the same story: Laptop 1 made 10 notes, one for each of the 8 habits plus the person, while Laptop 2 lumped all 8 together into a single note called “AI 시대 대체 불가능한 리더십” (“irreplaceable leadership in the AI era”).
Same input, same software, results this different: it had to be one of two things. Either the plugin had a bug that behaved differently depending on the machine’s environment, or the two machines’ settings themselves were different.
I found one suspect — LLM chunk size
Digging through the plugin’s settings, I found a value called ingestChunkSize. It’s the
number that decides how many raw sources ingest bundles into a single AI call. A larger
value bundles more sources per call, so there are fewer API calls and the whole run
finishes faster, but it also means stuffing that much more content into each call.
When I checked the two machines’ values, Laptop 1 was set to 2 and Laptop 2 to 6. It was an immediate prime suspect: with a 3x difference in how much each call handled, a nearly 3x difference in the number of resulting notes seemed to add up. But I couldn’t conclude anything from a hunch alone. To confirm whether this one number was really the cause, or whether something else was mixed in, I had to fix every other condition and measure while changing only this value.
So I designed a controlled experiment: the same 30 sources, run again while changing only the chunk size. The results, and how they told me far more than I expected, are the story of the next part.
Continued in Part 2.