Just lowering the LLM chunk size made the fidelity problems go away

In Part 1, I saw that with the same 30 sources and the same plugin, just switching machines (Laptop 1 vs. Laptop 2) dropped the number of resulting notes from 189 to 63. The prime suspect was ingestChunkSize, the setting that decides how many sources the note-creation feature (ingest) bundles into each call to the AI. Laptop 1 had it at 2, Laptop 2 at 6. To confirm whether this one value was really the cause, I had to fix everything else and measure again while changing only this value.

Measuring “is the content right,” not just note count — LLM summary accuracy (fidelity)

The difference in note count alone wasn’t enough. Fewer notes can mean “split less,” but that in itself doesn’t mean “wrong.” The question that really mattered was a different one: if you raise the chunk size and cram several sources in at once, does the accuracy with which the model summarizes them drop?

Answering that meant comparing the raw sources line by line against the wiki notes they produced: checking whether numbers, dates, names, quotes, and causal relationships survived exactly as in the source. In this post I lump these kinds of factual errors (fabrication, omission, distortion, and cross-contamination where content from another source leaks in) together as “fidelity problems.”

Auditing the chunk=6 result first (Run B) — LLM summaries dropping named people and quoted sources

First I audited the very 63 notes Laptop 2 had produced with chunk=6 against all 30 sources. 12 of the 30 sources (40%) had fidelity problems, and not typo-level ones either; they were fairly serious.

In the notes from a source titled “how to become an AI-Native Engineer,” the name of the author who actually wrote the piece (Shah Rahman of Meta) appeared nowhere. The mention of an Anthropic security incident was gone, and so was the attribution that Andrej Karpathy coined the term “vibe coding.” From another source, the name of the person behind a quote (Jeff Hancock, who coined the term “workslop”) was dropped entirely, and from yet another, the person who actually announced a C++ proposal (Vinnie Falco) disappeared along with his quote. In an article about Korean pension policy, neither of the two notes kept the name of the lawmaker who released the statistics, or the political causal chain that led to the policy (a presidential remark → a particular party pushing legislation).

The pattern was clear: named people, organizations, and sources evaporating whole in the compression. On top of that, there were two cases where wording from one source leaked verbatim into a note made from a completely different source. For example, an organizational-psychology note’s “related pages” list linked to an entirely unrelated article about benchmarks, which looked like a trace of another source that had been processed in the same chunk leaking in.

Setting the chunk back to 2 (Run C)

Same sources, same machine (Laptop 2), with only the chunk set back to 2. It produced 108 notes (more than 63 but short of Laptop 1’s 189, an awkward in-between number), and fidelity problems dropped sharply to 3/30 (10%). Even those three remaining cases had accurate body content; they were minor ones like a “related pages” link pointing somewhere wrong, or a subtle wording difference (rendering “mid-level engineer” slightly differently as “mid-career engineer”). Run B’s dominant pattern, “named people and organizations dropped entirely,” did not reproduce even once.

Lowering the chunk to the minimum, 1 (Run D)

Going one step further, I lowered the chunk to 1 (processing just one source at a time) and ran the same 30 sources again. The result: all 30 clean, zero fidelity problems. Of the four runs so far, it was the only perfect result.

The 4-way summary — chunk size and summary accuracy

As the chunk went 6→2→1, the fidelity-problem rate fell monotonically, 40%→10–13%→0%. All four runs pointed the same way: consistent evidence. The note count (atomicity) generally moved in the same direction as the chunk size, but not in a perfectly clean proportion: chunk=1’s 146 was actually lower than chunk=2’s (Laptop 1) 189. That meant there’s natural variation from run to run even with the same settings.

It felt like I could have drawn a conclusion here: “chunk size governs fidelity,” since all four runs pointed the same way. But just before I stamped that conclusion, a doubt came in from a completely different angle: did the two machines’ settings really differ only in chunk size? And on the very night I set out to check that doubt, an incident completely unrelated to the experiment blew up.

Continued in Part 3.