After cleaning up the git incident in Part 3,
this time instead of switching machines I redesigned the experiment to change one setting
at a time on the same Laptop 1. The goal was clear: separate which of ingestChunkSize
(the chunk size) and openaiReasoningEffort (the reasoning strength) really governs
fidelity (factual accuracy).
Changing only the chunk size — almost no difference (Run E, F)
On the same Laptop 1, with the same High setting, I changed only the chunk to 2 and then 1 and reran the 30 sources each time. The result wasn’t what I expected. With chunk=2, fidelity problems were 1/30 (3%); with chunk=1, 2/30 (7%). Both were differences at essentially noise level, a completely different pattern from the clear monotonic drop of 40%→10–13%→0% in Part 2.
Two interpretations were possible. First, when reasoning effort is high enough, the influence of chunk size itself might shrink (the two variables partly substituting for each other). Second, the dramatic difference in Part 2 might really have come not from chunk size but from the reasoning effort difference I’d suspected from the start (the possibility that Laptop 2 was actually on Medium), and the pure chunk-size effect might be as small as Run E/F showed.
This one experiment couldn’t tell which was right. I needed one more experiment that fixed the chunk and changed only reasoning effort.
This time, changing only reasoning effort (Run G) — High vs. Medium
With the chunk fixed at 1, I lowered only reasoning effort from High to Medium and ran the same 30 sources again. The result was clear. Fidelity problems: 0/30 (0%), even cleaner than with High (2/30). Instead, two completely different numbers swung widely. The note count dropped from 221 to 150, and processing time fell by more than half, from 25 min 35 s to 12 min 8 s.
The conclusion was now sharp: what governs fidelity (whether the facts are right) is chunk size, and reasoning effort has almost nothing to do with fidelity. What reasoning effort actually governs is atomicity (how finely notes get split) and processing time.
Which sources showed big differences split clearly too — the more narrative the text, the more LLM note splitting differs
Comparing the High vs. Medium note counts source by source, the difference split clearly by the structure of the source. A piece already written as a numbered list, like “AI 시대, 대체 불가능한 리더가 되는 8가지 방법” (“8 ways to become an irreplaceable leader in the AI era”), split into almost the same number of notes under High or Medium (10 vs. 10). On the other hand, narrative writing that flows as one continuous thread, like “how to become an AI-Native Engineer,” differed a lot: 15 notes under High, 4 under Medium. The more a piece requires the model to actively dig out sub-concepts woven into its sentences, the more it’s affected by reasoning effort, which decides “how deeply to think and split.”
Checking directly against the source, neither the 15 finely split notes nor the 4 lumped ones had factual errors. Medium didn’t split wrongly; it simply split less finely than High.
In short
The conclusion so far: keep ingestChunkSize low (1–2) to protect fidelity, and tune
how finely notes get split and how fast it runs separately with openaiReasoningEffort.
They were separate knobs solving different problems.
But this conclusion still had one hole. To confirm that Part 2’s dramatic 40%→10–13%→0% was really “the chunk-size effect alone,” I needed to know what reasoning effort value Laptop 2 had actually been using during those runs. That value was still unconfirmed.
Before I could fill that hole, a question from a completely different direction pushed the investigation somewhere new. It started with a curiosity that seemed unrelated to the original question: “why do some notes stay as separate Korean and English versions instead of merging?”
Continued in Part 5.