In Part 2, four runs
confirmed that lowering the chunk size (ingestChunkSize) 6→2→1 cut the rate of factual
errors exactly 40%→10–13%→0%. Before writing up the conclusion, I decided to continue the
experiment in the evening on Laptop 1, at home. But on the night I got ready to rerun it, an
incident arrived before any conclusion.
That night, conflict markers got committed as-is — when you alternate two computers on git
This experiment had Laptop 1 and Laptop 2 taking turns pushing to the same repository, so
I knew that if both machines touched it at once, git could end up in a situation it
couldn’t sort out. That night, exactly that happened. Laptop 2, which had run the
experiments during the day, committed and pushed its work at 17:09, and about four hours
later Laptop 1, without having pulled that in yet, committed its own work at 21:10. One
minute later, at 21:11, a merge commit combining the two machines’ different changes was
created, and the markers git leaves where it couldn’t merge automatically (conflict markers
starting with <<<<<<<) were committed as-is, uncleaned, into 77 files.
It wasn’t a plugin bug at all. It wasn’t even two people; it was one person alternating between two machines who created new work on one before finishing the push on the other. Once I knew the cause, the fix was simple: delete the conflicted wiki folder entirely and recreate it on Laptop 1 from the same 30 sources with note creation (ingest). Cleanup took 34 minutes.
The incident itself ended as a half-hour hiccup, but the lesson from it became an operating rule for every experiment after: only move to the other machine after the work on one machine is fully finished, push included. Never have both machines open and working at the same time.
Right after the cleanup, one setting caught my eye — the OpenAI reasoning effort setting
Right after cleaning up, a setting completely different from chunk size caught my eye. The
plugin has another setting called openaiReasoningEffort. It’s a three-level value,
low/medium/high, that decides how deeply the AI model “thinks” before it answers (the
default is medium). The description on the settings screen read, in Korean, “High: 깊은
추론 (정보 보존율 및 자가 검증 극대화, 소요 시간/비용 증가),” meaning “High: deep
reasoning (maximizes information retention and self-verification, more time/cost).”
Rereading that text, my heart sank. “Information retention”: wasn’t that pointing at
exactly the same thing as the fidelity problems I had been blaming on chunk size?
I checked Laptop 1’s setting: it was High. But I had never checked that value on Laptop 2, where most of the runs had been done. If Laptop 2 had been on Medium from the start, then a good part of what I’d believed was “the chunk-size effect” might really have been “the reasoning effort difference.” On top of that, the git repository that tracks the vault only follows the raw/wiki folders, not the plugin’s settings file, so there was no way to reconstruct Laptop 2’s actual setting at that point in time after the fact. And Laptop 2 was the work laptop I use during the day, so I couldn’t open it that night. Left with only a hunch, I just noted a guess (probably Medium).
Starting with what I could check — the real experiment begins
Not being able to check Laptop 2’s value was no reason to stop. There were two questions I wanted answered. Does reasoning effort really affect fidelity? And how much was that effect mixed into what I’d believed was “the chunk-size effect”? To answer that, this time instead of switching machines I had to rerun the experiment on the same machine, with the same settings, changing just one value at a time.
Continued in Part 4.