The Obsidian plugin I’m building has a feature (I call it ingest) that takes an article and has an LLM split the concepts, people, and companies in it into separate notes saved to a wiki. In Part 4, I found a bug where notes created together from the same article (sibling notes) copied the same paragraph word for word. This time I turned to fixing it. I tweaked the prompt wording and ran a few rounds of reproduction tests. The reproduction rate dropped, but the bug didn’t disappear completely. So I decided to build something more fundamental: a post-processing pipeline that, every time ingest creates several sibling notes, automatically checks them for overlapping paragraphs and fixes them before they’re written.
The design had two stages. First, turn the paragraphs into vectors and score mathematically how similar they are. Then pick only the high-scoring candidate pairs and hand them to something that can actually reread and judge them. While discussing this, I got a question: “Isn’t that just using an LLM again?” It was both right and wrong. Both are AI models called through an API, but in practice they do completely different jobs.
A chat LLM is a model that “produces answers” — LLM vs. embedding model
Models like gpt-5.6-luna, which this plugin uses to write and judge notes, produce an answer by predicting the next word that’s likely to follow the prompt, one at a time, and stringing them together. Ask it “Do these two paragraphs contain duplicated content?” and it composes a fresh answer to that question as a sentence (or JSON). The key point is that the output is text. Both the reasoning and the result come out as “words.”
An embedding model is a model that “tells you a position” — cosine similarity
Give an embedding model a paragraph, on the other hand, and it doesn’t return
a sentence. Instead it returns a single array of hundreds of numbers (a
vector). For example, text-embedding-3-small turns a paragraph into up to
1,536 numbers. A person can’t read those numbers and tell what they mean.
Instead, think of the vector as a point in a coordinate space, and the model is
trained so that sentences with similar meanings land closer together. Turn
“OpenAI is a full-stack organization” and “OpenAI is an organization that owns
the whole process from compute to product” into vectors, and the two points
land close to each other in that space. Computing this “closeness” as a single
number called cosine similarity (in theory between -1 and 1, but for text
embeddings like these it usually comes out between 0 and 1) gives the
“similarity score” I kept talking about this time. An embedding model doesn’t
judge. It only tells you a position, and deciding “similar or not” from that
position is the job of whoever receives the score (here, code first and an LLM
in the end).
Why it’s called “embedding” — word2vec
The name was originally a mathematical term. Taking some structure (say, a lower-dimensional shape) and “embedding” it into a larger space (higher dimensions) while preserving its original relationships (distances or shape) as much as possible is called an embedding. In text-handling AI, representations that embed words into a vector space so that words with similar meanings sit close together came to be called “word embeddings,” and the name stuck as Google’s word2vec popularized the approach in 2013. “Embedding a sentence in a vector space” means pushing the meaning of a sentence, something that originally couldn’t be measured with coordinates, into a space where it can be. So an “embedding model” is literally “a model that embeds text into a coordinate space,” and it was built for a different purpose than a chat LLM from the start.
So why not just ask an LLM every time? — LLM call cost vs. embeddings
In theory, you could get the same result by repeatedly putting two paragraphs in front of an LLM and asking “Do these overlap?” The problem is scale. If one article yields about ten sibling notes, they contain dozens of paragraphs, and the number of all pairs to check grows with the square of the paragraph count. Fifty paragraphs make more than a thousand pairs. Asking an LLM about each one, at nearly four seconds per judgment call as I measured this time, would take over an hour even processed in order. Embeddings, by contrast, return fifty vectors in under a second when the fifty paragraphs are sent as one batch. After that, “which pairs are really suspicious” is pure computation with no API calls at all, just a few multiplications between vectors (the cosine similarity). Only the few that pass that filter (usually single digits per chunk, into the teens at most) go to the LLM for the real judgment. An embedding isn’t a tool that gives you the answer. It’s a first-pass filter that flags “this is a candidate” so expensive LLM judgment isn’t wasted everywhere.
Embedding models from the same company come in tiers — text-embedding-3-small vs. large
OpenAI’s embedding models also come in two tiers, text-embedding-3-small and
text-embedding-3-large. Just as chat models have “mini” and full-size tiers,
they differ in training scale and parameter count, and so in how finely the
vectors capture meaning. This pipeline initially used small, and truncated to
256 dimensions at that (the OpenAI embeddings API offers an option to receive
vectors shorter than their original size, to save on storage). The existing
wiki cleanup feature (refactor), which finds duplicates among accumulated notes
and suggests merging them, already used it that way, and I had already set a
threshold (cosine similarity 0.80) for that setting through measurement.
But while validating the pipeline on real data, I found that true duplicates and harmless sentences that just happened to look alike were mixed together in almost the same score range (0.72–0.80). For example, “Codeberg’s approach is one of the models covered in the open-source AI policy comparison” and “LLVM’s approach is one of the models covered in the open-source AI policy comparison” describe completely different projects, yet their similar sentence frame produced a coincidental similarity that got the same score as a true duplicate: the sentence “Kakao is transferring its stake in AXZ, the operator of Daum, to Upstage” (Kakao and Upstage are Korean tech companies, and Daum is a Korean web portal), repeated almost verbatim in two sibling notes. When that happens, no amount of threshold tuning can separate the two.
Recomputing the same paragraph pairs with text-embedding-3-large flipped the
order. The true duplicates rose to 0.78–0.92, while look-alike sentences like
the Codeberg/LLVM pair dropped below 0.69. The range where the two had been
mixed under small split into a clear gap under large. Even with the same
cosine similarity formula, the results changed this much depending on how well
the model captured the subtler difference of “what the sentence is about” in
its vectors.
Large wasn’t a cure-all, of course. A remark by CEO Kim Sung-hoon (“freeing people from work they do reluctantly to make a living…”) repeated in two sibling notes with only slight wording changes, and a parameter figure (“models with 100 billion or fewer… models with 200 billion or more…”) repeated down to the numbers, still stayed between 0.68 and 0.74 under large and didn’t clear the new threshold. To a human reader they were nearly copied, yet the embedding model still saw that degree of similarity as merely “fairly similar.”
Lingering thoughts
Seeing the score order flip just by switching embedding models made it even clearer that trying to settle the answer with a single number, like “cosine similarity of 0.8 or higher means duplicate,” was a stretch from the start. Embeddings are great as a filter that narrows down candidates, but how precise that filter is depends entirely on which model you use, and even the best model leaves cases it can’t catch. This also confirmed again that the final judgment has to go to something that can actually read and understand the sentences: a chat LLM.