RAG Chunking Strategy: How to Decide Size, Overlap, and Split Boundaries

Model choice gets most of the attention in Retrieval-Augmented Generation (RAG) projects, but the ceiling on answer quality is usually set by something far more mundane: how you split documents into pieces. A bigger model will not rescue a badly chunked knowledge base, because the model never sees more than what the retrieval layer puts in front of it.

This article covers chunk size, overlap, split method, and metadata from a production standpoint: what each decision actually affects, how to measure it, and what changes when the content is Turkish.

What is a chunk, and why does it cap retrieval quality?

A chunk is the smallest unit of a document written to the vector store as its own record. Retrieval returns chunks, not documents, and the language model sees only the chunks that come back. For a fact to be answerable, it has to sit whole inside a single chunk. If the answer is split across two chunks and retrieval returns only one, the model answers from partial information — without knowing it is partial.

The real risk is that this failure is silent. Bad chunking throws no errors; it just makes answers slightly worse. Chunking is therefore a source of quality loss that stays invisible until someone measures it.

What should the chunk size be?

For general-purpose document retrieval, 200-500 tokens works well in practice, and 300 tokens is a reasonable default to start from. Small chunks (100-200 tokens) match more sharply because the vector represents a single idea, but the context needed to answer is often left outside. Large chunks (800+ tokens) preserve context while blurring the vector: text covering five topics produces an embedding that sits close to none of them.

Measure size in tokens rather than characters, and check your embedding model's input limit. Text beyond that limit is silently truncated, which means the last paragraph of your chunk may never have entered the vector at all. Plot a histogram of your chunk token counts as well — averages routinely look fine while the tail runs past the limit.

How much overlap should you use?

Overlap is the text shared between consecutive chunks, and it keeps sentences at a boundary from being lost. Common practice is 10-20% of the chunk size: 30-60 tokens for a 300-token chunk. The goal is to guarantee that information split mid-sentence appears intact in at least one chunk.

Raising overlap is tempting but not free. At 50%, index size and embedding cost roughly double, and retrieval starts returning chunks that are near-duplicates of each other — spending half of your top-k on a single piece of information. Treat overlap as insurance, not as a strategy.

Fixed size, structural boundaries, or semantic splitting?

There are three approaches, and the choice depends on document type. Fixed-size splitting cuts at a token count; it is the simplest to implement and reasonable for unstructured prose. Structural splitting follows headings, paragraphs, and list boundaries, and gives by far the best cost-to-benefit ratio on Markdown, HTML, technical documentation, and regulation — anything with visible hierarchy. Semantic splitting compares embeddings of consecutive sentences and cuts where the topic shifts; it is the most expensive and only earns its cost on long unstructured narrative.

In practice the right starting point is structural: use the document's own heading hierarchy, then split anything still larger than your target range by fixed size. Move to semantic splitting only when measurement shows structural is not enough — most projects never reach that point.

How should tables, code blocks, and lists be split?

They should not be. Separate a table from its header row and the remaining rows become meaningless: the information saying what each column represents now lives in a different chunk. The same holds for code; a function cut in half can neither run nor be explained. Treat these structures as atomic units even when they exceed your token target.

If a table genuinely exceeds the target, split it into row groups and repeat the header row in each piece. For long code files, split at function or class boundaries rather than line counts. The general rule: your split boundary should coincide with the content's own boundary of meaning.

What metadata belongs on a chunk?

At minimum: source document id, the heading hierarchy within the document, section or page number, and the content's date. Write the heading hierarchy into the start of the chunk text as well — a prefix like "Product > Pricing > Enterprise Plan" teaches the embedding where the chunk sits and grounds pieces that would read as meaningless in isolation.

Metadata's second job is filtering: excluding content that has aged out, keeping sources the user is not authorised to see out of retrieval, or scoping to a specific product. Its third is citation — if you cannot show the user which section of which document an answer came from, the system's accuracy cannot be audited.

When is the small-to-big (parent document) approach right?

In this approach retrieval runs over small chunks, but the model receives the larger unit that contains the match — the parent paragraph, section, or document. That removes the trade-off between retrieval sharpness and answer context: the small vector matches precisely, the large text is enough to write the answer from.

The cost is complexity: a mapping from each chunk to its parent, and more tokens entering the prompt. Where answers fail as "found the right document but explained it incompletely," this is the single highest-return change available. If answers are landing on the wrong document entirely, the problem is not chunk size but the retrieval layer.

How do you measure a chunking strategy?

Build a golden set of 50-100 questions and hand-label which document and section holds each answer. Then evaluate the retrieval layer alone — does the correct piece appear in the top k results (recall@k), and at what rank (MRR). Run this without the language model; evaluating through generated answers makes retrieval failures indistinguishable from generation failures.

Building a golden set takes time, which is exactly why most teams skip it. Skipped, chunk size becomes a matter of opinion; built, it becomes a two-hour experiment. You reuse the same set later when changing embedding models, so the cost is paid once.

What changes when chunking Turkish content?

Two things. First, tokenization: common BPE tokenizers are trained predominantly on English and split Turkish words into more pieces. Turkish text of the same character length produces noticeably more tokens than English, so a chunk size tuned by character count comes out larger than expected in Turkish and can silently exceed the embedding limit. Always measure size with the actual tokenizer you use.

Second, morphology: Turkish is agglutinative, and one root appears in many surface forms. This weakens keyword search such as BM25 on its own, because "faturalandırma" and "faturaların" look like unrelated terms. Vector search absorbs most of that difference; if you are building hybrid retrieval, benchmarking Turkish without stemming or lemmatization on the keyword side gives you a misleading result.

How many chunks should you return (top-k)?

Top-k is how many chunks retrieval hands to the model, and it has to be tuned together with chunk size: k=5 over 300-token chunks is roughly 1,500 tokens of context, while the same k over 800-token chunks passes four thousand. Small chunks call for a larger k, large chunks for a smaller one; tuning the two independently is a common mistake.

Larger k is not always better. As irrelevant chunks enter the prompt the model has to divide attention, and past a point answer quality declines. Look at the recall@k curve on your golden set: where recall flattens is the natural ceiling for k, and beyond it you are paying for noise.

Does re-ranking change the chunk size decision?

Yes, usually in favour of smaller chunks. A re-ranker rescores question-chunk pairs individually and reorders the candidate set that vector search returned. That lets you retrieve a wide candidate set cheaply (k=50, say) and select the best five with a precise model — small chunks keep their sharp-matching advantage while a large k compensates for their weaker recall.

Re-ranking adds latency and cost, so it is not required everywhere. As a rule: if the correct chunk is in the top 50 but not the top 5, a re-ranker solves your problem. If it is not in the top 50 either, the problem lies in chunking or the embedding model, and no re-ranker will recover it.

How does hybrid search affect chunking strategy?

Hybrid search combines vector similarity with a keyword score such as BM25, covering the cases where vector search is weakest: product codes, error messages, regulation clauses — anything needing exact match. The keyword side responds to chunk size differently from the vector side, because BM25 dilutes term density in long text, so scores converge and discriminate less as chunks grow.

In practice that makes mid-range chunks (300-500 tokens) a reasonable middle ground for both methods in a hybrid setup. For Turkish, do not benchmark without stemming on the keyword side; a suffix-blind comparison makes BM25 look weaker than it is and points you at the wrong conclusion.

The five most common mistakes

First, choosing a chunk size without measuring and never revisiting it. Second, treating overlap as a strategy and raising it, which inflates index cost and fills top-k with near-duplicates. Third, splitting tables and code blocks like ordinary prose. Fourth, deferring metadata — adding it later means rebuilding the entire index. Fifth, evaluating retrieval only through end-to-end answer quality, which leaves you unable to tell a retrieval failure from a generation one.

What these five share is that none of them looks like a bug. The system runs, answers appear, and they are simply worse than they should be — invisible to everyone because there is no point of comparison.