Chunking Strategies for Retrieval-Augmented Generation

In a RAG system, the retriever never sees your documents — it sees chunks. Each chunk is embedded, indexed, and returned (or not) as a unit. If the answer to a question is split across two chunks, or buried in a chunk dominated by unrelated text, retrieval quality suffers no matter how good the embedding model or the generator is.

Chunking is one of the most underrated levers in RAG. This explainer covers the main strategies, how overlap and metadata fit in, and how to choose with measurements rather than defaults.

Why chunk boundaries matter

A chunk has two jobs that pull in opposite directions. For retrieval, it should be focused: one topic per chunk produces an embedding that clearly represents that topic. For generation, it should be complete: the model needs enough surrounding context to understand and use the passage correctly.

• Too small: chunks lose context. A sentence like “This limit does not apply to enterprise plans” is useless without knowing which limit.
• Too large: chunks mix topics, embeddings blur, and each retrieved chunk consumes more of the context window.
• Badly placed boundaries: splitting mid-table, mid-list, or mid-code-block produces fragments that neither retrieve nor read well.

The right balance also depends on the questions users ask. Narrow factual lookups (“What is the API rate limit?”) favor smaller, focused chunks. Questions that need explanation or comparison (“How does billing work when a team downgrades?”) favor larger chunks or retrieving neighboring chunks together. Look at real queries before settling on a size.

Fixed-size chunking

The simplest strategy cuts text every N characters or tokens. It's predictable and easy to implement, and for uniform prose it can be a reasonable baseline. Its weakness is that it ignores meaning: boundaries land wherever the counter runs out, often mid-sentence.

Fixed-size chunking is still worth building first, because it gives you a baseline to beat. If a more sophisticated strategy doesn't measurably improve retrieval on your own questions, the extra parsing code isn't earning its keep.

Recursive chunking

Recursive chunking tries a list of separators from coarsest to finest — paragraph breaks, then line breaks, then sentence ends, then spaces — and only falls back to a finer separator when a piece is still too large. Boundaries tend to land on natural breaks, while chunk size stays within a budget. It's a strong general-purpose default.

Structure-aware chunking

Many corpora carry explicit structure: Markdown headings, HTML sections, API reference entries, numbered policy clauses. Structure-aware chunking splits on those boundaries first, so each chunk belongs to exactly one section, and then applies recursive splitting within sections that are too long. It also lets you attach the section path (for example, Billing > Refunds > Eligibility) as metadata.

Overlap and metadata

Overlap repeats the tail of one chunk at the start of the next, so a fact near a boundary appears in both. It helps when boundaries are imperfect, but it isn't free: overlapping chunks increase index size and can return near-duplicate results that waste context. Keep overlap modest relative to chunk size, and deduplicate results before building the prompt.

Metadata often matters as much as the text. Store the source document, section heading, URL or path, last-updated date, and access-control labels with each chunk. You can filter on them at query time, show them as citations, and prepend the section heading to the chunk text before embedding so short passages carry their context.

A structure-aware chunker with overlap

The chunker below splits Markdown on headings first, then recursively splits oversized sections on progressively finer separators. Sizes are measured in characters for simplicity; in production you'd typically measure in tokens using the same tokenizer as your embedding model.

1import re2from dataclasses import dataclass3 4SEPARATORS = ["\n\n", "\n", ". ", " "]  # coarsest to finest5 6 7@dataclass8class Chunk:9    text: str10    section: str  # heading the chunk came from; useful for citations and filters11 12 13def split_recursive(text: str, max_chars: int, seps: list[str] = SEPARATORS) -> list[str]:14    if len(text) <= max_chars:15        return [text]16    if not seps:  # no natural boundary left: hard cut17        return [text[i : i + max_chars] for i in range(0, len(text), max_chars)]18    sep, finer = seps[0], seps[1:]19    pieces: list[str] = []20    buffer = ""21    for part in text.split(sep):22        candidate = f"{buffer}{sep}{part}" if buffer else part23        if len(candidate) <= max_chars:24            buffer = candidate25            continue26        if buffer:27            pieces.append(buffer)28        if len(part) <= max_chars:29            buffer = part30        else:  # a single part is still too big: retry with finer separators31            pieces.extend(split_recursive(part, max_chars, finer))32            buffer = ""33    if buffer:34        pieces.append(buffer)35    return pieces36 37 38def chunk_markdown(doc: str, max_chars: int = 1200, overlap: int = 150) -> list[Chunk]:39    chunks: list[Chunk] = []40    for section in re.split(r"(?m)^(?=#{1,3} )", doc):  # split before each heading41        if not section.strip():42            continue43        first_line = section.split("\n", 1)[0]44        title = first_line.lstrip("#").strip() if first_line.startswith("#") else ""45        pieces = split_recursive(section.strip(), max_chars - overlap)46        for i, piece in enumerate(pieces):47            prefix = pieces[i - 1][-overlap:] if i > 0 else ""  # overlap stays within a section48            chunks.append(Chunk(text=prefix + piece, section=title))49    return chunks

Evaluate chunking with retrieval metrics

Chunking choices are empirical. Build a set of questions where you know which passage contains the answer, then measure retrieval directly, separately from the generated answer.

• Recall at k: how often the chunk containing the answer appears in the top k results.
• Answer completeness: whether the retrieved chunks, taken together, contain everything needed to answer.
• Context efficiency: how much of the retrieved text is actually relevant, which affects cost and answer focus.

You don't need a large labeled set to start. A few dozen questions written by someone who knows the corpus, each paired with the ID of the passage that answers it, is enough to compare strategies and catch obvious regressions. Expand the set over time with real user queries that retrieval got wrong.

Change one variable at a time — strategy, size, overlap, or heading prefixing — and re-index between runs. When you compare configurations, read the misses: they usually point to a specific structural problem, like FAQ answers separated from their questions, that a targeted rule can fix.