Chunking is 80% of your RAG quality

Every team that gets disappointing results from retrieval-augmented generation assumes they need a better model. Nine times out of ten they need better chunks. If the answer is split across two chunks and neither one is retrievable on its own, no model can rescue you.

Fixed-size chunking is a starting point, not a strategy

Splitting every 1,000 characters is fast and completely blind to meaning. It cuts tables in half, separates a heading from the clause beneath it, and orphans list items. Use it as a baseline to beat, never as a destination.

What we actually run

1. 1Parse to a structured tree first: headings, paragraphs, tables, lists.
2. 2Chunk on semantic boundaries, then merge neighbours under 200 tokens.
3. 3Prepend a breadcrumb header (document title › section › subsection) to every chunk.
4. 4Keep tables intact and store a natural-language summary alongside them.
5. 5Overlap by one sentence, not by a fixed character count.

const chunks = splitBySemantics(tree, {
  minTokens: 200,
  maxTokens: 900,
  breadcrumb: true,
  sentenceOverlap: 1,
});

Measure retrieval separately from generation

Build a set of 50 real questions with the paragraph you know contains the answer. Score recall@5 on retrieval alone. If recall is below 90%, tuning prompts is a waste of an afternoon.

You cannot prompt your way out of a retrieval problem.