My first RAG prototype took an afternoon. It answered every question I threw at it, and I was convinced the hard part was done. It wasn't.
Why prototypes lie
When you write the test questions yourself, you phrase them the way your documents are phrased. Real users don't.
Key idea: build your eval set from real questions, not your own guesses.
Chunking is a product decision
Split on document structure first, then on size.
def chunk(doc, max_tokens=400):
for section in split_by_headings(doc):
yield from split_by_size(section, max_tokens, prefix=section.heading)Evaluate before you tune
Measure retrieval and generation separately — a perfect answer from the wrong source is still a bug.
Takeaways
Start with real questions, chunk by structure, and build the eval set before you tune.