How do you choose chunking and indexing settings that improve RAG retrieval (not just embeddings)?

Conclusion

Most RAG quality problems are retrieval problems. Chunking and indexing decide what the retriever can “see.”

A practical default that works for many text corpora:

• chunk by structure (headings/sections) before character counts
• keep chunks self-contained (no missing definitions)
• attach strong metadata (doc_id, section, updated_at, tenant)
• filter aggressively (tenant, access, product area)
• re-index with a clear policy (not “whenever”)

If you don’t do this, you get the classic failure: the wrong chunk wins.

Explanation

Chunking is not about picking “500 tokens.” It’s about aligning your chunks with how users ask questions.

Bad chunking/indexing causes:

• irrelevant chunks outranking relevant ones
• missing context (definitions split across chunks)
• duplicated text that dominates retrieval
• stale policies beating newer docs
• cross-tenant leakage if metadata isn’t enforced

The goal is:

• high recall (you retrieve the right doc)
• high precision (you don’t retrieve noise)

Practical Guide

Step 1: define your retrieval target (5 minutes)

Answer one question:

• what should be retrieved: a paragraph, a section, or a full doc?

Rule:

• optimize chunk size for the unit you want to cite and show

Step 2: chunk by structure first (15 minutes)

Prefer:

• markdown headings
• HTML sections
• PDF page blocks (with titles)

Only fall back to fixed-size chunking when structure is missing.

Recommended pattern:

• section-based chunks
• with a max token cap
• with small overlap for continuity

Step 3: pick chunk size and overlap intentionally (10 minutes)

Starting points:

• 300–800 tokens per chunk for knowledge docs
• overlap 30–100 tokens for continuity

Rules:

• too small → retrieval misses context
• too large → irrelevant content pollutes the answer

Step 4: attach metadata that enables filters (10 minutes)

Minimum metadata:

• doc_id (stable)
• chunk_id
• source_type (wiki, ticket, upload)
• updated_at
• tenant_id / org_id
• access labels (role, team)
• product_area / tags

Rule:

• metadata must be enforced in retrieval, not only stored

Step 5: deduplicate and normalize (10 minutes)

RAG corpora often contain repeated boilerplate. Do this:

• strip nav/footer noise
• collapse repeated disclaimers
• detect near-duplicates and keep one canonical copy

This reduces “boilerplate wins retrieval.”

Step 6: define a re-index policy (10 minutes)

You need explicit rules:

• when docs are re-embedded (on edit, nightly batch, etc.)
• how deletions propagate
• how long stale chunks can survive

Minimum:

• on-write re-index for trusted sources
• quarantine + delayed index for untrusted sources

Step 7: measure retrieval quality (10 minutes)

Track:

• top-k hit rate on a small eval set
• % queries with low similarity scores
• duplicate chunk frequency in top-k

Log:

• retrieved_doc_ids + chunk_ids
• scores
• filters applied

If you can’t measure retrieval, you can’t tune it.

Pitfalls

• fixed-size chunking on structured docs (breaks sections)
• no tenant/access filtering (security risk)
• no dedup (boilerplate dominates)
• re-indexing without deletion handling (stale answers)
• tuning embeddings when metadata filters are the real fix

Checklist

• [ ] I defined the retrieval unit (paragraph/section/doc)
• [ ] Chunking is structure-first (headings/sections)
• [ ] Chunk size and overlap are intentional and documented
• [ ] Every chunk has doc_id + chunk_id
• [ ] Metadata includes tenant_id and access labels
• [ ] Retrieval enforces metadata filters (not optional)
• [ ] Boilerplate and near-duplicates are reduced
• [ ] Re-index policy is explicit (edits + deletions)
• [ ] Untrusted sources use quarantine/delayed indexing
• [ ] Retrieval logs doc_ids/chunk_ids and scores
• [ ] I track top-k hit rate on an eval set

FAQ

1) What chunk size should I use?

Use structure-first chunking. If you need a number, start around 500 tokens and tune based on hit rate and citation UX.

2) Is overlap always good?

No. Too much overlap increases duplication in top-k. Keep it small and measure.

3) What’s the fastest improvement for messy retrieval?

Add strong metadata filters (tenant/access/product area) and deduplicate boilerplate. It often beats fancy embedding tweaks.

Internal links

• Hub: Indexing
• Related:
  • RAG citations and grounding checklist
  • RAG data poisoning defense checklist
  • Minimal LLM logging checklist

Disclaimer

General engineering guidance only.