Chunking & Processing

Overview

Chunking is the process of breaking down large documents into smaller, semantically meaningful pieces that can be embedded and retrieved effectively. Poor chunking is one of the most common yet overlooked causes of RAG failures—it directly impacts retrieval quality, context coherence, and answer accuracy.

Why Chunking Matters

Effective chunking ensures:

Semantic coherence - Each chunk contains complete, meaningful information
Optimal retrieval - Relevant information can be found and returned
Context preservation - Important relationships and structure are maintained
Efficient embeddings - Chunks fit within token limits while maximizing information density

Poor chunking leads to:

Fragmented context - Critical information split across chunks
Lost structure - Formatting, tables, and code blocks broken apart
Retrieval failures - Relevant information exists but can't be found
Incomplete answers - Agents receive partial context and fill gaps with hallucinations

Common Chunking Challenges

Size Issues

Chunks too small - Context loss, fragmented information, excessive chunks
Chunks too large - Poor retrieval granularity, exceeded token limits
Inconsistent sizing - Variable quality across your knowledge base

Structure Preservation

Code blocks split incorrectly - Broken syntax, incomplete examples
Tables breaking across chunks - Lost relationships between rows/columns
Nested lists fragmented - Hierarchy and context destroyed
Footnotes separated - References disconnected from content

Format Handling

Markdown formatting lost - Headers, emphasis, and structure stripped
Mathematical notation corrupted - LaTeX and formulas broken
Multi-column layouts - Spatial relationships lost in conversion
HTML to text problems - Semantic meaning lost in conversion

Solutions in This Section

Browse these guides to optimize your chunking strategy:

Chunking Strategies

Different content types require different chunking approaches:

By Content Type

Content Type	Recommended Strategy	Chunk Size
Documentation	Semantic splitting by sections	500-1000 tokens
Code	Function/class boundaries	200-800 tokens
Conversations	Message groupings	300-600 tokens
Technical papers	Paragraph-aware splitting	600-1200 tokens
FAQs	Question-answer pairs	100-400 tokens

By Goal

Maximum recall - Smaller chunks (200-400 tokens) with more overlap
Maximum context - Larger chunks (800-1200 tokens) with less fragmentation
Balanced approach - Medium chunks (400-800 tokens) with 10-20% overlap

Best Practices

Respect document structure - Use headers, sections, and natural boundaries
Preserve semantic units - Don't split sentences, code blocks, or tables
Add metadata - Include source, section, and context in chunk metadata
Test retrieval quality - Measure how well chunking supports finding answers
Use overlap strategically - Help bridge context between chunks (10-20%)
Document-type specific strategies - Code needs different handling than prose
Monitor and iterate - Track chunk-level performance and adjust

Impact on RAG Performance

Chunking quality directly affects:

Metric	Impact of Good Chunking	Impact of Bad Chunking
Retrieval Precision	Relevant chunks returned	Irrelevant fragments retrieved
Retrieval Recall	Complete information found	Key details missed
Context Quality	Coherent, complete context	Fragmented, incomplete context
Answer Accuracy	Correct, well-supported answers	Hallucinations to fill gaps
Citation Quality	Clean source attribution	Unclear or broken references
Embedding Quality	Meaningful semantic vectors	Noisy, incoherent embeddings

Quick Diagnostics

Signs your chunking needs work:

✗ Agents give partial answers that feel "cut off"
✗ Retrieved chunks don't contain complete thoughts
✗ Code examples are broken or incomplete
✗ Tables appear without headers or context
✗ References to "see above" or "as mentioned" without the context
✗ High retrieval count but low answer quality

Signs your chunking is working well:

✓ Retrieved chunks are self-contained and meaningful
✓ Agents provide complete, coherent answers
✓ Citations point to relevant, complete content
✓ Code examples are syntactically complete
✓ Tables preserve their structure and relationships

Advanced Considerations

Semantic Chunking

Move beyond fixed token limits to:

Split at semantic boundaries (topic shifts, section changes)
Use embedding similarity to detect natural breakpoints
Preserve logical units (arguments, explanations, examples)

Hierarchical Chunking

Create multi-level chunk hierarchies:

Document level - Overview and metadata
Section level - Major topics and themes
Paragraph level - Specific details and facts

This enables coarse-to-fine retrieval strategies.

Dynamic Chunking

Adjust chunk size based on:

Document type and structure
Content density and complexity
Retrieval performance metrics
User query patterns

Remember: Perfect chunking is content-specific. What works for technical docs may not work for conversational data. Test, measure, and iterate based on your specific use case.

Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the ask query parameter:

GET /dev/rag-scenarios-and-solutions/chunking.md?ask=<question>

The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.

Key Takeaways