Semantic Redundancy

The Problem

Multiple chunks express the same information with different wording, wasting context window space and diluting retrieval quality.

Symptoms

• ❌ Same fact repeated in different words
• ❌ AI response redundant across multiple chunks
• ❌ Context window filled with paraphrases
• ❌ Lower-quality chunks push out better ones
• ❌ Storage wasted on duplicate meanings

Real-World Example

Knowledge base chunks:
→ Chunk A: "The API rate limit is 1000 requests per hour"
→ Chunk B: "You can make up to 1000 API calls every 60 minutes"
→ Chunk C: "Hourly API limit: 1k req/hr"

All three say the same thing (semantic duplicates)

Query retrieves all three:
→ Wastes 3 chunk slots for 1 fact
→ Context window: 3000 tokens for same info
→ Could have retrieved other unique facts

AI response repeats:
"The rate limit is 1000/hour. You can make 1000 calls per 60 minutes..."

---

Deep Technical Analysis

Detection Challenges

High Semantic Similarity:

Cosine similarity between embeddings:
→ Chunk A vs B: 0.92 (very high)
→ Chunk A vs C: 0.88

Threshold: 0.85 = semantic duplicates
→ Flag for review/consolidation

Paraphrase Detection:

Different words, same meaning:
→ "authenticate" vs "log in"
→ "terminate" vs "cancel"
→ "purchase" vs "buy"

Embeddings capture semantic similarity:
→ High cosine score despite different words

Sources of Redundancy

Multi-Source Ingestion:

Same info from multiple sources:
→ Help Center article: "Rate limit is 1000/hour"
→ API docs: "1000 requests per hour allowed"
→ FAQ: "API calls limited to 1k/hour"

All three ingested → redundancy

Document Repetition:

Within single document:
→ Executive summary: "Rate limit: 1000/hour"
→ Details section: "The system enforces 1000 req/hr"
→ Troubleshooting: "If hitting 1000/hour limit..."

Concept repeated for emphasis/clarity
But: Creates redundancy in chunks

Deduplication Strategies

Clustering:

1. Embed all chunks
2. Cluster by semantic similarity (DBSCAN, K-means)
3. Within each cluster:
   - Select best representative
   - Archive or discard others

Reduces redundancy systematically

Representative Selection:

Within semantic cluster, choose:
→ Longest chunk (most comprehensive)
→ Or: Most recent
→ Or: Highest source authority

Example cluster:
→ Chunk A: 200 tokens, official docs
→ Chunk B: 100 tokens, community post
→ Select: Chunk A (authoritative + comprehensive)

Consolidation

Merge Semantically Similar:

Instead of three chunks:
"Rate limit is 1000 requests per hour"
"You can make 1000 API calls every 60 minutes"
"Hourly API limit: 1k req/hr"

Consolidated:
"The API rate limit is 1000 requests per hour (1k req/hr).
You can make up to 1000 API calls every 60 minutes."

Single comprehensive chunk

Cross-Reference:

If multiple sources say same thing:
→ Keep one chunk
→ Add metadata: sources: ["doc_A", "doc_B", "doc_C"]

Shows: Multiple sources confirm this fact
But: Store once

---

How to Solve

Run semantic similarity clustering (cosine > 0.85) to detect redundancy + select best representative from each cluster (most comprehensive/authoritative) + consolidate semantically identical chunks into single chunk + add source attribution metadata for merged chunks + periodically audit for new redundancy + prefer single comprehensive source over multiple paraphrases. See Semantic Deduplication.

---

Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the ask query parameter:

GET /dev/rag-scenarios-and-solutions/data-quality/semantic-redundancy.md?ask=<question>

The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.