Knowledge Base Drift

The Problem

Over time, accumulated updates, duplicates, and inconsistencies cause knowledge base quality to degrade, reducing answer accuracy.

Symptoms

• ❌ Duplicate content with slight variations
• ❌ Inconsistent terminology across docs
• ❌ Orphaned chunks from deleted docs
• ❌ Degrading retrieval quality over time
• ❌ Conflicting information proliferating

Real-World Example

Month 1: Clean knowledge base
→ 1,000 docs, well-organized

Month 12: Drifted knowledge base
→ 5,000 docs (5x growth)
→ 200 duplicates (re-ingested without dedup)
→ 500 orphaned chunks (source docs deleted)
→ Inconsistent terms ("log in" vs "sign in" vs "authenticate")

Retrieval quality:
→ Month 1: 90% accuracy
→ Month 12: 65% accuracy (degraded)

---

Deep Technical Analysis

Incremental Degradation

Duplicate Accumulation:

Document updated multiple times:
→ v1 ingested → chunks A, B, C
→ v2 updated → chunks D, E, F added
→ v1 chunks NOT removed

Result:
→ A, B, C (old) + D, E, F (new) coexist
→ Retrieves old info
→ Conflicting answers

Orphaned Data:

Source doc deleted from CMS:
→ Chunks remain in vector DB
→ No automatic cleanup
→ Stale data persists

Cites deleted/non-existent sources

Terminology Drift

Inconsistent Naming:

Early docs: "User authentication"
Later docs: "Login system"
Recent docs: "Identity management"

Same concept, different terms:
→ Retrieval fragmented
→ Misses related docs
→ Incomplete answers

Canonical Terms:

Solution: Maintain glossary
→ Standardize on "Authentication"
→ Map aliases: "login", "sign in", "auth"
→ Normalize at ingestion or query time

Index Fragmentation

Vector DB Performance:

After many updates/deletes:
→ Index structure fragmented
→ Search slower
→ Quality degrades

Requires:
→ Periodic reindexing
→ Compaction
→ Optimization

Data Quality Metrics

Staleness Detection:

Monitor per-chunk age:
→ Chunks not updated in 6+ months
→ Flag for review
→ Possibly obsolete

Automated alerts:
→ "Document X not updated since 2022"
→ Review/remove

Duplicate Detection:

Semantic similarity between chunks:
→ If cosine similarity > 0.95
→ Likely duplicate
→ Consolidate or remove

---

How to Solve

Implement version-aware ingestion (delete old chunks on update) + run periodic deduplication (detect semantic duplicates) + track chunk age and flag stale content + use canonical terminology (glossary + normalization) + schedule index compaction quarterly + monitor retrieval quality metrics (accuracy trend) + perform annual knowledge base audit/cleanup. See Knowledge Drift.

---

Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the ask query parameter:

GET /dev/rag-scenarios-and-solutions/accuracy/factual-drift.md?ask=<question>

The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.