Wrong Answers from RAG

The Problem

Retrieved context is relevant but AI generates factually incorrect answers, misinterprets the context, or combines information incorrectly.

Symptoms

• ❌ Context has right info, answer is wrong
• ❌ Misreads numbers, dates, names
• ❌ Reverses cause and effect
• ❌ Combines facts from different contexts incorrectly
• ❌ Ignores critical qualifiers ("not", "except")

Real-World Example

Retrieved context:
"Enterprise plan costs $500/month. Basic plan costs $50/month."

User query: "How much is the basic plan?"

AI response: "The Basic plan costs $500 per month."

Problem: Confused Enterprise and Basic pricing
Context was correct, interpretation wrong

---

Deep Technical Analysis

Semantic Confusion

Similar Entity Names:

Context: "Product A supports 10 users. Product B supports 100 users."
Query: "How many users does Product B support?"
AI: "Product B supports 10 users."

LLM confused A and B (single character difference)

Negation Handling:

Context: "Feature X is NOT available in the free tier."
Query: "Is feature X in free tier?"
AI: "Yes, feature X is available in the free tier."

LLM missed "NOT" - critical negation ignored

Context Assembly Issues

Conflicting Chunks:

Chunk 1: "Rate limit is 100 requests/hour"
Chunk 2: "New rate limit (as of Jan 2024): 1000 requests/hour"

Both retrieved, but Chunk 1 ranked higher:
→ AI uses outdated information
→ Wrong answer despite correct info present

Partial Context:

Full doc: "API key authentication required for all endpoints except /health"

Retrieved chunk: "API key authentication required for all endpoints"
→ Critical exception cut off by chunking boundary
→ AI gives incomplete answer

LLM Reasoning Errors

Multi-Hop Reasoning:

Context:
- "UserA has access to ProjectX"
- "ProjectX contains DocumentY"

Query: "Can UserA access DocumentY?"

Requires inference:
UserA → ProjectX → DocumentY = Yes

LLM may fail to chain the reasoning:
→ "I don't have information about UserA accessing DocumentY"

Quantitative Errors:

Context: "Increased by 25% from previous quarter (1000 units to 1250)"

Query: "What was the increase?"

AI: "The increase was 250 units (25%)"
→ Should be 1250 units
→ Confuses percent with absolute value

---

How to Solve

Implement reranking to surface most relevant context first + use chain-of-thought prompting for multi-step reasoning + add explicit negation handling in prompts + apply answer validation (fact-checking against context) + use higher-quality models (GPT-4 over GPT-3.5) for complex interpretation + test with eval set of tricky questions. See Answer Quality.

---

Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the ask query parameter:

GET /dev/rag-scenarios-and-solutions/accuracy/wrong-answers.md?ask=<question>

The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.