Temperature Setting Issues

The Problem

Incorrect temperature settings cause either robotic, repetitive responses or creative but factually incorrect answers, degrading RAG quality.

Symptoms

• ❌ Identical phrasing for similar queries
• ❌ Overly formal/robotic responses
• ❌ Creative but wrong information added
• ❌ Inconsistent response style
• ❌ Cannot balance accuracy vs natural language

Real-World Example

Temperature 0.0 (deterministic):
Query 1: "How to authenticate?"
Response: "To authenticate, use the API key in the Authorization header."

Query 2: "What's the auth method?"
Response: "To authenticate, use the API key in the Authorization header."

Exact same wording → robotic

Temperature 1.5 (creative):
Query: "API rate limit?"
Response: "The API implements a sophisticated adaptive rate limiting
system that adjusts based on your usage patterns and account tier,
typically allowing between 800-1200 requests per hour..."

Added details not in context → hallucination

---

Deep Technical Analysis

Temperature Parameter Mechanics

Controls randomness in token selection:

How It Works:

LLM outputs probability distribution over vocabulary:
→ "authenticate": 0.45 (highest)
→ "verify": 0.25
→ "authorize": 0.15
→ "confirm": 0.10
→ ...

Temperature = 0: Always pick highest (deterministic)
→ "authenticate" every time

Temperature = 1: Sample from distribution
→ "authenticate" 45% of time
→ "verify" 25% of time

Temperature = 2: Flatten distribution
→ More uniform sampling
→ Higher diversity

RAG-Specific Considerations

RAG needs factual accuracy:

Low Temperature (0.0-0.3) Benefits:

Pros:
+ Consistent responses
+ Factually grounded
+ Reproducible
+ Less hallucination risk

Cons:
- Repetitive phrasing
- Robotic tone
- Lacks conversational flow
- Same intro every time

High Temperature (0.8-1.5) Risks:

Pros:
+ Natural language
+ Varied phrasing
+ Conversational tone

Cons:
- Creative additions (hallucination)
- Paraphrases may drift from facts
- Less controllable
- Inconsistent quality

The Sweet Spot

Balancing accuracy and fluency:

Temperature 0.3-0.5:

Optimal for RAG:
→ Mostly follows context
→ Some variation in phrasing
→ Natural but grounded
→ Minimal hallucination

Testing shows:
→ 0.3: Very safe, slightly robotic
→ 0.5: Good balance
→ 0.7: More creative, higher risk

Context-Dependent Adjustment:

Factual queries: temperature=0.2
→ "What is the rate limit?"
→ Exact fact needed

Explanatory queries: temperature=0.5
→ "How does authentication work?"
→ More natural explanation acceptable

Creative queries: temperature=0.7
→ "Explain this concept to a beginner"
→ Pedagogical flexibility useful

Top-P (Nucleus) Sampling

Alternative to temperature:

Top-P Mechanism:

Instead of temperature, use top_p:
→ Sample from smallest set of tokens where cumulative
  probability ≥ p

top_p=0.9:
→ "authenticate" (0.45) + "verify" (0.25) + "authorize" (0.15) 
  + others = 0.90
→ Only consider these tokens
→ Ignore very low probability options

More controlled than high temperature

Combined Settings:

temperature=0.7, top_p=0.9:
→ Soften probabilities (temperature)
→ But cap at top 90% (top_p)
→ Balanced creativity with safety

Recommended for RAG:
→ temperature=0.5
→ top_p=0.95

---

How to Solve

Set temperature=0.3-0.5 for RAG (factual grounding) + use top_p=0.9-0.95 for additional control + adjust per query type (lower for factual, higher for explanatory) + test with eval set to find optimal balance + never exceed 0.7 for production RAG. See Temperature Tuning.

---

Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the ask query parameter:

GET /dev/rag-scenarios-and-solutions/llm/temperature-tuning.md?ask=<question>

The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.