Quality leader
Claude Opus 5 leads overall at 93.0.
Snapshot · 2026-10-09-rag-cz-v1
An LLM benchmark for production-grade RAG pipelines.
LLM Compass helps engineering teams choose a model for real RAG work: document intake, extraction, planning, reranking and final answer synthesis. We measure quality, cost, latency, token efficiency and independent judge agreement instead of reducing the decision to one marketing score.
Model Universe
x: tokens · y: average quality · color: author
Premium judges sharpen the finalist comparison, while the broader panel keeps the full longlist comparable and cost-sustainable.
Premium judges GPT-5.5 and Claude Sonnet 4.6 are used only for High Priority LLM models to control benchmark cost.
RAG pipeline · P01 → P07
Seven benchmark steps, scored independently.
P01
Document intake
Classifies document type, language and structure before the rest of the RAG pipeline runs.
Mean 86.3 · hardest rank 3
P02
Structured extraction
Extracts key fields, entities and citations into a stable structured output.
Mean 85.1 · hardest rank 2
P03
Chunk enrichment
Adds metadata, context and retrieval signals to chunks before search and ranking.
Mean 88 · hardest rank 6
P04
Query routing
Chooses the retrieval strategy and sources that best match the user query.
Mean 84.9 · hardest rank 1
P05
Multi-hop plan
Plans multi-step evidence gathering across documents, entities and dependencies.
Mean 87.8 · hardest rank 5
P06
Rerank and conflicts
Reranks evidence, compresses context and tracks conflicts between retrieved sources.
Mean 86.5 · hardest rank 4
P07
Grounded synthesis
Builds the final answer from evidence, citations and explicit uncertainty handling.
Mean 92 · hardest rank 7
Top quality leaders
01Claude Opus 5maxout8.2kreas.4.2kcost$7.3493.002Claude Opus 5mediumout7.4kreas.334cost$5.7392.803Claude Opus 5nothinkout8.4kreas.0cost$5.9792.804GPT-5.6 Solmaxout4.5kreas.6.2kcost$11.5792.605Grok 4.5highout3.6kreas.756cost$0.82292.506GPT-5.6 Solmediumout4.2kreas.815cost$8.2292.407GPT-5.6 Lunamaxout4.8kreas.9.9kcost$1.6392.408GPT-5.6 Terramaxout4.6kreas.5.7kcost$3.1692.409Kimi K3maxout5.4kreas.8.1kcost$3.9892.210Grok 4.5minimalout3.8kreas.86cost$0.83292.2
01Kimi K3maxout5.4kreas.8.1kcost$3.9892.202Kimi K3highout5.0kreas.810cost$2.1891.903GLM 5.2enabledout4.2kreas.6.9kcost$1.2891.604DeepSeek V4 Flash 0731highout4.6kreas.14.2kcost$0.18091.505Kimi K3lowout4.2kreas.86cost$1.8791.306DeepSeek V4 Flash 0731maxout4.7kreas.16.4kcost$0.19791.207MiniMax M3adaptiveout5.9kreas.4.1kcost$0.60591.208Hy4 Previewenabledout6.6kreas.22.1kcost$1.0790.809MiniMax M3nothinkout5.7kreas.-cost$0.46090.610MiniMax M3UD-IQ3_XXSenabledout5.3kreas.3.1kcost$0.15790.6
01Claude Opus 5maxout8.2kreas.4.2kcost$7.3493.002Claude Opus 5mediumout7.4kreas.334cost$5.7392.803Claude Opus 5nothinkout8.4kreas.0cost$5.9792.804GPT-5.6 Solmaxout4.5kreas.6.2kcost$11.5792.605Grok 4.5highout3.6kreas.756cost$0.82292.506GPT-5.6 Solmediumout4.2kreas.815cost$8.2292.407GPT-5.6 Lunamaxout4.8kreas.9.9kcost$1.6392.408GPT-5.6 Terramaxout4.6kreas.5.7kcost$3.1692.409Grok 4.5minimalout3.8kreas.86cost$0.83292.210GPT-5.5maxout4.7kreas.1.2kcost$3.9792.0
Top findings · auto-generated from snapshot
What this snapshot says
Efficient
Gemini 3.1 Flash Lite GA sits on the Pareto frontier: 80.2 at 1.4k tokens.
Methodology
The premium panel has 3 judges and evaluates High Priority models only for cost control; secondary models are judged by 11 full-panel judges.
Trend
Reasoning-mode models change by -0.7 points between R1 and R2; non-thinking models change by -0.3 points.