RetrievalBench · Validation release
Stop tuning retrieval by vibes.
Know whether your RAG retrieval change actually improved before you ship it.
Compare retrieval configurations head-to-head, see exactly which queries improved or regressed, and understand whether the extra cost and latency are worth it.
Validation release — no payment will be collected today.
The problem
You changed your retrieval pipeline. Now what?
Before
- 512-token chunks
- dense retrieval
- top-k 5
After
- 800-token chunks
- hybrid retrieval
- reranker
- top-k 10
Is the new version actually better?
- Which queries improved?
- Which queries got worse?
- Is the extra cost and latency worth it?
Compare
Compare baseline vs candidate.
Run the same evaluation set against two retrieval configurations and read the difference instead of guessing it.
| Metric | Baseline | Candidate | Delta |
|---|---|---|---|
| Recall@5 | 73.8% | 82.1% | +8.3 |
| MRR | 69.4% | 75.2% | +5.8 |
| nDCG@10 | 76.1% | 80.9% | +4.8 |
| P95 latency | 190 ms | 279 ms | +47% |
| Cost / query | $0.003 | $0.006 | +100% |
Verdict
Improvement with regressions
Retrieval quality improved overall, but 7 queries regressed. Most regressions occurred in exact terminology and multi-hop queries.
18
Improved
7
Regressed
59
Unchanged
Diagnose
See what actually got worse.
Average scores hide the queries you broke. Every regression is listed with enough context to act on it.
Query #23
What is the refund policy for annual subscriptions?
baselineRelevant document — rank #2✓candidateRelevant document — rank #11✕Possible issue
Exact-term result was demoted after reranking.
Query #41
Which regions support data residency for enterprise plans?
baselineNo relevant document retrieved✕candidateNo relevant document retrieved✕Possible issue
This may be a corpus coverage or ingestion problem rather than a ranking problem.
Scope
What you get.
- Baseline vs candidate retrieval comparison
- Recall@K, MRR, and nDCG
- Query-level regression detection
- Improved / regressed / unchanged breakdown
- Latency comparison
- Cost comparison
- Failure-oriented report
- Markdown / HTML report export
- Simple benchmark dataset format
Fit
Built for teams that already have a RAG system.
This is for you if
- You have a RAG pipeline running.
- You already have evaluation or test queries.
- You actively change retrieval configuration.
- You want a repeatable comparison.
- You don't want to inspect results manually every time.
Probably not for you if
- You're still building your first RAG demo.
- You don't have any evaluation queries yet.
- You need a complete observability platform.
- You're looking for enterprise monitoring and tracing.
Pricing
Early Access
One-time early access pricing for the validation release.
Validation release — no payment will be collected today.