Pick an embedding model by benchmarking three candidates on 100 of your own labelled queries, not by leaderboard rank. The MTEB authors evaluated 33 models and found no method dominated across all tasks. Rank tells you a model is good at something. It cannot tell you it is good at yours.
- 0Methods that dominated across all tasks in the MTEB study
- 8Distinct embedding task types MTEB spans
- 58Datasets covered by the benchmark
- 112Languages represented across those datasets
- 33Models benchmarked in the original MTEB paper
The assumption worth correcting first: that picking an embedding model is a research task with a right answer somebody has already published. It is a measurement task, and the measurement has to happen on your data.
Muennighoff, Tazi, Magne and Reimers built the benchmark most teams now cite, then reported the opposite of what most teams take from it. Their conclusion was that the field “has yet to converge on a universal text embedding method.”
Why does the top of the leaderboard keep letting teams down?
Because a leaderboard reports an average across tasks, and you do not run an average. You run one task, on one kind of text, with one definition of relevance.
MTEB spans 8 task types across 58 datasets and 112 languages. A model that wins overall may have won on clustering and summarisation while placing mid-table on retrieval. If retrieval is your entire product, that average actively misled you.
Figure 1 — What the leaderboard measures vs what you need
The leaderboard question
“Which model scores highest averaged over 8 task types, 58 public datasets and 112 languages?”
Answered on general web and academic text, with relevance defined by dataset authors you have never met.
A useful shortlist
Your question
“Which model puts the right internal document in the top 5 for the queries our users actually type?”
Answered on your jargon, your abbreviations, your document lengths, your definition of relevant.
The actual decision
Use the left to produce the right. Take the top three to five models on the retrieval slice, then let your own eval pick the winner. The leaderboard narrows the field; it does not close it.
What actually determines the right model?
Six properties, and only one of them appears on a leaderboard.
| Property | Why it decides the outcome | How to check it |
|---|---|---|
| Domain fit | Legal, clinical and code text behave nothing like web prose | Your own eval set. Nothing else substitutes. |
| Max sequence length | Silent truncation destroys recall with no error raised | Model card, then measure your token length distribution |
| Dimensionality | Drives index memory, query latency and storage cost | Arithmetic, shown further down |
| Asymmetric support | Short queries against long documents need query and passage handled differently | Check whether the model expects prefixes or instructions |
| Language coverage | Multilingual averages hide weak individual languages | Evaluate per language, never pooled |
| Re-embedding cost | Switching later means re-embedding the entire corpus | Corpus size multiplied by per-token cost |
Max sequence length is the one that bites hardest, because exceeding it usually fails silently. The model truncates, embeds the fragment it kept, and returns a perfectly valid vector representing half your document.
How do you build a domain eval set in an afternoon?
This is the step teams skip because it sounds like a week of work. It is about four hours.
- Pull 100 real queries from logs. Real ones, including the malformed and the terse. Invented queries are always cleaner than reality and will flatter every model equally.
- For each query, find the document that should win. A domain expert doing this at roughly two minutes each finishes in half a day.
- Mark acceptable alternates. Retrieval is rarely one-right-answer. Recording two or three acceptable documents prevents you from punishing a model for being reasonable.
- Keep 20 queries sealed. You will tune chunking and prompts against the other 80 and quietly overfit them.
- Record why each match is correct. One line. When two models disagree six months later, this note is what settles it.
Watch for this
Do not let a language model generate your eval queries from your documents. It writes queries containing the document’s own vocabulary, which makes every embedding model look excellent and tells you nothing. Real users do not know your terminology, and that gap is precisely what you are trying to measure.
Which retrieval metrics should you track?
Three, and they answer different questions.
| Metric | Answers | Track it when |
|---|---|---|
| Recall@k | Did the right document make the cut at all? | A reranker or an LLM reads all k results |
| MRR | How high did the first correct result land? | Users see a ranked list and click |
| nDCG@k | Is the whole ordering sensible, with graded relevance? | Some results are partially right |
For most RAG systems, Recall@10 is the metric that predicts answer quality, because a reranker or the generating model sees everything retrieved. Precision at rank 1 matters far less than teams assume when a language model reads all ten.
Track cost and latency alongside quality in the same table. A model two points better on Recall@10 at four times the embedding cost is a decision, not a win.
How much do dimensions really cost you?
Dimensionality is where quality and infrastructure trade against each other, and almost nobody runs the arithmetic before committing.
Worked example — 5 million chunks
Raw vector storage at 4 bytes per float32 dimension, before index overhead, replicas or metadata. Structural arithmetic, not vendor pricing.
| Dimensions | Bytes per vector | 5M chunks, raw |
|---|---|---|
| 384 | 1,536 | ~7.7 GB |
| 768 | 3,072 | ~15.4 GB |
| 1,536 | 6,144 | ~30.7 GB |
| 3,072 | 12,288 | ~61.4 GB |
Crossing from 15 GB to 61 GB frequently means a larger instance class, and that step change usually costs more than the embedding calls ever did.
There is a way out of the trade. Matryoshka Representation Learning (Kusupati et al., arXiv:2205.13147) trains embeddings so that leading dimensions carry most of the information, letting you truncate a 3,072-dimension vector to 768 and keep much of the quality.
If a candidate model supports it, evaluate the truncated version too. Teams routinely pay for full-width vectors that a truncated variant would have matched on their own eval.
The seven-step selection protocol
Figure 2 — Selection flow, cheapest elimination first
Why constraints come first: a model you cannot legally deploy or that truncates your documents is disqualified regardless of score. Eliminating on hard constraints before running any benchmark saves most of the work.
What about multilingual and specialist text?
MTEB covers 112 languages, which sounds reassuring and is easy to misread. Coverage means a language appeared, not that performance in it was good.
Evaluate each language you actually serve on its own eval set. A pooled multilingual score can stay respectable while one language you depend on performs badly, and pooling is exactly what hides it.
Specialist domains deserve the same suspicion. Code, clinical notes and legal text each use vocabulary where general-purpose semantic similarity misfires: two clauses differing by one negation are near-identical to a general model and opposite in meaning to a lawyer.
Where teams go wrong
Changing two things at once
Swapping the embedding model and the chunking strategy in the same experiment produces a number you cannot attribute. Chunking often moves retrieval quality more than the model does, so the confound usually favours the wrong conclusion.
Fix chunking first with your current model. Then compare models on fixed chunks.
Ignoring the query-document asymmetry
Many retrieval models expect queries and passages to be encoded differently, often through a required prefix or instruction. Omitting it silently degrades results, and the model gives no warning.
Read the model card before concluding a model is weak. A meaningful share of “this model underperformed” reports are really this mistake.
Treating the choice as permanent
Switching models means re-embedding everything, so teams treat selection as irreversible and over-deliberate. The better response is to make switching cheap: keep the embedding step behind an interface, store the model name and version alongside every vector, and script the re-embedding job before you need it.
What do experienced teams do differently?
They keep the eval set alive.
A hundred labelled queries built once and never touched decays as the product changes. Teams that get durable value add ten queries a month from real failures, so the eval grows into a regression suite that catches degradation from any change, not just a model swap.
They also test hybrid retrieval before upgrading the model. Combining dense embeddings with keyword search frequently beats a better embedding model on its own, particularly for queries containing identifiers, product codes or exact names that semantic similarity handles poorly.
A short glossary
- MTEB
- The Massive Text Embedding Benchmark, spanning 8 task types, 58 datasets and 112 languages.
- Recall@k
- The share of queries where a correct document appears anywhere in the top k results.
- MRR
- Mean reciprocal rank: the average of one divided by the position of the first correct result.
- Asymmetric retrieval
- Matching short queries against long passages, which many models handle with distinct query and document encodings.
- Matryoshka embeddings
- Embeddings trained so leading dimensions carry most information, allowing truncation to smaller sizes with limited quality loss.
Where this is heading
A judgment rather than a finding: embedding model choice is going to matter less each year, and the eval set you build will matter more.
The top models cluster tighter every release, and truncatable representations are removing the dimension trade-off that used to force real architectural commitment. What does not commoditise is knowing what good retrieval looks like on your corpus.
The counter-case is worth stating. Domain-specialised embeddings for code, biomedicine and law keep beating generalists in their niches, and if you work in one of those, the gap is not closing on the same schedule. Convergence at the top of a general leaderboard says little about a specialist domain.
Key takeaways
- The MTEB authors benchmarked 33 models and found no method dominated across all tasks, so leaderboard rank cannot decide your model.
- MTEB spans 8 task types, 58 datasets and 112 languages; an overall average can hide weak retrieval performance.
- A hundred real labelled queries from your logs takes about an afternoon and outweighs every public benchmark.
- Recall@10 predicts RAG answer quality better than precision at rank 1, because the generating model reads everything retrieved.
- Exceeding max sequence length truncates silently, producing valid vectors for partial documents.
- Dimensions drive index cost: 5 million chunks is roughly 7.7 GB at 384 dimensions and 61.4 GB at 3,072.
- Fix chunking before comparing models, or the confound will point you at the wrong conclusion.
Frequently asked questions
Should I just use the top model on the MTEB leaderboard?
Use it to build a shortlist, not to make the decision. The benchmark’s own authors reported that no method dominated across all tasks, and the headline figure averages 8 task types. Take the leaders on the retrieval slice, then let your own labelled queries pick between them.
How many queries do I need in an evaluation set?
A hundred real queries is enough to separate candidate models reliably, with 80 for iteration and 20 sealed for final confirmation. Below roughly 50 the noise between models exceeds the difference you are trying to measure, and results stop being trustworthy.
Do more dimensions mean better retrieval?
Not reliably, and they always cost more. Dimensions drive storage, memory and query latency, with 5 million chunks running roughly 7.7 GB at 384 dimensions against 61.4 GB at 3,072. Test a truncated variant before paying for full width.
What happens if my documents exceed the max sequence length?
The model truncates and embeds only what fits, usually without any error. You get a valid vector representing a fragment, and recall degrades in a way that looks like poor model quality. Measure your token length distribution before selecting, not after.
Can I mix embedding models in one index?
No. Vectors from different models occupy different spaces, so distances between them are meaningless even when dimensions happen to match. Switching models requires re-embedding the whole corpus, which is why the model name and version belong in your index metadata.
Is a fine-tuned embedding model worth it?
Sometimes, but try hybrid search and reranking first, since both are cheaper and often close the gap. Fine-tuning pays off mainly for genuinely specialist vocabulary where general semantic similarity misfires, such as legal clauses, clinical notes or code identifiers.
Why did my model score worse than the benchmark suggested?
The most common cause is skipping a required query or passage prefix, since many retrieval models encode the two asymmetrically. Other frequent causes are silent truncation and comparing models across different chunking strategies. Check the model card before concluding the model is weak.
Should I evaluate multilingual performance as one number?
No. Pooled multilingual scores hide weak individual languages, and coverage of 112 languages in a benchmark means those languages appeared, not that performance in each was strong. Build a separate eval set for every language you actually serve.
References
- Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. arXiv:2210.07316, submitted 13 October 2022, revised 19 March 2023.
- Embeddings Benchmark. MTEB open-source code and public leaderboard.
- Kusupati, A., et al. Matryoshka Representation Learning. arXiv:2205.13147.
- Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv:2104.08663.
