Skip to content

Benchmark

MTEB

Massive Text Embedding Benchmark, a standard suite for scoring text embedding models on retrieval, classification, clustering, reranking and other tasks.

Known aliases

  • Massive Text Embedding Benchmark
  • MTEB
  • MTEB Code

Current stories

buildOne report1 publisher

EmbeddingGemma 2 maps five modalities into one 768-dimension vector space

Google DeepMind released EmbeddingGemma 2, a 740M-parameter open model that runs on a phone and embeds text, code, images, video and audio in one space. Teams running a separate embedder per modality can consolidate on it if Google's reported benchmark numbers hold on their own data.

Publishers:dev.to

Reality

Evidence50
Adoption20
Hype gap+25
Incentives60
Confidence55
buildConfirmed13 publishers

Google's EmbeddingGemma 2 fits multimodal search into 567MB of phone RAM

Google released EmbeddingGemma 2, an Apache 2.0 model that maps text, code, images, video and audio into one embedding space with 740M parameters. Phone apps get offline cross-media search from one set of weights, within limits set by a shared 8K-token window and lossy vector truncation.

Perspective Coverage

17 publishers
Builder
Builder 59%
Operator
Operator 29%
Investor
Investor 12%

Reality

Evidence55
Adoption22
Hype gap+20
Incentives70
Confidence62
buildOne report1 publisher

Choosing the embedding model first locks the vec0 table to a fixed 768-dimension schema

A dev.to walkthrough replaces a managed vector store and a hosted embedding API with sqlite-vec and a local model on port 11434. Its schema declares float[768], so changing model dimension means re-embedding everything.

Publishers:dev.to

Reality

Evidence42
Adoption
Not scored
Hype gap+25
Incentives22
Confidence48
buildOne report1 publisher

Toast 1 claims MTEB parity with OpenAI. The cost math in the pitch is off by 1000x.

Mixedbread's embedding model is reported to edge text-embedding-3-large by half a point. The savings case rests on an arithmetic slip and pricing that does not exist yet.

Publishers:dev.to

Reality

Evidence16
Adoption8
Hype gap+68
Incentives72
Confidence62
buildOne report1 publisher

Debug the index before you swap the model: the RAG checklist works, the 73% does not

A dev.to postmortem argues most RAG failures happen in retrieval. The debugging order it recommends survives scrutiny. The headline percentage it leans on does not.

Publishers:dev.to

Reality

Evidence26
Adoption
Not scored
Hype gap+38
Incentives30
Confidence48