SkMTEB — Slovak Text Embedding Benchmark
SkMTEB is the first comprehensive benchmark for evaluating text embedding models on Slovak, covering 31 datasets across 7 task types. It is the Slovak equivalent of the international MTEB benchmark, which has become the standard for comparing embedding models across languages. Text embeddings underpin modern AI applications — semantic search, RAG pipelines, and document classification — and SkMTEB enables their systematic evaluation for Slovak for the first time.
- Task
- Text Embedding
- Metric
- Avg Score
- Modality
- Text
- Updated
- June 23, 2026
Articles and publications
Metric notes
- Avg Score
- Average score is computed as a macro-average: results are first averaged within each task type, then across all 7 task types. Higher is better. Each task type uses its own primary metric.
Models are ranked by their average score across all 31 datasets and 7 task types (Avg Score). Each task type uses its own primary metric — such as nDCG@10 for retrieval or Spearman correlation for STS — which are then macro-averaged first within, then across task types. Higher is better.
Info
- Task types
- The benchmark covers seven task types: retrieval (finding relevant documents for a query), classification (assigning labels to text), clustering (grouping semantically similar texts), reranking (ordering results by relevance), STS — semantic textual similarity, bitext mining (finding translation pairs across languages), and pair classification (predicting the relationship between two texts).
- Models
- As part of this work, two open-source Slovak embedding models were developed — e5-sk-small (45M parameters) and e5-sk-large (365M parameters) — by applying vocabulary trimming and fine-tuning to Multilingual E5. Despite size reductions of up to 62%, both models achieve performance competitive with proprietary APIs and are suitable for local deployment.