SkMTEB — Slovak Text Embedding Benchmark

SkMTEB is the first comprehensive benchmark for evaluating text embedding models on Slovak, covering 31 datasets across 7 task types. It is the Slovak equivalent of the international MTEB benchmark, which has become the standard for comparing embedding models across languages. Text embeddings underpin modern AI applications — semantic search, RAG pipelines, and document classification — and SkMTEB enables their systematic evaluation for Slovak for the first time.

Task: Text Embedding
Metric: Avg Score
Modality: Text
Published: June 23, 2026
Updated: June 23, 2026
Models: 31
Dataset details in the catalog: SkMTEB
Links: Website
Paper
GitHub

Ranked by: All datasets (31)

Metric direction: higher is better
Published: June 23, 2026
Number of models: 31 models

Model size

1
77.49
2
77.23
3
75.10
4
75.07
5
74.70
6
74.53
7
74.43
8
74.25
9
73.70
10
72.58
11
72.54
12
72.44
13
72.39
14
71.76
15
71.26

Click a model name to highlight it across all metrics.

Full results table

Original benchmark results: 31 models, 9 columns. On mobile, scroll horizontally to view the table. Click an underlined column heading to sort the models in the chart above by that column’s values. Models with underlined names have their own page in the resource catalog.

SkMTEB — Slovak Text Embedding Benchmark: full results table
Model	Overall		By task type
Model
Small (<130M)
e5-sk-small45M	70.56	72.01	91.34	60.84	40.95	66.05	84.94	78.64	81.32
granite-embedding-107m-multilingual107M	65.81	67.31	85.85	56.04	40.80	60.88	78.10	70.93	78.54
static-similarity-mrl-multilingual-v1108M	58.51	60.90	87.98	48.87	17.47	63.66	70.76	59.35	78.22
multilingual-e5-small118M	70.32	71.78	91.29	59.81	41.09	64.66	85.13	79.21	81.25
paraphrase-multilingual-MiniLM-L12-v2118M	67.86	69.16	94.91	60.83	37.44	67.82	73.46	66.03	83.62
slovakbert-skquad-mnlr125M	67.49	68.91	85.07	62.49	38.07	65.13	76.91	72.82	81.90
slovakbert-sts-stsb125M	63.44	65.37	81.75	63.05	31.82	68.90	67.15	59.20	85.69
sturovec-base125M	68.99	70.13	86.97	62.19	41.55	63.27	80.66	76.54	79.73
Base (≥130M, <350M)
nomic-embed-text-v1.5137M	51.52	55.03	46.64	51.40	28.77	61.98	68.60	55.39	72.43
granite-embedding-278m-multilingual278M	67.69	69.09	87.56	57.81	42.41	61.40	81.25	73.92	79.30
multilingual-e5-base278M	72.39	73.57	94.76	62.35	40.91	64.10	85.71	83.57	83.59
paraphrase-multilingual-mpnet-base-v2278M	70.34	71.67	96.23	64.05	38.95	68.21	76.79	70.03	87.44
gte-multilingual-base305M	71.76	73.26	94.36	61.70	39.28	67.24	83.04	81.63	85.57
embeddinggemma-300m308M	69.25	70.56	83.89	62.08	43.76	65.54	81.00	78.40	79.28
nomic-embed-text-v2-moe330M	72.58	73.84	90.03	63.98	43.53	64.19	85.15	85.37	84.60
Large (≥350M)
e5-sk-large365M	74.70	75.88	96.39	66.34	41.43	67.32	87.81	85.60	86.25
LaBSE471M	66.44	67.52	97.48	58.73	34.28	64.39	73.52	63.39	80.85
multilingual-e5-large560M	74.25	75.49	96.29	65.34	40.35	66.78	87.96	85.80	85.90
multilingual-e5-large-instruct560M	77.49	78.44	97.09	70.28	49.69	70.55	86.49	86.08	88.86
bge-m3568M	74.43	75.55	96.29	66.81	39.97	67.03	86.72	85.63	86.39
snowflake-arctic-embed-l-v2.0568M	72.54	73.63	93.46	63.40	40.41	63.11	87.11	85.17	82.76
jina-embeddings-v30.6B	75.10	76.20	96.43	66.89	44.63	66.44	83.44	85.77	89.82
Qwen3-Embedding-0.6B596M	70.53	71.80	90.12	62.84	44.28	64.46	82.02	75.73	83.14
jina-embeddings-v40.6B	72.44	73.87	90.49	62.99	44.65	64.43	85.25	83.64	84.65
Qwen3-Embedding-4B4B	73.70	74.96	94.21	64.26	44.22	65.82	86.29	84.11	85.82
Qwen3-Embedding-8B8B	74.53	75.75	94.43	65.94	43.36	66.65	87.04	86.27	86.54
API access
embed-v4.0	71.26	72.82	90.23	60.12	40.13	63.12	87.45	85.71	82.95
text-embedding-3-small	70.48	71.39	91.82	62.13	43.64	63.24	82.37	76.96	79.56
text-embedding-3-large	75.07	75.89	96.76	66.91	44.22	66.58	86.96	85.55	84.21
gemini-embedding-001	77.23	78.07	96.76	71.01	46.26	67.08	88.27	88.34	88.74
amazon-titan-embed-text-v2	67.24	69.21	84.98	59.26	33.39	62.78	83.99	77.43	82.67

Good to know

Models are ranked by their average score across all 31 datasets and 7 task types (Avg Score). Each task type uses its own primary metric — such as nDCG@10 for retrieval or Spearman correlation for STS — which are then macro-averaged first within, then across task types. Higher is better.
The results were published on SLAIH on June 23, 2026. Individual models may have been evaluated on different dates.
A dash means the model has no result for that task. It is not a zero.
If a model is missing from the table, no result for it is listed here. Its absence does not mean it performs worse.
These results apply to this benchmark and its metric. They do not measure the model’s overall quality.

Datasets and methodology

Metric notes

Avg Score: Average score is computed as a macro-average: results are first averaged within each task type, then across all 7 task types. Higher is better. Each task type uses its own primary metric.

Info

Task types: The benchmark covers seven task types: retrieval (finding relevant documents for a query), classification (assigning labels to text), clustering (grouping semantically similar texts), reranking (ordering results by relevance), STS — semantic textual similarity, bitext mining (finding translation pairs across languages), and pair classification (predicting the relationship between two texts).
Models: As part of this work, two open-source Slovak embedding models were developed — e5-sk-small (45M parameters) and e5-sk-large (365M parameters) — by applying vocabulary trimming and fine-tuning to Multilingual E5. Despite size reductions of up to 62%, both models achieve performance competitive with proprietary APIs and are suitable for local deployment.

SkMTEB — Slovak Text Embedding Benchmark: Datasets and methodology

Publications

Benchmark details — GitHub (SlovakNLP)
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation (ACL 2026)

Use the results

Data

Full results table with 31 rows, encoded in UTF-8.

Download CSV

Citation

When you use the results, cite the benchmark and its original publication.

SkMTEB — Slovak Text Embedding Benchmark. SLAIH – Slovak Language AI Hub, June 23, 2026. https://slaih.sk/en/benchmarks/skmteb
Benchmark details — GitHub. SlovakNLP. https://slovak-nlp.github.io/skmteb/