SkMTEB: The First Benchmark for Slovak Embeddings
Title: SkMTEB: The First Benchmark for Slovak Embeddings Perex: SkMTEB — the first comprehensive benchmark for text embeddings in Slovak — has been accepted at ACL 2026, one of the most prestigious international conferences in natural language processing. The benchmark covers 31 datasets across 7 task types and establishes a standard comparable to MTEB for English. The work is led by a team from the Kempelen Institute of Intelligent Technologies (KInIT), the Technical University of Košice (TUKE), and Comenius University Bratislava (UK), together with other partners from the Slovak AI ecosystem.
SkMTEB is an important milestone not only for the benchmark itself, but for Slovak NLP as a whole. Until now, there was no reliable way to compare which models truly understand Slovak — SkMTEB addresses this gap and gives developers, companies, and public institutions a standardized measuring tool.
The benchmark covers 31 datasets across 7 task types including retrieval, semantic similarity, classification, clustering, and more. For comparison: the multilingual MMTEB covered Slovak with only 8 datasets — SkMTEB represents a nearly fourfold expansion.
The research also introduces two new models — e5-sk-small (45M parameters) and e5-sk-large (365M) — developed through vocabulary trimming and fine-tuning on Slovak data. Despite their significantly reduced size, both models achieve results comparable to proprietary API models. They run locally on standard hardware without sending data to third parties — which is of particular practical importance for public administration, healthcare, and companies working with sensitive data. The smaller model is lightweight enough to run directly in a web browser.
Benchmark, modely aj kód sú dostupné open-source:
🤗 Models