Benchmarks

SkMTEB is the first comprehensive benchmark for evaluating text embedding models on Slovak, covering 31 datasets across 7 task types. It is the Slovak equivalent of the international MTEB benchmark, which has become the standard for comparing embedding models across languages. Text embeddings underpin modern AI applications — semantic search, RAG pipelines, and document classification — and SkMTEB enables their systematic evaluation for Slovak for the first time.

This is the first dedicated Speech-to-Text (STT) benchmark for Slovak: 354 hours of curated audio across 22 models. Existing resources focus on English or multilingual datasets where Slovak is only a few percent of the data. This benchmark fills that gap, comparing models on Slovak data across different domains.

skLEP is the first comprehensive benchmark designed specifically for evaluating Slovak natural language understanding (NLU) models. It covers seven tasks across three groups: token-level classification, sentence-pair tasks, and document-level classification. Created as the Slovak equivalent of the GLUE benchmark, it fills a long-standing gap in the systematic evaluation of language models for Slovak.