skLEP — Slovak Natural Language Understanding Benchmark

skLEP is the first comprehensive benchmark designed specifically for evaluating Slovak natural language understanding (NLU) models. It covers seven tasks across three groups: token-level classification, sentence-pair tasks, and document-level classification. Created as the Slovak equivalent of the GLUE benchmark, it fills a long-standing gap in the systematic evaluation of language models for Slovak.

Task: Natural Language Understanding
Metric: F1, RER
Modality: Text
Updated: June 23, 2026

Articles and publications

skLEP: A Slovak General Language Understanding Benchmark (ACL 2025)
skLEP publication page (KInIT)

Metric notes

F1: Macro-averaged F1 is the primary metric across most tasks. It combines precision and recall and averages them across all classes regardless of size. Higher is better.
RER: Relative Error Reduction expresses the relative decrease in error rate compared to the SlovakBERT baseline model. Higher means greater improvement over the baseline.

Models are ranked by macro-averaged F1 across all seven NLU tasks. RER (Relative Error Reduction) expresses how much a model reduces the error rate relative to the SlovakBERT baseline — a positive RER means the model outperforms it. STS is the only task scored by Spearman correlation rather than F1; it is converted to the 0–100 scale for aggregation.

Token-Level Tasks

Universal Dependencies (UD): The Universal Dependencies task (Nivre et al., 2020) is a community initiative aimed at creating and expanding syntactically annotated corpora (treebanks) for more than 100 languages using a unified annotation scheme. It provides information on parts of speech (POS), lemmas, syntactic dependencies, arguments, and modifiers across more than 200 datasets. Within skLEP we use the POS subset from the Slovak Dependency Treebank (Gajdošová et al., 2016) available in UD.
Universal NER (UNER): The Universal NER task (Mayhew et al., 2024) provides high-quality, linguistically consistent annotations for named entity recognition (NER) in a multilingual setting. The Slovak portion (also from the Slovak Dependency Treebank) contains manually annotated entities of type person (PER), organisation (ORG), and location (LOC).
WikiGoldSK (WGSK): WikiGoldSK (Suba et al., 2023) is a manually annotated NER dataset for Slovak that addresses the limitations of so-called "silver-standard" datasets. The data come from Slovak Wikipedia and were annotated according to guidelines inspired by the BSNLP-2017 shared task (Piskorski et al., 2017). The dataset uses the CoNLL-2003 tagset and extends it with a MISC category (e.g. films, awards, events, or media).

Sentence-Pair Tasks

Recognizing Textual Entailment (RTE): The RTE task originates from the GLUE benchmark (Wang, 2018) and combines several datasets from entailment challenges into a binary classification task (entailment vs. not entailment). Because no Slovak version existed, the dataset was translated from English and subsequently revised by a native speaker. The test set was partially re-annotated manually, which is why it differs from the original (1,660 Slovak vs. 3,000 English examples).
Natural Language Inference (NLI): The NLI task evaluates a model's ability to determine the relationship between two sentences: entailment, contradiction, or neutral. Instead of the MNLI dataset (used in GLUE) we use XNLI (Conneau et al., 2018). The Slovak version was created via a translation pipeline with subsequent review by a native speaker.
Semantic Textual Similarity (STS): The STS task measures the semantic similarity of two sentences on a scale from 0 (no similarity) to 5 (identical meaning). Because no Slovak dataset existed, the data were translated and subsequently revised while preserving the original gold labels.

Document-Level Tasks

Hate Speech Classification (HS): The dataset originates from the Slovak Hate Speech and Offensive Language Database and is designed to detect hateful and offensive content on social media. Posts are labelled in binary fashion (1 = hateful/offensive, 0 = other). The data were collected by scraping and cleaned using text clustering.
Sentiment Analysis (SA): The sentiment analysis dataset was originally created as Reviews3 (Pecar et al., 2019) and later revised (Gurgurov et al., 2025). It contains Slovak customer reviews annotated as positive, negative, or neutral. In the version used in skLEP, neutral examples were removed and the task was simplified to binary classification.
Question Answering (SK-QuAD): SK-QuAD (Hládek et al., 2023) is the first manually annotated question-answering dataset for Slovak, compatible with SQuAD v2.0. It contains more than 91,000 question–answer pairs, including both answerable and unanswerable questions.