For the taskUnderstand text
skLEP — Slovak Language Understanding Benchmark
skLEP is the first comprehensive benchmark designed specifically for evaluating Slovak natural language understanding (NLU) models. It covers nine tasks across three groups: token-level classification, sentence-pair tasks, and document-level classification. Created as the Slovak equivalent of the GLUE benchmark, it fills a long-standing gap in the systematic evaluation of language models for Slovak.
- Task
- Natural Language Understanding
- Metric
- F1, RER
- Modality
- Text
- Published
- June 23, 2026
- Updated
- June 23, 2026
- Models
- 16
- Dataset details in the catalog
- skLEP
- Contributors
Ranked by: Task average
- Metric direction
- higher is better
- Published
- Number of models
- 16 models
- 185.17
- 284.27
- 384.17
- 483.95
- 483.95
- 683.36
- 782.96
- 882.30
- 982.01
- 1081.16
- 1180.56
- 1278.75
- 1378.54
- 1478.48
- 1574.46
- 1672.82
Click a model name to highlight it across all metrics.
Full results table
Original benchmark results: 16 models, 20 columns. On mobile, scroll horizontally to view the table. Click an underlined column heading to sort the models in the chart above by that column’s values. Models with underlined names have their own page in the resource catalog.
| Model | Overall | Token level | Sentence pairs | Document level | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UD, POS tagging | UNER, named entities | WGSK, named entities | RTE, entailment | NLI, inference | STS, sentence similarity | HS, hate speech | SA, sentiment | QA, question answering | ||||||||||||
| 🇸🇰SlovakBERT | +0.00 | 83.95 | +0.00 | 98.04 | +0.00 | 81.92 | +0.00 | 92.14 | +0.00 | 65.20 | +0.00 | 82.75 | +0.00 | 83.18 | +0.00 | 80.31 | +0.00 | 97.63 | +0.00 | 74.36 |
| 🇸🇰HPLTBase | -1.29 | 82.96 | +9.71 | 98.23 | +13.05 | 84.28 | +16.50 | 93.44 | -24.12 | 56.81 | -11.83 | 80.71 | -9.01 | 81.66 | -7.68 | 78.80 | -1.23 | 97.60 | +3.04 | 75.14 |
| 🇸🇰FERNET-CCBase | +0.55 | 84.27 | -8.67 | 97.87 | +9.22 | 83.59 | +8.12 | 92.78 | +8.72 | 68.23 | -5.31 | 81.83 | +2.50 | 83.60 | -6.40 | 79.05 | -1.23 | 97.60 | -1.97 | 73.85 |
| 🌐DistilmBERTBase | -41.42 | 78.54 | -45.84 | 97.14 | -34.96 | 75.60 | -59.00 | 87.50 | -16.79 | 59.36 | -50.45 | 74.05 | -56.38 | 73.70 | -24.24 | 75.54 | -59.27 | 96.23 | -25.84 | 67.73 |
| 🌐MiniLML12-Base | -41.21 | 78.75 | -36.16 | 97.33 | -67.55 | 69.71 | -82.88 | 85.63 | -18.75 | 58.67 | -52.50 | 73.69 | -23.25 | 79.27 | -23.21 | 75.74 | -59.27 | 96.23 | -7.35 | 72.47 |
| 🌐mBERTBase | -30.82 | 81.16 | -78.38 | 96.50 | -32.51 | 76.04 | -56.79 | 87.68 | +2.83 | 66.18 | -28.09 | 77.90 | -26.21 | 78.77 | -15.00 | 77.36 | -39.03 | 96.71 | -4.23 | 73.28 |
| 🌐XLM-RBase | -7.58 | 82.30 | -6.09 | 97.92 | -1.29 | 81.69 | -2.05 | 91.98 | -17.42 | 59.14 | -9.36 | 81.14 | -14.59 | 80.73 | -16.03 | 77.15 | +2.82 | 97.70 | -4.20 | 73.28 |
| 🌐XLM-VBase | -33.83 | 80.56 | -20.69 | 97.63 | -28.86 | 76.70 | -14.13 | 91.03 | -21.98 | 57.55 | -15.62 | 80.06 | -19.84 | 79.84 | -29.50 | 74.50 | -151.06 | 94.05 | -2.76 | 73.65 |
| 🌐mDeBERTaV3Base | +6.43 | 85.17 | -1.22 | 98.02 | +8.40 | 83.44 | +9.51 | 92.89 | +16.51 | 70.94 | +9.60 | 84.41 | +9.76 | 84.82 | -10.25 | 78.29 | +9.56 | 97.86 | +5.96 | 75.89 |
| 🌐XLM-RLarge | -11.90 | 83.36 | +9.95 | 98.23 | +7.19 | 83.22 | +1.29 | 92.24 | -22.56 | 57.35 | +17.71 | 85.80 | +24.80 | 87.35 | -30.91 | 74.22 | -125.41 | 94.66 | +10.86 | 77.14 |
| 🌐mmBERTSmall | -12.74 | 82.01 | -12.31 | 97.80 | -25.84 | 77.24 | -12.29 | 91.18 | -1.15 | 64.80 | -9.62 | 81.10 | -23.95 | 79.16 | -24.26 | 75.54 | -2.70 | 97.57 | -2.53 | 73.71 |
| 🌐mmBERTBase | -2.75 | 84.17 | -3.74 | 97.97 | -7.90 | 80.49 | +3.97 | 92.45 | +22.68 | 73.09 | +4.06 | 83.45 | +2.25 | 83.56 | -31.71 | 74.07 | -17.57 | 97.22 | +3.24 | 75.19 |
| 🇺🇸DeBERTaV3Base | -52.40 | 78.48 | -77.17 | 96.53 | -66.97 | 69.81 | -71.13 | 86.55 | -10.73 | 61.47 | -27.20 | 78.06 | -48.12 | 75.09 | -38.62 | 72.71 | -120.01 | 94.79 | -11.65 | 71.37 |
| 🇺🇸DeBERTaV3Large | -10.22 | 83.95 | -25.22 | 97.55 | -18.13 | 78.64 | -22.60 | 90.36 | +26.14 | 74.30 | +13.77 | 85.13 | -7.38 | 81.94 | -21.03 | 76.17 | -39.03 | 96.71 | +1.45 | 74.73 |
| 🇺🇸ModernBERTBase | -132.43 | 72.82 | -332.62 | 91.52 | -184.06 | 48.64 | -282.12 | 69.97 | -21.35 | 57.77 | -65.66 | 71.42 | -57.07 | 73.58 | -31.68 | 74.07 | -84.92 | 95.62 | – | – |
| 🇺🇸ModernBERTLarge | -110.09 | 74.46 | -212.22 | 93.88 | -169.39 | 51.29 | -171.84 | 78.63 | -36.23 | 52.59 | -43.85 | 75.19 | -43.06 | 75.94 | -28.73 | 74.65 | -175.35 | 93.47 | – | – |
Good to know
- Models are ranked by the average (AVG) across all nine tasks. UD, UNER, WGSK and QA are scored with F1, RTE, NLI, HS and SA with accuracy, and STS with Pearson correlation converted to a 0–100 scale. RER (Relative Error Reduction) shows how much a model reduces the error rate relative to the SlovakBERT baseline; a positive value means it outperforms it.
- The results were published on SLAIH on June 23, 2026. Individual models may have been evaluated on different dates.
- A dash means the model has no result for that task. It is not a zero.
- If a model is missing from the table, no result for it is listed here. Its absence does not mean it performs worse.
- These results apply to this benchmark and its metric. They do not measure the model’s overall quality.
Datasets and methodology
Metric notes
- F1
- Macro-averaged F1, used for UD, UNER, WGSK and QA. It combines precision and recall and averages them across all classes regardless of size. Higher is better.
- Acc.
- Accuracy, used for RTE, NLI, HS and SA. The share of examples the model classifies correctly. Higher is better.
- Pear.
- Pearson correlation between the model's similarity scores and human ratings, used for STS. Converted to a 0–100 scale for the average. Higher is better.
- RER
- Relative Error Reduction expresses the relative decrease in error rate compared to the SlovakBERT baseline model. Higher means greater improvement over the baseline.
Token-Level Tasks
- Universal Dependencies (UD)
- The Universal Dependencies task (Nivre et al., 2020) is a community initiative aimed at creating and expanding syntactically annotated corpora (treebanks) for more than 100 languages using a unified annotation scheme. It provides information on parts of speech (POS), lemmas, syntactic dependencies, arguments, and modifiers across more than 200 datasets. Within skLEP we use the POS subset from the Slovak Dependency Treebank (Gajdošová et al., 2016) available in UD.
- Universal NER (UNER)
- The Universal NER task (Mayhew et al., 2024) provides high-quality, linguistically consistent annotations for named entity recognition (NER) in a multilingual setting. The Slovak portion (also from the Slovak Dependency Treebank) contains manually annotated entities of type person (PER), organisation (ORG), and location (LOC).
- WikiGoldSK (WGSK)
- WikiGoldSK (Suba et al., 2023) is a manually annotated NER dataset for Slovak that addresses the limitations of so-called "silver-standard" datasets. The data come from Slovak Wikipedia and were annotated according to guidelines inspired by the BSNLP-2017 shared task (Piskorski et al., 2017). The dataset uses the CoNLL-2003 tagset and extends it with a MISC category (e.g. films, awards, events, or media).
Sentence-Pair Tasks
- Recognizing Textual Entailment (RTE)
- The RTE task originates from the GLUE benchmark (Wang, 2018) and combines several datasets from entailment challenges into a binary classification task (entailment vs. not entailment). Because no Slovak version existed, the dataset was translated from English and subsequently revised by a native speaker. The test set was partially re-annotated manually, which is why it differs from the original (1,660 Slovak vs. 3,000 English examples).
- Natural Language Inference (NLI)
- The NLI task evaluates a model's ability to determine the relationship between two sentences: entailment, contradiction, or neutral. Instead of the MNLI dataset (used in GLUE) we use XNLI (Conneau et al., 2018). The Slovak version was created via a translation pipeline with subsequent review by a native speaker.
- Semantic Textual Similarity (STS)
- The STS task measures the semantic similarity of two sentences on a scale from 0 (no similarity) to 5 (identical meaning). Because no Slovak dataset existed, the data were translated and subsequently revised while preserving the original gold labels.
Document-Level Tasks
- Hate Speech Classification (HS)
- The dataset originates from the Slovak Hate Speech and Offensive Language Database and is designed to detect hateful and offensive content on social media. Posts are labelled in binary fashion (1 = hateful/offensive, 0 = other). The data were collected by scraping and cleaned using text clustering.
- Sentiment Analysis (SA)
- The sentiment analysis dataset was originally created as Reviews3 (Pecar et al., 2019) and later revised (Gurgurov et al., 2025). It contains Slovak customer reviews annotated as positive, negative, or neutral. In the version used in skLEP, neutral examples were removed and the task was simplified to binary classification.
- Question Answering (SK-QuAD)
- SK-QuAD (Hládek et al., 2023) is the first manually annotated question-answering dataset for Slovak, compatible with SQuAD v2.0. It contains more than 91,000 question–answer pairs, including both answerable and unanswerable questions.
Publications
Use the results
Citation
When you use the results, cite the benchmark and its original publication.
skLEP — Slovak Language Understanding Benchmark. SLAIH – Slovak Language AI Hub, June 23, 2026. https://slaih.sk/en/benchmarks/sklep skLEP: A Slovak General Language Understanding Benchmark. ACL 2025. https://arxiv.org/abs/2506.21508