How (not) to read benchmarks and leaderboards

The internet hosts a huge number of benchmarks, and each one is to evaluate a different aspect of AI models. Some benchmarks focus on the ability to follow instructions, others on the ability to work with images. Not every benchmark is useful to your specific use case, though — it always depends on what you intend to use the model for. But what actually is a benchmark? A benchmark represents an objective and reproducible way of evaluating the capabilities of models — language models, embedding models or various others. That's why it's important to first clarify what we want to evaluate as companies often rely more on an impressive demo than on verifiable data.

Types of benchmarks

Benchmarks can be divided into two basic types depending on what they aim to evaluate. The first type consists of benchmarks evaluating specific properties or a particular domain. Typical domains include law or telecommunications as an example. Specific properties include agentic behavior, tool use by language models, embeddings, or even language-specific benchmarks, as is the case with Slovak. Their purpose is to evaluate models on one particular task or capability, which can be indicative for further use cases (for example for companies or research). For example, if you're developing a coding agent, a specialized benchmark like SWE-bench — which tests solving real-world software engineering tasks — will tell you more than a general score from a language leaderboard. On the other hand, there are benchmarks that evaluate models in general. Their goal is to assess multiple capabilities at once and thereby provide a picture of which model can do the most. These are usually language benchmarks whose task is to evaluate generative capabilities across a broader spectrum of tasks: from classification, through instruction following and programming, all the way to advanced reasoning. Evaluating language models, however, is somewhat more complicated, which we'll discuss more later.

So far we've been discussing benchmarks, but what we encounter more often are leaderboards. So what's the difference between a leaderboard and a benchmark? A leaderboard represents a presentation layer over the results of one or more benchmarks, which determines the ranking of models, usually from best to worst. With leaderboards that aggregate results from multiple benchmarks, however, you need to be careful about how you read the results: an aggregated score can hide differences in evaluation methodology. For instance, different prompts or evaluation on different data. Imagine, for example, two models with the same aggregated score of 75%, where the first achieves 95% in factual accuracy and 55% in solving math problems, and the second the exact opposite. If we need a model for summarizing corporate contracts (where factual accuracy is crucial), the first model is clearly the better choice, even though the aggregated score suggests they're equal. Combining multiple benchmarks into a single score diminishes a leaderboard's transparency and meaningfulness, hiding much more behind that number than seems at first glance.

What are benchmarks good for and what do they serve?

So what are benchmarks actually good for and who are they intended for? Imagine that your company has a model deployed, whether purchased from another company or your own. Benchmarks help you verify the capabilities of these models on various types of tasks, especially the ones that matter most to your case.

A concrete example: a company wants to build a chatbot over internal documentation that works on the RAG (retrieval-augmented generation) principle. That is, a system that first searches for relevant parts of the documents in the documentation (the "retrieval" part), and only then generates an answer based on them (the "generation" part). The quality of such a system stands or falls on the embedding model that handles the search. If the model incorrectly assesses the similarity between the question and the documents, the chatbot receives the wrong context as input, and the answer will be poor — even if the generative part is otherwise very good. This is exactly where embedding benchmarks come in: benchmarks for vector representations of text (for example, SkMTEB for Slovak). These benchmarks let you compare several candidate models on tasks close to the retrieval part and choose the one that will be most reliable for your specific use case (instead of relying on a general score from an international leaderboard that may not cover Slovak at all).

Evaluation also helps us figure out whether a more capable model exists on the market (perhaps even a more cost-effective one), or whether our own model handles the tasks it was designed for well, without losing capabilities in other areas we rely on in practice. Benchmarks and leaderboards are therefore a useful tool for comparing our models against others. Based on that comparison, we can decide which model to deploy into production. And if we're already using a given model, we can clearly justify why we should stick with it or why we should switch to another — which is especially useful for internal decision-making.

What makes a benchmark high-quality?

To help individuals and organizations make informed decisions, a benchmark must satisfy several quality criteria.

One of the main aspects of a high-quality benchmark is task coverage. This depends primarily on whether it's a specific or generic benchmark, but the goal is always to cover as diverse a spectrum of tasks as possible. For general benchmarks, we try to cover multiple types of tasks: from classification, through free-text question answering to agentic behavior or reasoning. For benchmarks focused on specific tasks or domains, the goal is to cover multiple aspects of that domain, including edge cases. For language-specific benchmarks, we want the data to contain context relevant to that particular language. For Slovak, then, we should focus on Slovak grammar and linguistic nuances, and, for example, not ask questions in Slovak that are specific to the American context.

The second prerequisite for a high-quality benchmark is high-quality data. Above all, the data should represent a realistic sample that the model could encounter in production deployment — for Slovak we shouldn't be evaluating on US-centric questions. At the same time, it's important to make sure that the data on which the benchmark evaluates isn't also data the models were trained on (so-called data leakage). Verifying data quality, however, is often problematic — especially when it comes to whether the data was part of training, since large language models have already seen most of the internet. For this reason, approaches exist for checking whether data from our benchmark was used in training, for example by identifying it via a so-called canary string (a unique string inserted into the data, which can later be used to verify whether it turns up in the model's training data).

The last aspect to mention is robust methodology. A high-quality benchmark should have carefully thought-out evaluation procedures for models. What matters, then, isn't just what we measure, but also how. A benchmark that evaluates using a single prompt template or without repeated runs won't tell us much about models' capabilities, since even small differences in phrasing can drastically alter outputs and rankings. Simply changing the wording from "Choose the correct answer" to "Which of the following options is correct?" can shift some models' scores by several percentage points. Methodology should therefore be robust. It should include a variety of instructions and multiple setups, whether examples are provided or not.

Equally important is how the results themselves are evaluated. There are, for example, benchmarks that evaluate the correctness of an answer based on logits (token probabilities), which is in a sense methodologically correct, but doesn't actually reflect how large language models are used in practice. In real-world deployment, a language model answers via generated text, not via token probabilities. So, even if the correct answer among the offered options is more probable than the others, it doesn't mean the model will actually generate it. The result can be that the benchmark shows a higher score than what the model actually achieves in production. It's therefore necessary to take into account how robust the evaluation is and whether it truly reflects the way models are used in practice.

How to interpret benchmarks and their results?

Final scores depend heavily on the metrics used, the data subset tested, and the aggregation method. For instance, weighting all tasks equally produces very different results than weighting by sample size — in the latter case, tasks with larger amounts of data can skew the overall result. The same benchmark can thus produce different results depending on how it was measured (for example, the difference between evaluation based on logits versus exact match with generated text). It's the same benchmark, but a different way of obtaining the results — one will come out better, the other worse.

The already-mentioned average can hide model weaknesses in another way, too. Beyond the problem of result aggregation, there's a second aspect that the average result doesn't capture. The average tells us only part of the story; what's also important is knowing how individual models perform on specific tasks or areas. Some models may be better at reasoning and math problems, others at agentic tasks, tool use, or programming. It's therefore necessary to look at the broader context relevant to our particular use case.

Conclusion

Benchmarks help us decide which model is the right one for our specific use case. Everything depends, though, on whether we've chosen an appropriate benchmark, whether we can correctly interpret its results, and whether it gives us a realistic picture of how models are used in practice. On some benchmarks today, most models already score somewhere between 80 and 90%, so the differences between them are therefore minimal. In such cases we say the benchmark is saturated — that is, it has hit its ceiling and no longer sufficiently distinguishes the quality of individual models. In that case, it's more appropriate to reach for a different, more challenging benchmark that still allows a more objective comparison across models. Achieving a 100% score is (in most cases) not even realistically attainable, so if models' results don't differ significantly from one another over the long term, that's a further sign that the benchmark in question is saturated.

Within our own initiative, several benchmarks focused on Slovak have been developed to evaluate different types of model capabilities — whether a benchmark for Slovak language understanding (skLEP), or a benchmark for text representations (SkMTEB), which can serve as a reference point when selecting models for RAG applications in Slovak.