Every time a new artificial-intelligence model is released, its maker publishes a table of scores meant to show that it beats the competition. These figures come from benchmarks, standardized tests whose inner workings are rarely explained. This article describes what AI model benchmarks really measure, the traps that can make a score misleading, the capabilities that published scores leave out, and why evaluating an agent is a different problem. It is written for anyone who has to form an opinion about a model at launch without being an evaluation specialist. This is the first part of a three-article series.

Here are the seven sections of the article. Each title links to the matching section.

• What is an AI model benchmark?: what a benchmark is, its three components and the main families of tests.
• What does a benchmark score actually measure?: why a score measures a complete system, and the seven elements of the measurement protocol.
• How can a score mislead without any cheating?: the six mechanisms that make a score misleading without any fraud.
• When does a gap between two scores become significant?: how to tell a real difference in capability from the mere randomness of measurement.
• Which capabilities do AI model benchmarks leave out?: the fifteen evaluation dimensions, and the ones launch tables pass over in silence.
• Why doesn’t a good model automatically make a good agent?: what separates a benchmark question from an agent task, and why cost per task becomes decisive.
• The six questions to ask in front of a score table: a reading grid to apply to any score table.

What is an AI model benchmark?

A benchmark is a standardized exam for an artificial-intelligence model. Most often, it has three parts:

• A fixed set of questions or tasks, identical for every model tested.
• A reference answer for each question, that is, the answer considered correct.
• A scoring method, which compares the model’s output to that reference answer.

A few benchmarks work differently and have no reference answer at all. They show human beings two competing answers and aggregate their preferences.

The result is a single score, usually a success rate. This figure acts as a common yardstick: two models that have taken the same test under the same conditions become comparable.

Benchmarks fall into families, each shedding light on one narrow skill. The six families below are presented one by one. Each subsection states what the family measures and, when its name is not enough, spells out what it actually contains.

Knowledge and reasoning

This family covers general knowledge and doctorate-level scientific questions.

Mathematics

This family brings together two very different levels under the same word: school-competition problems and research problems. The first level is that of mathematics competitions reserved for pupils and students. The test most used to evaluate models is the AIME (American Invitational Mathematics Examination). It is an American contest of fifteen questions to be solved in three hours, reserved for the top performers of the AMC (American Mathematics Competitions) 10 and AMC 12 contests, and each answer is a whole number between 0 and 999 (Mathematical Association of America). The second level is that of research: tests such as FrontierMath use original statements, written specifically for the test by professional mathematicians and never published before, which even a specialist takes several hours to solve.

Multimodal

This family covers reasoning that combines several media, in practice mostly text and images. Tests covering audio and video exist, but remain rare in standard evaluations.

Abstract reasoning

This family asks the model to guess a rule from a few examples, then apply it to a case it has never seen. It is evaluated with visual puzzles. The reference test, ARC-AGI, shows the model two to five pairs of coloured grids. In each pair, an input grid has been transformed into an output grid according to a rule that is written nowhere, for example “replace each shape with its mirror image”. The model must guess this rule from the examples, then apply it to a grid it has never seen. The grids are at most thirty cells across and use up to ten colours (Chollet and colleagues, ARC-AGI-2).

Coding and agentic tasks

This family asks the model to fix real bugs, taken from the history of public software and replayed for the test, or to carry out a complete task in a simulated computing environment.

Human preference

This family has human beings choose between two competing answers. It relies on a pairwise-comparison setup. On the LMArena platform, a user asks their own question. It is sent to two models whose names are hidden from them. They read the two answers and pick the one they prefer, without knowing which model produced which. The names are revealed only after the vote, which prevents a brand’s reputation from influencing the choice. The millions of votes collected this way are then converted into a ranking by a statistical pairwise-comparison model (Chiang and colleagues, 2024).

None of these families measures “intelligence” in general. A model can excel at one and disappoint at another. The second article in this series will review these tests one by one, with their strengths and weaknesses.

What does a benchmark score actually measure?

This is the most important point in this article. A score does not measure the model alone. It measures the result produced by a set of elements of which the model is only one piece, during one precise run and under precise conditions. Changing any one of these elements changes the score, without the model having changed.

This set of conditions has a name: the protocol of measurement. A complete protocol describes seven elements, taken one by one below. Knowing this list is directly useful for reading a score table, because every element the maker does not state is an unknown that may explain a gap.

Element 1: the model version

One and the same commercial name often covers several successive versions with different results. A score therefore makes sense only together with the precise version of the model as it existed on the date of the test, and not merely its commercial name.

Element 2: the prompt

The prompt bundles two things. The first is the system prompt, that is, the general instruction text placed before each question, which sets the model’s role and the expected format of its answer. The second is the possible presence of examples of correct answers slipped in before the real question. This technique has a name, few-shot: the model is shown two or three already-solved questions before being asked the one that counts. It raises scores. The comparison is therefore skewed if one maker uses it and the other does not. When no example is provided, this is called zero-shot.

Element 3: the evaluation harness

The harness is the software that administers the exam. It chains three tasks. It puts each question to the model in a given format, it retrieves the produced answer, then it applies the scoring method described in the next point. Two different harnesses can produce two different scores for the same model, either because they do not format the questions the same way, or because they do not extract the final answer from the produced text the same way.

Element 4: the scoring method

It is the harness that scores, but it can do so in four ways, which do not have the same reliability or the same cost.

• Exact match. The model’s answer must match the reference answer character for character. This is simple but harsh, because a good answer written differently is counted wrong.
• Normalized match. The harness cleans up the answer before comparing it, by removing punctuation, standardizing capitalization or rewriting mathematical formulas. This cleaning step varies from one harness to another and, on its own, moves scores.
• Running automated tests. For code, the produced text is not inspected: it is executed and checked against a battery of tests. The result is objective, provided the tests themselves are correct.
• Model as judge. A second model decides whether the produced answer is equivalent to the expected reference answer. This method accepts free-form wording, but it introduces the judge’s biases into the measurement.

Human-preference tests fall outside these four cases, since their scoring is not automatic at all: people vote.

Element 5: the number of attempts

A model may be allowed a single try per question, or several. Publications indicate this choice with a notation borrowed from code evaluation, defined in 2021 by the authors of the Codex model (Chen and colleagues).

• pass@1: the model produces a single answer per question, and that answer must be right. This is the measure closest to real use.
• pass@k: the question counts as passed as soon as at least one of the k answers is correct, whatever its position among the k. The best answer is not chosen: each of the k answers is checked automatically, which is possible because the correct answer is known (in mathematics) or because the produced code is run against a battery of tests, and it is enough for one of them to succeed. A pass@10 score is therefore greater than or equal to the same model’s pass@1, because its success condition is easier to meet. In return, this measure says nothing about reliability: a model that succeeds only once in ten is counted as passing on the same footing as a model that succeeds ten times out of ten.
• Majority vote, written maj@k or cons@k: the model produces k answers and the most frequent one is kept.

Three consequences for reading. A pass@k score compares only to another pass@k score with the same k. A score published without a scoring note is most often a pass@1, but nothing guarantees it. Finally, a score obtained with sixty-four answers per question costs sixty-four times more compute than a pass@1, which the table never says.

Element 6: the compute budget

This is the amount of compute the model is allowed to spend on answering. It is expressed in two ways. The first is a level of reasoning effort set by the maker, for example low, medium or high. The second is a limit on intermediate text. Intermediate text is the reasoning the model writes for itself before giving its final answer: it details its steps, tests ideas, corrects itself, and this output is generally not shown to the user. The larger this budget, the higher the score and the more the answer costs.

Element 7: the allowed tools

Some protocols let the model use tools during the exam, such as running code, browsing the web or reading files. Others forbid it and accept only text produced from memory. The same benchmark therefore sometimes exists in two versions whose scores are not comparable. This element is rarely stated in a score table, even though it often weighs more heavily than all the others.

An example: three scores for the same model

An example published by OpenAI shows the size of the effect. On its o1 model, evaluated on the AIME 2024 (the mathematics test described above), the maker published three very different scores, obtained by changing only two conditions, the number of attempts and how the final answer is selected among those attempts. The detail is in their presentation of their reasoning models.

• The first measure kept the single answer produced by the model, and gave 74% success.
• The second kept the majority answer among sixty-four generated answers, a majority vote written maj@64, and rose to 83%.
• The third had a scoring model trained for this select, among a thousand generated answers, the one it judged best, and reached 93%.

The score grows from one measure to the next, and it grows sharply. None of these three figures is false, but the three do not describe the same thing.

What to conclude for comparing two scores

Two scores compare only if their protocols are aligned, element by element. Any difference in version, prompt, harness, scoring method, number of attempts, compute budget or allowed tools must be treated as a possible explanation of the gap, on the same footing as the difference in capability between the models.

A published score should therefore always come with a description of its complete protocol, that is, the seven elements listed above, and not the harness alone. Without this description, the figure is neither verifiable nor reproducible by a third party.

The converse is worth stating. Even when the declared protocol is identical on both sides, the comparison can remain imperfect. The harness may in fact suit one model better than the other, for example because it formats questions in a way close to what one of the two encountered during its training.

How can a score mislead without any cheating?

A score can mislead even though no one has cheated. Six distinct mechanisms produce this effect. Each is established by public cases, whose sources are cited in the paragraphs that follow.

Trap 1: self-reporting

Most scores published at a model’s release are produced by its maker, which has a commercial interest in presenting the most favourable figures. Nothing obliges it to publish the results that work against it, nor to have its measurements checked by an independent third party.

Trap 2: cherry-picking the tests shown

The maker chooses the benchmarks it highlights, naturally favouring those where it wins. A well-known test, missing from a launch table, is therefore information in itself.

Trap 3: the measurement conditions

This trap consists in exploiting the leeway left by the protocol described in the previous section. A maker that tests its model with the maximum compute budget and several attempts per question mechanically obtains better figures than a competitor tested in a single attempt and with a reduced budget. Both figures are accurate taken separately. It is bringing them together that is misleading, all the more so as this difference in conditions is not always flagged.

Trap 4: training-data contamination

It happens that a test’s questions turn up, with their answers, in the data used to train a model. The model has then already met the topic and its solution during its training, and its score measures its memory rather than its reasoning ability.

Three methods exist, of two kinds: two prevent contamination upstream, the third detects it after the fact.

• The hidden marker. The test designers insert into their files a unique string of characters, together with a notice forbidding the use of this data for training. An honest maker searches for this marker in its data and sets aside the documents that contain it. The method has an obvious limit: it works only if everyone plays along (Search-Time Data Contamination, 2025).
• The private question set. Part of the questions is never published and the official evaluation is done on that part. This is ARC-AGI’s choice, which keeps a private set of one hundred tasks reserved for rankings (Chollet and colleagues, ARC-AGI-2).
• New questions. This is the only one of the three that detects contamination after the fact, instead of preventing it. An entirely new question set is written, of difficulty equivalent to the original test, then the two scores are compared. The GSM1k study applied this method to a very widespread arithmetic test: some families of models lose up to eight points on the new questions, which betrays partial learning of the old ones (Zhang and colleagues, 2024).

Contamination is one of the two reasons OpenAI stopped using the SWE-bench Verified coding benchmark in February 2026. The maker cites both this contamination and a high share of faulty test cases, which reject correct solutions: 59.4% of the audited failed tasks were in this situation.

Trap 5: errors in the questions themselves

Test sets contain errors, and sometimes many. In June 2026, Epoch AI corrected its FrontierMath benchmark after finding errors in 42% of the problems. An independent study estimates that about 6.5% of the questions in the famous MMLU test are faulty, most often because the reference answer is wrong or because the statement is ambiguous. A model that answers a badly-posed question correctly is counted as failing.

Trap 6: test saturation

When all the good models brush against a perfect score, the test no longer distinguishes anything. The gaps that remain then rest on the few most doubtful questions of the set, the very ones covered by the previous trap. The test remains useful for spotting a weak model, but it becomes unusable for separating the best.

What these six traps share, and what sets them apart

Four of these six traps stem from one and the same principle, Goodhart’s law, which holds that a measure that becomes a target ceases to be a good measure. As soon as a score becomes a selling point, makers optimize for that score, and the score loses its information value. Self-reporting, test cherry-picking, measurement conditions and contamination all describe this behaviour: an actor adjusts the way it measures or communicates because the score has become a commercial stake.

The other two traps have a different origin and involve no behaviour. Errors in the questions stem from a manufacturing defect of the test. Saturation stems from wear linked to the models’ progress.

When does a gap between two scores become significant?

Faced with a gap of two or three points between two models, the figure alone is not enough to conclude. The right question is whether this difference is solid enough to repeat, or whether it may come from the mere randomness of measurement. Three elements help answer: the number of questions in the test, the detail of the questions where the models differ, and the stability of the scores from one run to another.

Test size: a first landmark

The first landmark is the number of questions in the test. On the AIME, which has only fifteen, a single extra correct answer moves the score by 6.7 points: a gap of that size therefore weighs only one question, and a single lucky answer is enough to produce it. On HLE, which has 2,500 questions of equal weight, each question is worth only 0.04 point, and a gap of two points corresponds to about fifty answers. The rule is simple: the fewer questions a test has, the larger the gap must be to mean anything. But the number of questions does not settle everything. Even on a test with thousands of them, it is not enough on its own to establish that a gap is real.

Looking at the questions where the models differ

To go further, one has to remember that the two models take exactly the same questions. Their answers can therefore be compared question by question, and it is the questions on which they differ that carry the information. Two situations can give the same final gap of fifty answers. In the first, the two models differ on only fifty questions, and one wins them all: the advantage is clear. In the second, they differ on 1,350 questions, one winning 700 and the other 650: the final gap is the same, but it is drowned in a large number of disagreements that almost cancel out, and it is far more fragile.

Two statistical methods exploit this reasoning, and it is enough to know their principle. The paired test compares the two models answer by answer on the same questions, instead of comparing their two overall scores, and it measures whether the advantage on the questions of disagreement is too regular to be put down to chance. Resampling, for its part, replays the comparison hundreds of times, each time on a random draw of the same questions: if the same model wins in the great majority of these draws, the advantage is solid; if it wins about one time in two, it is fragile.

The models are not deterministic

A second source of uncertainty lies in the very working of the models: they are not deterministic. Asked the same question twice, a model can give two different answers, so that the same test, replayed, does not yield exactly the same score (Thinking Machines Lab, 2025). A model must therefore be run several times to measure the size of this variation, and one should be wary of a score obtained in a single run, as a score published without an indication of the number of runs almost always is.

Publish the gap, not just the two scores

All of this has a consequence for the way results are presented. Publishing each model’s score, for example 82% for one and 80% for the other, leaves the reader to do the subtraction and judge by eye whether two points count. It is more useful to publish the gap between the two directly, that is two points, while indicating how uncertain this figure is.

Two notions differ here. Uncertainty is the general idea that a measured score is never exact, because it depends on the sample of questions asked and on the vagaries of execution. The confidence interval is the way to quantify this uncertainty: it is the range within which the true value very probably lies. Saying “80%, give or take 3 points” amounts to giving the range from 77% to 83%.

The gap between two models is itself quantified with its own range. Take our two models again: A gets 82%, B gets 80%, so the gap is 2 points. But this 2 is no more exact than the two scores it comes from: it too has a margin of uncertainty.

How is this margin obtained? By the repeated-draw method described above, resampling. Each model’s answer to each of the test’s questions is known. A fake test of the same size is then built, by drawing its questions at random from the real ones, with the right to pick the same one several times, and the gap between the two scores is recomputed on this fake test. This draw is repeated hundreds of times, which gives hundreds of slightly different gap values. The range is the interval that gathers the great majority of these values: if they nearly all fall between 0.5 and 3.5 points, the margin of uncertainty of the gap is this range from 0.5 to 3.5 points. It is therefore obtained not by subtracting the ranges of the two scores, but by recomputing the gap itself a great many times, on different draws of the same questions.

This range of the gap gives the answer directly, through a single question: does it contain the value zero? A gap of zero would mean the two models are tied. If the range runs from 0.5 to 3.5, it never drops to zero: even in the case least favourable to A, the gap stays at 0.5 point in its favour, so A is really ahead. If instead it ran from −1 to 3, it would contain zero, a tie would remain possible, and nothing could be concluded.

It would be a mistake, to decide, simply to look at whether the ranges of the two scores overlap. They overlap here, A’s (79% to 85%) and B’s (77% to 83%) sharing the zone from 79% to 83%, which would suggest a tie. This criterion is misleading, because requiring that the two ranges not touch amounts to requiring that A’s worst possible score, 79%, still beat B’s best possible score, 83%. That is a far too severe condition: it assumes the two models would have their worst bad luck at the same moment, each in the direction that hurts it. The right measure therefore remains the range computed directly on the gap, judged on a single question: does it contain zero or not?

Significant does not mean important

Finally, a gap judged statistically significant does not thereby mean that it counts in practice, nor that a model is superior overall. It indicates only that, on this precise test and under the conditions of the protocol used, the observed advantage is hard to explain by the mere randomness of measurement. It then remains to judge whether this gap is large enough, and the test representative enough of your real use, to weigh on a decision.

Which capabilities do AI model benchmarks leave out?

The scores highlighted at a model’s launch give an incomplete picture of what ought to be evaluated. The survey carried out for this series identifies fifteen evaluation dimensions. Tables published at launch most often cover only six or seven of them. The fifteen dimensions are listed below, each with what it measures and its usual presence in makers’ launch communications.

Dimension 1: academic knowledge

This dimension measures the breadth of the model’s knowledge across a large number of disciplines, from history to medicine. It is evaluated with multiple-choice questionnaires covering several dozen subjects. It is present in almost every launch communication.

Dimension 2: expert reasoning

This dimension measures the ability to solve doctorate-level scientific questions, written by specialists and designed so that a simple documentary search is not enough to answer. It is present in almost every launch communication.

Dimension 3: mathematics

This dimension measures the solving of mathematical problems, from school contests to the research statements described above. It is present in almost every launch communication, because it produces spectacular figures.

Dimension 4: abstract reasoning

This dimension measures the ability to discover a novel rule from a few examples, then apply it correctly to a new case. It is frequently present in launch communications, but with a rarely detailed protocol.

Dimension 5: multimodal understanding

This dimension measures reasoning over content that mixes text and images, for example reading a chart, interpreting a technical diagram or analysing a screenshot. It is frequently present in launch communications.

Dimension 6: human preference

This dimension measures what users actually prefer, by having them compare two answers to their own question. It is frequently highlighted, because it is the only one that directly reflects the feel of use.

Dimension 7: code and agentic behaviour

This dimension measures the solving of software tasks that require several steps, such as fixing a bug in a real code repository. It has been present in almost every launch communication since 2024, the year coding tests on real repositories, in particular SWE-bench, established themselves as the reference measure.

Dimension 8: instruction following

This dimension measures compliance with explicit format constraints, for example answering only in JSON, not exceeding a hundred words or using no polite formulas. It is rarely published, even though it directly governs the model’s reliability and its deployment in an application chain.

Dimension 9: factuality

This dimension measures the ability to answer correctly, and above all to abstain rather than invent an answer when the information is missing. It is rarely published in launch communications.

Dimension 10: long context

This dimension measures the ability to retrieve a precise piece of information in a very long document, and to link together pieces of information far apart from one another. It appears occasionally, usually in the form of a single-sentence retrieval test.

Dimension 11: multilingual

This dimension measures the model’s performance in languages other than English. It is rarely published, which is a serious gap for French-language use, since nothing guarantees that a score obtained in English carries over to French.

Dimension 12: robustness

This dimension measures the stability of the score when questions are reworded without changing their meaning. It is almost always absent from launch communications, even though it indicates whether a score reflects a real capability or an adjustment to the test’s wording.

Dimension 13: calibration

This dimension measures the match between the confidence the model displays and its real accuracy. It is almost always absent from launch communications, even though it governs professional use: a model that is 70% accurate and correctly flags its 30% of uncertainty is more usable than a model that is 75% accurate but asserts everything with the same assurance, because one knows when to check it.

Dimension 14: safety

This dimension measures dangerous behaviours, as well as excessive or, on the contrary, insufficient refusals. It does not appear in the score tables but in a separate document, published at the same time as the model.

Dimension 15: efficiency

This dimension measures the cost, latency and compute consumption needed to reach the announced score. It is almost always absent, even though a score obtained with ten times the compute budget is not comparable to a score obtained in a single pass. Cost per task is nonetheless the first decision criterion in a real project.

The second article in this series takes up one by one the dimensions covered by public benchmarks, with the corresponding tests and their governance.

Why doesn’t a good model automatically make a good agent?

Since 2024, a model is no longer used only to answer a question: it is placed at the heart of an agent. An agent is a model inserted into a software loop that lets it plan a task, use tools such as reading a file, running code or calling a service, act over several steps and keep the memory of what it has done.

But an agent task does not look like a static benchmark question, that is, the classic form described above: a statement, a reference answer, a scoring. The following table compares the two forms, criterion by criterion. It deliberately contrasts two extreme situations, a fully static benchmark on one side and a complete agent task on the other. Many real tests sit between the two. A coding benchmark, for example, gives the model access to tools, like an agent task, while keeping automatic scoring, like a static benchmark.

Comparison criterion	Static benchmark	Agent task
Duration	A single turn	Dozens of steps
Answer	Known in advance	A deliverable judged on criteria
Tools	None, by construction	Terminal, browser, code, files
Frequent failure causes	Ignorance, misreading, format error	Losing the thread, no recovery after an error
Cost	A single answer produced	Dozens of successive calls to the model

The failure-cause row deserves a nuance. On a static benchmark, ignorance is not the only cause of error: a model can also misread an ambiguous statement, or give the right answer in a format the harness cannot recognize, and the score will count it as failing in both cases.

This difference in nature between a static benchmark and an agent task has an arithmetic consequence, which can be illustrated by a deliberately simplified calculation. Imagine a task that requires twenty successive steps, and a model that succeeds at each step in 95% of cases. These two figures are chosen for the example and are not measured: they represent an agent task of common length and a model that is very reliable at each step. For the task to succeed, all twenty steps must succeed. The probability of success is therefore 0.95 multiplied by itself twenty times, that is about 36%. In other words, a model that succeeds at each step in 95% of cases still fails the complete task nearly two times out of three, its probability of success dropping to about 36%.

This theoretical calculation assumes that no error can be recovered, which is not the case for a real agent. It can check its results, backtrack, retry a step or ask for human intervention, which softens the degradation without cancelling it. The quality of planning becomes decisive, and cost per task, negligible on an isolated question, becomes a first-order criterion.

The fifteen capabilities described above remain a useful starting point for deciding what to test in an agent: an agent that codes, for example, draws on code, instruction following, factuality and reasoning. But they are not enough; they must be translated into the context of the target task, then supplemented with what is specific to the agent, namely reliability over a long sequence of steps, correct use of tools and the ability to recover after an error.

For an agent, the harness weighs so heavily that it is part of the product being evaluated: the same model shows different scores depending on the agent scaffolding used. The third article in this series will show this concretely on a real case, and a separate article series, devoted to evaluating agents in situation, will then extend this topic beyond benchmarks.

The six questions to ask in front of a score table

From everything above follows a simple reading grid. Faced with a score table published at a model’s launch, ask these six questions in order.

1. Who produced the figure? The maker itself, or an independent evaluator?
2. On which version of the test, and on what date was the figure recorded? The same benchmark name can cover different question sets, and a living ranking changes within a few days.
3. With which protocol? The seven elements described above, and in particular the number of attempts, the compute budget and the allowed tools.
4. Does the gap exceed the measurement noise? On a test of two hundred questions, several points are needed for the gap to be something other than chance.
5. Does the gap show up across several families of tests? A model leading on a single test may have been prepared for that test.
6. Does the test measure your use? An academic questionnaire does not predict an agent task. The fifteen capabilities listed above give you the list of what ideally ought to be checked, to be weighed against your real use.

A benchmark score remains useful, provided it is read as the result of an experiment and not as a property of the model. This reading has a verification cost, and it is this cost that explains why launch communications keep working as a commercial argument: almost no one traces things back to the protocol. The six questions above bring this work down to a few minutes per table.

The next article reviews the market’s benchmarks one by one. For each test, it says who runs it and in which version it exists. Above all, it states how much trust its scores deserve. In the meantime, you can continue the discussion on my LinkedIn profile.

Bibliography and sources

• Mathematical Association of America, invitational competitions including the AIME.
• Chollet and colleagues, “ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems”.
• Chiang and colleagues, “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference”.
• Chen and colleagues, “Evaluating Large Language Models Trained on Code”, which defines the pass@k notation.
• OpenAI, “Learning to reason with LLMs”.
• OpenAI, “Why we no longer evaluate SWE-bench Verified”.
• Zhang and colleagues, “A Careful Examination of Large Language Model Performance on Grade School Arithmetic”, the GSM1k study.
• “Search-Time Data Contamination”, on hidden markers and contamination.
• Epoch AI, FrontierMath v2.
• Gema and colleagues, “Are We Done with MMLU?”.
• Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference”.