AI GUIDEWhat does the score measure?
“Model A scored higher” is not enough to choose a tool. Ask: higher on what tasks, with which tools, how many attempts and what counts as success? A model solving a short coding exercise is not automatically ready to maintain your agency's projects.
A benchmark is a defined set of tasks and scoring rules. It is useful evidence about those conditions. It is not a guarantee about every task that sounds similar. Separate the measured result, the author's explanation and any prediction about the future.
This lesson follows the general Academy article on what is changing in AI. Its examples are synthetic except for the explicitly identified METR finding. You need no statistical software to follow the reasoning.
The seven-month claim
In its March 2025 work, METR described a historical trend in the length of software tasks that frontier agents could complete at a specified reliability. The widely quoted roughly seven-month doubling referred to the 50%-success time horizon, measured using how long human professionals took on the tasks.
That is more specific than “AI capability doubles every seven months”. It is not a count of all possible tasks or a guarantee that an agent will finish your next project. Extrapolation into future capabilities is a prediction. The linked research explains the methodology and limitations; consult the current evaluation pages for newer measurements.
Make a task-specific comparison
Suppose two fictional assistants prepare product-data summaries. A answers 9 of 10 examples correctly at €0.02 each. B answers 8 of 10 correctly at €0.01 each. Those numbers are invented for arithmetic, not product claims.
The observed success rates are 90% and 80%. The ten-run costs are €0.20 and €0.10. That does not settle the decision. Were the wrong answers minor formatting problems or invented prices? How much human correction was needed? Were the same inputs used? Ten cases are also a small sample.
Design a more useful comparison by recording factual correctness, preservation of unknowns, time and cost per checked result. Keep the source material and evaluation rules the same. Use repeated runs if variation matters, and include the cost of retries and human review. A cheap first answer can become expensive after correction.
Turn a headline into a question
Exercise: take the fictional headline “AI can now run an entire agency”. Rewrite it as a task you can test: “Can this assistant prepare a project brief from supplied notes without inventing a budget or a deadline?” Define the input, expected facts, prohibited inventions and review method before trying it.
Then ask what the test does not cover: sales, staffing, client relationships, access permissions and delivery responsibility remain separate tasks. A good result on the brief cannot establish that the agency is autonomous.
You have understood the lesson when you can name the measurement, its conditions, its limits and the next piece of evidence you need. Keep forecasts as possibilities to watch, not requirements for today's business decision. Update the evidence for an existing topic rather than creating another article every time a model climbs a leaderboard.
Sources
Read the original sources to explore the topic and check specific claims.