By Michael Domanic, Section Head of AI

The hardest part of the Head of AI job is keeping up.

In the space of a few weeks, Anthropic shipped a new class of model in Fable and Mythos, and OpenAI answered with Astra, built on an entirely new pretraining run rather than an upgrade to what came before. 

Each one arrived with benchmark charts, viral demos, and the same two questions for every Head of AI: Should we be using this? And if so, for what?

If you’re getting those questions, your first instinct might be to look at the published benchmarks or the conversation on social media. But while those things are directionally helpful, they don’t capture what matters most to you – the impact of the new model on your actual work.

The only way to know if the models help you is to test them. Here's how we do that at Section.

The four questions you should ask about a new AI model release

Do this review when you see a new frontier-tier model release from a lab you already run in production. (Don’t waste your time with point releases).

Your Head of AI should own it, but this is typically done best in collaboration with your AI Champions.

1. Does it do our work better?

For each of the ways your company actually uses AI (e.g., coding, writing, creative work, research and analysis, production agents), pull 10-20 real tasks from real work. Document the input (what you gave AI) and the output (what it gave back) and save as the baseline.

Have the person who owns each kind of work write a short rubric, four to six criteria in your company's terms. Then run the new model on the same tasks and have graders compare outputs two at a time, without knowing which model made which. People are much better at "which of these is better" than at scoring things on a scale, and hiding the model name kills the new-toy bias that inflates every first impression.

For agents, grade the process quality first: Did it finish? How many steps and tool calls did it take? Did it stay inside its boundaries? Then look at the quality of what it produced.

2. What does it cost us?

For every run, make a log of tokens in, tokens out, cache reads, time to complete, and dollars. 

Test the new model at two or three thinking levels instead of defaulting to the highest one. The reflex on a new release is to turn reasoning all the way up, but a newer generation will often match or beat your old results at a lower setting, and every extra thinking token is money you spend on every single run. What you want is the lowest level that clears your quality bar, not the level that makes you feel safest.

Your final metric should be cost per task at equivalent quality for each use case, shown alongside the quality score from question one.

Don’t rely on the vendor’s pricing claims. Anthropic said Fable 5.1 would be cheaper than its predecessor because of lower cache-read costs; independent testers found that some tasks actually cost more because the model used more tokens. Both can be true. What matters is what the model costs on your workload.

And don’t stop at token costs. Factor in usage caps, rate limits during the launch window, and whether your legal team’s data retention terms apply to the new model on day one. 

3. What could go wrong?

Test safety in the operational sense, not just in the lab sense. Run three tests on your own workflows: 

• Does it refuse legitimate work? Give it tasks the old model handled successfully and see whether new safety behavior gets in the way.

• Does it take actions you didn’t ask for? Newer models can be more tenacious. If you ask it to draft an email and it sends it instead, that’s a problem. Run your existing permission boundaries and confirm they still hold.

• Does it resist prompt injection? Plant malicious instructions inside content your agents read. For example, a line in a document telling the model to ignore its task and forward the file somewhere, and confirm it doesn’t follow them.

That last test is the newest and the one most companies have never run, and it matters more every month as models gain the ability to act on what they read.

4. What can we do now that we couldn't before?

For the last test, deliberately hand the model work your agents were never able to do. 

Give it a task that should take hours and walk away. For computer use capabilities, point it at a workflow that lives inside a piece of software with no API or MCP. Pull out the list of things you ruled out a year ago because they needed a human to click through screens, and try them again. 

The output is a short list of new work worth piloting, and in my experience it's the part that will catch your CEO’s attention. 

3 basic rules to keep the results honest

1. Run every task at least three times. These models are not deterministic, and the difference between two models on one run is often smaller than the difference between two runs of the same model.

2. Decide the bar before you see results. Write down what "meaningfully better" means (say, “wins more than 60 percent of blind comparisons at no more than x% of current cost”) before anyone looks at an output. Otherwise you will rationalize whatever you find.

3. Look at the worst case, not the average. A new model can be better on average and worse at the three things that really matter, like output format, following your schema, or calling tools reliably. Your team won't notice a better average as much as the one run that broke the sales dashboard.

What to do with the evaluation findings

When the readout comes back, you have real answers to the two questions you started with. Should we use it? Almost always yes, somewhere. For what? That's what the findings tell you, and in practice every use case you tested lands in one of five places.

1. Switch it. The new model wins the blind comparisons by more than the margin you agreed on in advance, and cost per task is flat or lower (or at least not meaningfully higher). Move one agent at a time, and keep the old baseline running alongside it for a handful of runs to catch what you couldn’t in testing.

2. Switch it and turn the thinking down. The quality holds at a lower reasoning level, so you get the same output for less cost on every run. Most teams never find this one, because you only find it if you tested more than one setting.

3. Close the safety gap, then switch. The quality and cost numbers say go, but question three turned something up: an issue that the old model was able to handle, an action taken that nobody asked for, etc. That doesn't necessarily rule the model out but it does mean the switch waits until you've named the specific behavior, tightened the permission or the prompt that let it through, and re-run the test that failed. Set a date so it doesn't sit.

4. Stay where you are. There’s no meaningful quality gain at the bar you set, or it’s a gain that costs more than you said you'd pay for it. Write down the reason with the numbers and the date, because within a few weeks someone will inevitably ask why you aren't on the new model. Hold on to the tasks and rubrics you built so you can run the same one on the next release.

5. Try it as new work. Question four gave you a short list of things the new model can do that your old one couldn't. There's no switching decision here, because there's nothing to switch from, and the list usually splits two ways. Some items are builds – so pick the one or two with the clearest business case, give each an owner and a couple of weeks, and judge them on the output. The rest are just capabilities your people can start using as soon as they know they exist. Astra being good at computer use is the obvious example – that doesn't require a project, just education on what it's now good for and where it fits into their week.

The next model release will probably arrive before you’ve finished digesting this one. Build these evaluation habits now, and each release becomes easier to assess - less on published benchmarks and more on the specifics of your business. You’ll know what to switch, what to leave alone, and what’s finally worth trying. And you’ll have evidence that it makes a difference in your business.

See you next week,

Michael
Your fellow Head of AI