AI evals: how we judge our AI's work, every morning.
What an eval is, why a business running Claude or ChatGPT needs one, and how we judge our own AI's work daily and turn a bad finding into a fixed skill. Published 18 October 2026.
The short answer
An AI eval is a repeatable check of whether the AI did a piece of work well, judged on the same evidence every time. Sentry AI runs one every morning on its own company and on the client teams it looks after. It reads what was done the day before, never what was typed, picks at most one problem, waits to see it twice, then fixes the skill behind it with a small change that can be undone. The result is measured against the time the job took by hand in week one.
What an eval is, in plain words
A teacher marking homework against the same rubric every week is running an eval. So is a kitchen tasting every sauce before it leaves the pass. For AI it has three parts.
A fixed question
Did this piece of work come out right, and did it take less time than doing it by hand? The question stays the same every day, so the answers can be compared.
The same evidence every time
What was actually done: which skills ran, which tools failed, what people accepted or threw away. Not an impression, and not a sample someone picked.
A judgement you act on
Each eval ends in a decision: leave it, fix it, or ask a person. An eval that only produces a chart has not finished its job.
Why a business running Claude or ChatGPT needs one
Evals started as a tool for the labs that build models. The same idea matters more inside a business, because the question there is not whether the model is clever but whether your work got better.
Seats are not work
A seat that is paid for and opened tells you nothing about whether the job got faster. The gap between licences bought and work done is where most AI spend goes quiet.
Quality moves when anything moves
A new model, a changed prompt or a new person on the team can make a workflow better or worse overnight. Without a daily check, you find out from a customer.
A policy cannot see practice
A written AI policy says what should happen. An eval shows what did, which is what a board, an auditor or a client's due diligence will ask for.
Time saved needs a before
You cannot claim AI saved time without a baseline taken before it arrived. Evals and the baseline are one practice, not two.
Every morning
How we judge our own AI's work
We run this on Sentry AI first, then on the teams we look after. Six steps, the same every day.
- Step 1
Read yesterday's work
Before the team starts, the eval reads how AI was used the day before: which skills loaded, which were opened and abandoned, which tools failed the same way twice, which jobs people still did by hand that a skill already covers. It reads what was done, never what was typed.
- Step 2
Pick one thing
At most one finding a day. A small daily change can be checked; a weekly pile of changes cannot.
- Step 3
Wait for it twice
A new finding is only a candidate. It has to happen again in a separate session to be confirmed, and if it goes quiet for three days it lapses. Finishing a confirmed finding always beats starting a new one.
- Step 4
Fix the skill
A confirmed finding becomes one small edit to the skill that does that job: the instructions the AI follows for that piece of work. The change is versioned and can be undone, and only skill and knowledge files can change this way.
- Step 5
Ask a person when a person decides
Some findings are not a skill problem but a decision: who should have access, whether a workflow should exist at all. Those go to the person who owns them as one plain question, at most one a week.
- Step 6
Write it down
Every finding carries its dates: first seen, confirmed, fixed, and whether it came back. That record is the audit trail, and it is how we know a fix worked.
From a bad finding to a fixed skill
Most findings are one of four patterns. Each points at a skill: the written instructions the AI follows for one job, such as a weekly report or a supplier email. Fixing the skill fixes the job for everyone who uses it, the next time they use it.
- A skill that loads and is then abandoned
- The same tool failing the same way twice
- A job done by hand that a skill already covers
- A question people keep asking that nobody has answered
- It has to be seen twice before anything changes
- One small change a day, never a rewrite
- Only skill and knowledge files can change this way
- Every change is versioned and can be undone
- Decisions about people and access always go to a person
The skills live in the company brain, next to the knowledge they draw on, so a fix made for one person reaches every seat that uses the same skill.
Quality, then outcomes
An eval is only worth it if the work got faster
Every engagement names up to three goals. In week one we time each job by hand with the person who does it, and the default target is twice as fast as that baseline. When a skill is running, the same job is timed again on a real run and a person approves the figure before it counts.
Three rules keep the record honest. A time saving needs a frequency, minutes a week or hours a month, or it counts for nothing. An estimate is labelled as an estimate. A job that got slower stays on the record as slower, because that is the finding. This is what our method for moving a team onto AI is measured against.
What an eval is not
It is not a dashboard of errors that someone is meant to watch, and it is not reading staff messages. The prompt text, the AI's answers and the contents of files are never captured; the eval works from what was done. The same line is drawn on our shadow AI page, and questions of accountability sit with AI governance.
Frequently asked questions
What is an AI eval?
An eval is a repeatable check of whether an AI did a piece of work well. It asks a fixed question of real work, gathers the same evidence each time, and gives a judgement you can compare from one day to the next. The point is that quality becomes something you measure, not something you feel.
Does a small business running Claude or ChatGPT need evals?
If people use AI for real work, yes. Buying seats does not tell you whether the work is getting faster or better, and quality changes when a model, a prompt or a process changes. A light daily eval tells you which workflows are working, which are being abandoned, and what to fix next.
Do your evals read what staff type into the AI?
No. Our evals read what was done, not what was said: which skills were loaded, which tools were called and whether they failed, and whether people accepted what the AI proposed. The text of prompts, the AI's answers and the contents of files are never captured.
How often do you run them?
Every morning, before the team starts, on our own company first and on the client teams we look after. Each run picks at most one thing to fix, so changes stay small and easy to check.
What happens when an eval finds a problem?
It waits to see the problem a second time in a separate session, so one bad afternoon does not change anything. Once confirmed, the fix is one small change to the skill that does that job, recorded with a date and the evidence, and it can be undone. Anything a person has to decide goes to that person as a plain question instead.
How do evals connect to time saved?
Every engagement starts by timing the job by hand in week one. The target is usually twice as fast. When a skill is running, the same job is timed again, and a person approves the figure before it counts. A unit with no frequency counts for nothing, and a job that got slower stays on the record as slower.
Do you know how well your AI worked yesterday?
In a short call we will show you what a daily eval would look at in your business, and which job we would time first.
Ask an AI about this page
Opens the assistant with this page loaded: read it, summarise it, cite it.