Concept guide · How AI works

Good-looking is not the same as correct.

Fluent writing is a presentation skill, not proof the answer is right. The habit that catches a confident wrong answer is an eval: a small, fixed set of test cases, each with a known right answer, that you re-run whenever anything changes.

Keep known examples and rerun them after changes. They can catch the failures you included; passing does not prove every future answer will be correct.

What it actually is

OpenAI's documentation defines it simply: "Evaluations (often called evals) test model outputs to ensure they meet style and content criteria that you specify." A grader decides whether each output is correct, and the run is tallied as passed, failed or errored. It is the same idea as marking against an answer key.

A normal example

You use AI to pull the total from an invoice. You gather ten real past invoices where you already know the ten correct totals. Nine answers look perfect. On invoice seven the total sits in a footer, and the model returns the subtotal instead. Confident, well-formatted, wrong, and caught only because you held the answer key.

The proof: ten cases where confidence fails

A single impressive result proves nothing. A fixed set of ten, scored pass or fail against known answers, shows which of those examples passed and which failed. It does not measure accuracy on every task you may give it. That is the inspectable object: ten rows, each marked pass or fail, that you keep and re-run.

Not a benchmark. Not a hundred cases. The handful that would catch the mistake you cannot afford.

Try it now

Build a ten-case set

0 of 4 done

Pick one repeated task where the answer can be checked.
Collect ten real past examples where you know the right result, including the two awkward ones that have caught you out before.
Run all ten and mark each pass or fail against the known answer.
Keep the set and re-run it after any change of prompt, model or tool.

Your notesStays on this page. Nothing you type here is saved or sent.

Check: If every case passes but you quietly left the hard ones out, the set is flattering you. A set that cannot fail is not testing anything.

Ask Bit to check

Ask Bit opens with a first line written for you. Your notes are not put in the link; copy and paste them in.

The decision this gives you

Build the smallest eval that would catch your worst failure, and run it before you trust a change. It is the difference between "this looked fine the once I tried it" and "this passes the ten cases that matter, including the two that used to break."

Common mistake

Judging quality by asking the model "are you sure?" or by how polished the writing is. Both measure confidence, not correctness. A model will restate a wrong answer just as fluently the second time.

The human bit

You define what "correct" means for your work, and you own the answer key. A model cannot grade itself into being right, because it is the thing being tested.

Where this stops being exact

This explainer is anchored to the concept, a small pass-or-fail set anyone can keep, not to any one vendor's dashboard. OpenAI's documentation states it "is deprecating the Evals platform," which becomes read-only for existing users on 31 October 2026 and is "scheduled to shut down on November 30, 2026." The habit outlasts the product. A spreadsheet of ten cases and their right answers is an eval.

Official source

OpenAI: evaluations (evals) guide ↗

The deprecation dates above were checked against OpenAI's notice on 19 August 2026.

Use this in the product

Try the research task →Read: how to check an AI answer →

Next conceptAn agent is a model using tools in a loop