Accuracy is not enough: how to know when your AI can be trusted

“If your model says there’s a 50% chance of rain tomorrow and it never rains, congratulations: you’ve built a weather app.”

Most machine-learning projects are judged by one number: accuracy. It is a useful number. It is also the reason a lot of AI that looks fine in a notebook goes wrong in front of customers.

This article is the written version of our talk at PyCon DE & PyData 2025, Accuracy is not Enough: Building Trustworthy AI with Conformal Prediction. It walks through three ideas that we use in almost every system we build or audit: calibrated probabilities, conformal prediction and knowing when to hand a case to a human. The full recording is below, and all the code is open source in the talk repository.

An 80% model that says “90%” and means “50%”

Imagine a data scientist at a hospital. Call him John. Oncologists want to know, before starting immunotherapy, how likely a patient is to develop a serious immune-related side effect. They hand John historical patient data with a label: did the patient develop the side effect or not.

John does everything right by the textbook. He cleans the data, engineers features and trains a classifier that is 80% accurate. Since the doctors asked for a probability, not a yes/no, he calls scikit-learn’s predict_proba.

A new patient comes in. The model says 90% chance of a side effect. The patient goes ahead with treatment anyway and is fine. Bad luck, John thinks: there was a 10% chance of that. Then more patients arrive with the same profile and the same 90% prediction. Out of 19 of them, roughly half develop the side effect.

The model said 90%. Reality said 50%. Nothing about the 80% accuracy warned anyone. The problem is not the model’s ranking of patients; it is that its scores are not probabilities, even though the method is called predict_proba.

Step one: calibrate

A model is calibrated when, among all the cases where it says 70%, the event happens about 70% of the time. You can see it with a calibration curve: bin the predictions (0–10%, 10–20% and so on), and for each bin plot the predicted probability against how often the event actually happened. A perfect model sits on the diagonal. John’s model bends far below it at the top: when it says 90%, the truth is closer to 50%.

To put a single number on it, use the Brier score, the average squared gap between the predicted probability and what happened (1 if the event occurred, 0 if not):

Brier = mean( (p − outcome)² )

Predicting 90% for something that didn’t happen costs 0.81; predicting 90% for something that did costs 0.01. Lower is better. One caveat: for very rare events the Brier score barely moves, so look at the curve as well.

Two ways to fix calibration:

• Isotonic regression learns a monotone mapping from scores to observed frequencies on held-out data. It is the classic fix, but on small data it overfits and often stays overconfident.
• Venn–Abers calibration fits two isotonic regressions, one assuming the new case is a 0 and one assuming it is a 1. The result is an interval for the probability with a statistical validity guarantee, and the width of the interval tells you how uncertain the model is. In our experiments it lands far closer to the diagonal than isotonic regression.

Calibrated probabilities are what make a model usable for decisions. Without them, a “90%” is a mood, not a measurement.

Step two: conformal prediction, a guarantee instead of a guess

Calibration makes the numbers honest on average. Conformal prediction goes further: it turns any model’s point prediction into a prediction set (or interval, for regression) that contains the true answer with a probability you choose, for example 95%.

Instead of “the answer is 4”, the model says “the answer is in {4, 7}, and sets built this way contain the right answer 95% of the time”. That is a guarantee, not a hope, and it needs remarkably little:

• One assumption: the data is exchangeable (roughly: new cases look like the ones you calibrated on).
• Any model, any task, any data distribution. It wraps around whatever you already have.

The recipe for the simplest version, split conformal prediction:

1. Pick a nonconformity score: how strange a candidate answer looks for this input. A common choice is 1 − predicted probability of that answer.
2. Compute the score of the true answer on a held-out calibration set.
3. Take the right quantile of those scores (about the 95th percentile for 95% coverage).
4. For a new input, include every candidate answer whose score is below that threshold.

In Python, with nothing but NumPy:

import numpy as np

# probs_cal: (n, K) predicted probabilities on a held-out calibration set
# y_cal:     (n,)   true labels;  alpha = 0.05 means 95% coverage
n = len(y_cal)
scores = 1 - probs_cal[np.arange(n), y_cal]           # nonconformity of the true label
level = np.ceil((n + 1) * (1 - alpha)) / n
qhat = np.quantile(scores, level, method="higher")

pred_sets = (1 - probs_test) <= qhat                  # (m, K) boolean prediction sets

You choose the coverage level. Ask for 100% and you get every possible answer, which is true and useless, like saying a die will land on one of its sides. A better model gives tighter sets; a perfect model would always return a single answer. The guarantee holds whatever score you choose; a good score just makes the sets smaller.

What the sets mean in practice

Back at the hospital, suppose there are three outcomes: no side effect, mild, or severe. At 95% coverage, a patient might get:

• {no side effect}: a confident, easy decision.
• {no side effect, mild}: still an easy decision for most patients.
• {mild, severe} or {no side effect, severe}: the model is genuinely unsure, and the set says so. This is the case a clinician should look at closely.

The size of the set is the uncertainty, expressed in a way doctors and managers understand.

The same idea for language models

Chatbots fail differently: they answer fluently and wrongly. When an airline’s chatbot invented a refund policy, a Canadian tribunal ordered the airline to honour it in 2024. The company was liable for what its AI said.

What if a language model could answer with a guarantee, or admit it isn’t sure?

Conformal Language Modeling (Quach et al., 2023) samples several answers from the model and keeps adding them to a set until a calibrated stopping rule guarantees, at your chosen level, that at least one acceptable answer is in it. It can also break long answers into parts and flag the specific sentences likely to be hallucinated. It is available in the TorchCP library. The catch: it needs the model’s internal probabilities (logits), so it only works with models you host yourself.

What about ChatGPT? The paper API Is Enough (Su et al., 2024) gets conformal guarantees using only the API. Ask the same question many times and score each distinct answer with three signals:

• frequency: how often the model gave this answer;
• normalised entropy: how scattered the answers are overall;
• semantic similarity to the most frequent answer.

score = 1 − frequency + λ₁ · entropy − λ₂ · similarity

The two weights let you tune how much each signal counts. Calibrate the threshold on a set of questions with known answers, and you get prediction sets with the usual guarantee, from a closed model. The authors didn’t publish code, so we wrote an open implementation.

Try a simplified version below (frequency only). The model is sampled 20 times; the line is the calibrated threshold. Raise the coverage and watch the sets grow on the harder questions:

Ask “What is the capital of France?” and you get {Paris} at 90%. Ask a trick question and you get two answers at once, which is exactly the signal you want: do not let the system answer this one alone. Open-ended questions (“What are the best ways to fix the economy?”) produce large sets, because there is no single right answer. Conformal methods work best on tasks with checkable answers, which happens to describe most business use cases.

Robots that know when to ask

The most intuitive example comes from robotics. In Robots That Ask For Help (Ren et al., 2023, with Google DeepMind), a person asks a robot: “Put the bowl in the microwave.” There is a metal bowl and a plastic bowl. The language-model planner rates both options as likely, so the conformal set contains two actions. Instead of guessing (and putting metal in a microwave), the robot asks: “Which bowl, the metal one or the plastic one?”

That is the whole philosophy in one line: when the set has more than one answer, ask a human. You set the coverage level, so you control the trade-off between automation and safety.

Limits worth knowing

• Exchangeability matters. If live data drifts far from the calibration data, the guarantee weakens. Monitor and recalibrate.
• Coverage is on average by default. Basic conformal prediction guarantees 95% across all cases, not within every subgroup. Variants such as class-conditional (Mondrian) conformal prediction fix that when it matters, for example for a rare but costly class.
• Weak models give wide sets. That is honest, not a bug: the method won’t make a poor model look good.
• Some questions have no single right answer. For open-ended generation, use other evaluation methods alongside it.

What this means if you run AI in production

All models are wrong; some are useful. The useful ones share three habits:

1. Probabilities that mean what they say. Calibrate before anyone makes decisions on a score.
2. Uncertainty you can act on. Prediction sets and intervals tell you when the model is confident and when it isn’t, with a guarantee you choose.
3. A decision rule for doubt. Confident cases are automated; uncertain ones go to a person. In one audit system our team built this way, the error rate was guaranteed to stay under 2% while manual review fell by about two thirds.

That is the standard we hold every system to before it ships, and the first thing we check when we audit someone else’s AI.

Go deeper

• Code from the talk: github.com/aiva00/pydata_conformal_prediction (Venn–Abers calibration, conformal prediction on images, conformal language modelling, and our API Is Enough implementation).
• Talk page: PyCon DE & PyData 2025.
• Libraries: MAPIE for tabular data, TorchCP for deep learning and LLMs, crepes for conformal predictive distributions.
• Reading: Angelopoulos & Bates, A Gentle Introduction to Conformal Prediction, and Valeriy Manokhin’s Awesome Conformal Prediction list.