AI evaluation & stress-testing
Find out where your AI breaks, before your customers do.
Independent testing for LLM apps, RAG, agents and machine-learning models. In 2–4 weeks you know how often it is wrong, where it can be attacked, and exactly what to fix first.
Fixed scope · fixed price · works on systems we didn’t build
Who it’s for
Most AI ships on a handful of manual checks. That’s how failures reach customers.
You’re about to launch an LLM feature
An assistant, a RAG search, a document workflow or an agent, and nobody can say how often it is wrong.
Your AI is live and you’re nervous
Customers found a hallucination, a model update changed behaviour, or a jailbreak made the rounds.
You rely on a predictive model
Credit, risk, pricing, fraud or forecasting models where a confident wrong answer costs real money.
You need evidence for someone else
A client, an investor, an auditor or the EU AI Act asks how you know the system is safe to use.
What we test
Nine ways an AI system fails. We check all of them.
Every audit is tailored, but these are the questions we answer for nearly every system.
- 01
Task accuracy
Measured on a test set built from your real cases, with a target agreed up front.
- 02
Hallucination & grounding
Does every claim trace back to your sources? Where does it make things up?
- 03
Prompt injection & jailbreaks
Hundreds of adversarial prompts, including attacks hidden in documents and emails.
- 04
Agents & tool use
Wrong tools, unsafe actions, loops and permission leaks, tested end to end.
- 05
Policy & tone drift
Does it stay inside your rules and brand voice after every prompt or model change?
- 06
Calibrated uncertainty
Confidence scores that mean what they say, and conformal prediction for guaranteed error bounds.
- 07
Robustness & drift
Edge cases, messy inputs, new data distributions and the slow decay after launch.
- 08
Fairness
Error rates broken down by the groups that matter for your use case and your regulator.
- 09
Cost & latency
What each answer costs and how long it takes, at the volume you actually run.
Our edge
Not just “it seems fine”. A guarantee.
Most testing tells you how a system did on yesterday’s examples. We also use conformal prediction, a statistical method from our research, so the system knows when it is unsure and its error rate comes with a mathematical bound.
It works even on closed models like ChatGPT. When the model is confident, it answers. When it isn’t, it hands the case to a person instead of guessing. Our team has used it to guarantee an error rate under 2% in an audit system.
Ask the model
$ sample × 20 · 0/20
- Paris0%
- Lyon0%
- Nice0%
Line = calibrated threshold: answers sampled at least 12% of the time make the set.
✓ ANSWER Paris · the set contains the right answer 95% of the time
Pick a question and a coverage level. Higher coverage, bigger sets: the model admits what it doesn’t know.
AI System Audit
Three weeks from “we hope” to “we know”.
Week 1
Scope & test design
We agree what “good” means in numbers and build a test set from your real cases and known failure modes.
Week 2
Attack & measure
We run the scenario, adversarial and edge-case suites and measure every metric against its target.
Week 3
Report & fixes
A plain-language report: what fails, how often, what it could cost, and fixes ranked by risk and effort.
After
Release gate
Optional: we wire the tests into your pipeline, so every future update has to pass before it ships.
2–4 weeks · fixed scope · fixed price
You get
- Evaluation suite tailored to your system, yours to keep
- Failure-mode report with examples and estimated impact
- Ranked fix list, with the quick wins first
- Release-gate thresholds and a monitoring plan
- Executive summary you can share with clients or auditors
Questions
Before you ask.
Do you need access to our model or its weights?
No. We can test any system through the same interface your users have, including closed models such as GPT, Claude or Gemini behind an API. Access to logs and prompts makes the audit faster, but it is not required.
Can you evaluate a system you didn’t build?
Yes, that is most of our evaluation work. We are independent, so the report tells you how the system behaves, not how good a vendor would like it to look.
How much does an audit cost?
It is a fixed price, agreed after a free 30-minute call once we know the size of the system and how much of it needs testing. No open-ended retainer.
Does this help with the EU AI Act?
It gives you the technical evidence the Act asks for: test results, accuracy and robustness measurements, risk documentation and monitoring. We are not lawyers, so we work alongside your legal advice rather than replacing it.
What about our data?
We sign an NDA before we see anything, work in your environment where possible, and use only the data the audit needs. Nothing is used to train other systems.
What happens after the audit?
You can fix things in-house using the ranked list, or we fix them with you. Either way the test suite stays with you, and we can turn it into a release gate that runs on every update.
■ Free 30-minute call
Is your AI safe to ship?
Tell us what it does and what worries you. In 30 minutes you’ll know what we would test first and what an audit would cost.