How an AI tool is built and tested
About 25 minutes · plain language
Before you trust an AI tool, you need to know how it was built and how it was tested. You do not need to be a statistician. This lesson explains, in plain words, the few ideas that tell you most of what you need to know.
- Read sensitivity, specificity, discrimination and calibration in an AI paper.
- Recognise data leakage and why external validation matters.
Key terms
- Model
- The result of training: a mathematical rule that turns input data into an output, such as a risk score.Example: A model that turns age, vital signs and blood results into a sepsis risk.
- Training and test data
- Training data are used to build the model; test data are kept apart to check how well it works.Example: 70% of records used to train, 30% held back to test.
- Sensitivity
- Of all the people who really have the condition, the proportion the tool correctly identifies.Example: Sensitivity 90%: the tool catches 90 of 100 patients who will deteriorate.
- Specificity
- Of all the people who do not have the condition, the proportion the tool correctly leaves alone.Example: Specificity 80%: 20 of 100 stable patients get a false alarm.
- False positive / false negative
- A false positive is an alarm for someone who is fine; a false negative is a missed case.Example: An early-warning score that misses a septic patient gives a false negative.
- Positive predictive value (PPV)
- Of all the alerts a tool raises, the proportion that are true. It falls when the condition is rare.Example: PPV 15%: of 100 alerts, about 15 are real and 85 are false alarms.
- Discrimination (AUC or C-statistic)
- How well a model ranks higher-risk people above lower-risk people. 0.5 is no better than chance; 1.0 is perfect.Example: AUC 0.80 means good, not perfect, ranking.
- Calibration
- Whether predicted risks match what really happens.Example: Of 100 patients given a 20% risk, about 20 should have the event.
- Internal validation
- Testing the model on data from the same source it was built on.Example: Built and tested in the same hospital.
- External validation
- Testing the model on data from a different place or time.Example: A model from a US hospital tested in an Omani hospital.
- Data leakage
- When information from the test data, or from after the outcome, slips into training and makes results look better than they are.Example: The same patient appears in both training and test data.
Data come first
A model learns from past data. If those data come from one hospital, one population or one period, the model learns their habits too: how staff record things, which patients were admitted, and gaps in the records. A model can only be as good, and as fair, as the data behind it.
Reading the numbers
Most AI studies report four kinds of number. Sensitivity and specificity come from the 2×2 table in the figure. There is always a trade-off: set the alarm to catch more true cases and it will also raise more false alarms.
Discrimination (AUC) tells you how well the model ranks patients. Calibration tells you whether its risk numbers can be taken at face value. A model can rank patients well but still say “30% risk” when the true risk is 10%. If staff act on the number, poor calibration matters.
A fifth number matters most to the person who receives the alert: the positive predictive value (PPV), the share of alerts that are true. It depends on how common the condition is. If 2 in 100 patients will deteriorate, a tool with 90% sensitivity and 90% specificity raises about 12 alerts per 100 patients, and only about 2 of them are true: a PPV of about 15%. This is why a tool that looks accurate on paper can still flood a ward with false alarms (Lesson 10).
Why external validation matters
Performance usually drops when a model is used somewhere new. Patients, equipment and recording habits differ. A widely used commercial sepsis model is a well-known example: when an independent team tested it in their hospital, it performed much worse than its developer had reported and missed about two-thirds of patients with sepsis. So the key question is not “how accurate is it?” but “how accurate is it here, for patients like ours?”
Watch for data leakage
If the test data are not truly separate from the training data, the results look too good. Common causes are the same patient appearing in both sets, duplicate records, or using information recorded after the outcome. Good studies explain exactly how they kept the data apart, and guidelines such as TRIPOD+AI ask them to.
What good reporting looks like
TRIPOD+AI lists 27 items that a prediction-model study should report, from the data source to calibration. PROBAST+AI helps reviewers judge whether a study is at risk of bias. You can use them as a reading checklist even if you never build a model.
Case: the 0.90 that became 0.70
A company offers a pressure-injury model with an AUC of 0.90 from its development hospital. The nursing team asks three questions: Was it externally validated? Is calibration reported? Were patients like ours, mostly older adults with diabetes, in the data? The answers are “no”, “no” and “not sure”. They run a three-month silent test: the model scores patients but nobody sees the scores. In their wards its AUC is 0.70 and it overestimates risk. They decide not to use it yet.
Common mistakes
- Judging a model by AUC alone.
- Accepting results with no external validation.
- Ignoring how the data were split.
Check your understanding
1. A tool has high sensitivity but low specificity. What will staff notice?
It will catch most true cases but also raise many false alarms.
2. What is the difference between internal and external validation?
Internal uses data from the same source the model was built on; external uses data from a different place or time.
3. Why does calibration matter if staff look at the risk number?
Because a poorly calibrated model gives risk numbers that do not match reality, so staff may over- or under-react.
Try it
- Find one AI prediction study in your field. Locate its sensitivity, specificity, AUC and calibration.
- Check whether it reports external validation, using TRIPOD+AI as a guide.
Remember
- A model is only as good as its data.
- Look for sensitivity, specificity, discrimination and calibration, not just one number.
- Ask: has it been tested somewhere like here?
- Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi.org
- Moons KGM, Damen JAA, Kaul T, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. doi.org
- World Health Organization. Generating evidence for artificial intelligence-based medical devices: a framework for training, validation and evaluation. Geneva: WHO; 2021. www.who.int
- US Food and Drug Administration, Health Canada, Medicines and Healthcare products Regulatory Agency. Good machine learning practice for medical device development: guiding principles. 2021. www.gov.uk
- Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181(8):1065-70. pubmed.ncbi.nlm.nih.gov
Lesson text is our own plain-language summary of these sources; it is not endorsed by their authors. Check the original before relying on any detail.