Judging an AI tool before you use it
About 20 minutes · plain language
Before any AI tool is used in care, teaching or research, someone should ask the right questions. This lesson gives you ten questions that cover most of what matters, and shows how to use them with vendors and colleagues.
- Ask ten questions of any AI tool before it is used in care, teaching or research.
Key terms
- Intended use
- Exactly what the tool is for, for whom, and in which setting.Example: “Predicts fall risk in adult medical inpatients.”
- Vendor
- The company that makes or sells a tool.Example: A software company offering an AI scheduling system.
- Local evaluation
- Testing a tool in your own setting before relying on it.Example: A silent trial on your ward.
- Performance drift
- When a tool becomes less accurate over time because patients, practice or data change.Example: A model trained before a new documentation system was introduced.
- Monitoring
- Regularly checking that a tool still works safely after launch.Example: Monthly review of missed cases.
- What exactly is it for, and for whom?
- Who made it, and who pays for it?
- What data was it trained on?
- Where and how was it tested?
- How well does it work in each patient group?
- What happens when it is wrong, and who will notice?
- Does it meet the law and approvals where we work?
- Where do our data go?
- Who is trained to use and override it?
- How will we monitor and update it?
The ten questions
The figure lists ten questions. The first five ask what the tool is for, who made it, what data it learned from, how it was tested, and whether it works equally well in each patient group. The second five ask what happens when it is wrong, whether it meets the law, where data go, who is trained to override it, and how it will be monitored.
How to use them
Ask them of vendors, colleagues and yourself. A vendor who cannot answer questions 3 to 5 (data, testing, performance in each group) has not shown that the tool works for your patients. Checklists from regulators and coalitions, such as WHO’s regulatory considerations and the CHAI assurance guide, go into more detail.
Look for evidence in your setting
Published accuracy from elsewhere is a starting point, not proof. A short local evaluation, such as a silent trial in which the tool runs without affecting care, shows how it performs with your patients and your workflow.
Plan monitoring from day one
Performance can drift as patients, practice or recording change. Before launch, decide what you will measure, how often, who will look, and what result will make you pause the tool.
Case: an essay-feedback tool for a nursing faculty
A faculty considers an AI tool that gives students feedback on essays. Using the ten questions, staff find the vendor cannot say where student essays are stored or how well the tool works for students writing in their second language. The faculty runs a small pilot with student consent, compares AI feedback with teachers’ feedback, and agrees a data-protection contract before deciding.
Common mistakes
- Buying a tool on the strength of a demonstration.
- No plan for monitoring after launch.
- Not asking where the data go.
Check your understanding
1. A vendor says “our tool is 95% accurate” but cannot say where it was tested. What is missing?
Evidence of how and where it was tested, and how it performs in your patient groups.
2. What is performance drift?
When a tool becomes less accurate over time because patients, practice or data change.
Try it
- Use the ten questions on one tool you already use. Which can you not answer? Who could?
Remember
- Ask the ten questions before adopting any AI tool.
- Test locally; published accuracy is a starting point.
- Plan monitoring before launch.
- World Health Organization. Regulatory considerations on artificial intelligence for health. Geneva: WHO; 2023. www.who.int
- Coalition for Health AI. Assurance standards guide (draft for public comment). CHAI; 2024. chai.org
- International Medical Device Regulators Forum. Good machine learning practice for medical device development: guiding principles (IMDRF/AIML WG/N88 FINAL:2025). IMDRF; 2025. imdrf.org
- Lekadir K, Frangi AF, Porras AR, et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ. 2025;388:e081554. doi.org
- National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg (MD): NIST; 2023. doi.org
- Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181(8):1065-70. pubmed.ncbi.nlm.nih.gov
Lesson text is our own plain-language summary of these sources; it is not endorsed by their authors. Check the original before relying on any detail.