Hello Messorians,
Last quarter I sat through a vendor pitch for an AI tool that was going to "revolutionize the patient journey through generative clinical intelligence." About 40 minutes in, I still couldn't tell you what the product actually did. By the end of the hour, I realized that was the point.
This is the healthcare AI moment. Real, transformative tools exist. They are deployed at scale. They are changing how clinicians work, and they are also being sold alongside a flood of vaporware, repackaged dashboards, and PowerPoint products that will not survive their first contract renewal.
If you are in a position to recommend, buy, deploy, or evaluate AI in healthcare, and increasingly most of you are, you need a filter. This issue is the one I use.
Here are 7 questions. Memorize them and ask them in every vendor meeting. The vendors who can answer cleanly are the ones worth your time, and the vendors who deflect are doing you a favor by exposing themselves early.
1. What specific clinical workflow does this touch, and at what step?
Real tools name the workflow. Hype tools talk about "transforming clinical operations." If a vendor cannot describe the exact moment in the workflow where their tool inserts itself, the tool does not have a workflow. It has a marketing deck.
2. Where is the validation study, and what does it actually say?
Ask for the peer-reviewed validation, not the case study, not the white paper, and not the testimonial video. The peer-reviewed paper, with sample size, setting, comparator group, primary endpoint, and effect size.
If they don't have one, they're not ready for enterprise deployment. If they do, read it carefully.
3. How does it integrate, specifically?
The right answers sound like: "We're an Epic Connection Hub partner with a published integration through SMART on FHIR." Or: "We consume HL7 v2 ADT messages and write back through a CDS Hooks endpoint." Or: "Our app is published in the Showroom and uses USCDI v3 data elements."
The wrong answer is: "We integrate with all major EHRs." Translation: we have a CSV exporter and a hope.
4. Where exactly is the human in the loop?
Every healthcare AI tool needs a human checkpoint. Real vendors can show you the exact screen where the human reviews, the exact decision the human makes, and the audit trail of the override. Hype vendors wave at "human oversight" without ever showing you the interface.
If the vendor cannot show you the override workflow on the demo, the override workflow does not exist.
5. What happens when the AI is wrong?
Real vendors have an answer to this question. They know the false positive rate, they know what error categories are most common, they have a feedback mechanism for clinicians to flag bad outputs, and they can tell you exactly how they retrain.
Hype vendors say "our model is highly accurate" and change the subject. Push back when they say this because every model is wrong sometimes. The question is whether the system is built to catch it.
6. Show me the ROI math, with assumptions.
Real vendors have a model. They will walk you through it. "12 minutes saved per visit, 3,200 visits per provider per year, $X per minute of provider time, two-year ramp to full adoption, here's the spreadsheet." You can argue with the assumptions. That is the point.
Hype vendors offer a "value proposition" with no math. If you cannot reproduce the ROI on your own with their assumptions, the ROI is not real.
7. Ask the CMIO question.
This is the one that separates the merely good vendors from the great ones. Ask: "What was the last clinician complaint you got about this tool, and how did you address it?"
Good vendors have a specific answer. They will name the issue, name the fix, and tell you what release it shipped in. Great vendors will tell you about a complaint you didn't know to worry about. Bad vendors will tell you that clinicians love the product. That is the answer of a company that does not talk to clinicians.
The meta-test.
If a vendor's pitch deck leans on the phrase "generative AI agentic clinical platform" and they cannot finish the seven questions above in a single one-hour meeting, the product is not ready. Send them away. They will be back in 18 months with something real, or they will not be back at all. Either way, you have your answer.
A hype filter is not negativity. It’s about taking the field seriously. Vendors building real tools are doing important work in healthcare and should be judged carefully. Vendors selling buzzwords waste your time, budget, and clinical staff’s patience. Knowing the difference is now part of the job.
Memorize the 7 questions because they will save you a year of regret.
Until next week.
Sharif
Founder, Informessor
