The label stopped carrying information
“AI-powered” now describes everything from a genuine reasoning feature to a keyword matcher with a new badge. The word tells you nothing, so the evaluation has to be built from questions about behaviour instead.
Ask the vocabulary question first, because it costs nothing: is this a chatbot, a copilot, or an agent — what can it change, and what does it do on its own? A vendor who cannot answer crisply is either selling something less capable than implied or something more autonomous than you intended to buy, and the three carry very different risk profiles.
1. What does it do when it doesn’t know?
The most revealing question you can ask. A vendor whose product has a supported way to decline and escalate has thought about production; one whose demo always answers has not.
Push for specifics: what triggers the escalation, where does it go, and what does the customer see. “It hands off to a human” is a slide; “it routes to your queue with the transcript and retrieved sources attached when the topic is on your stakes list” is a product.
2. Can I see it fail?
Ask them to run it on your messiest real inputs, not their examples. Reluctance here is the single strongest signal in the whole process, because every product looks good on curated data.
Prepare the inputs in advance and make them genuinely awkward: the ambiguous request, the angry customer, the case with missing information, the one your own team argues about. Watching a vendor’s product handle the case your staff find hard tells you more than a month of reference calls.
Ask to see it handle your worst data. How they respond tells you more than the answer would.
3. What happens to my data?
Retention, region, training use, sub-processors, and deletion. These are contractual answers with the same weight as any dependency on someone else’s model, and vague responses should be treated as no.
Two follow-ups people forget. Can you delete a specific customer’s data on request across everything including logs and any evaluation sets — the obligation that outlives the feature. And which underlying provider are they using, since your data inherits that provider’s terms as well as theirs.
4. How do I measure whether it works?
A vendor who can tell you which numbers to watch and how to get them is confident. One who offers only usage statistics is selling activity — the same distinction as deflection rate versus confirmed resolution.
Ask whether you can export the raw interactions to measure it yourself. A product that only reports its own performance through its own dashboard is asking you to take its word for the thing you are paying to verify.
5. Who reviews the output, and how?
Ask what the review workflow looks like in practice and how much staff time it needs per week. That is the hidden cost that determines whether the tool pays back, and it rarely appears in pricing.
Be specific about who that person is in your organisation, because the answer is often nobody. A tool requiring five hours a week of expert review from a team with no spare capacity will be adopted, unreviewed, and quietly degrade — the ownership gap arriving pre-installed.
6. What if we leave?
Whether your data, corrections and configuration come with you. The corrections especially — they are the accumulated asset, and a product that keeps them has locked you in more effectively than any contract term.
Two questions about the vendor, not the product
How long has this specific feature been in production with paying customers, and can you speak to one at your size. Many AI products are months old, which is not disqualifying but changes what you should expect and what you should commit to contractually.
And what happens at renewal if their underlying model costs change. Vendors reselling capacity are exposed to pricing they don’t control, and that exposure eventually reaches you.
How to run the evaluation
Score candidates on your own inputs rather than on feature lists — the same evaluation set you would write before building anything works for buying, and it makes two vendors comparable on something other than presentation.
Then buy the smallest useful commitment. A short paid pilot on one workflow with a defined stopping criterion tells you more than a year’s contract and costs less to exit — the same shape as a two-week proof of concept, with someone else doing the building.