What to Ask Before You Buy Any AI Employee

Demos are designed to go well. That is their job. The useful questions are the ones a demo cannot answer, and most of them are about what happens when things are not going well.

Here are ten. Several of them are awkward for us to answer too — that is deliberate. A vendor who welcomes all ten is telling you something.

1. What happens when it doesn't know?

The single most predictive question. Every system meets a question it cannot answer. What you need is a specific, described behaviour: it says it doesn't know and offers a handoff, or it escalates to a named person with the conversation attached.

The wrong answers are "it always knows" and any hesitation. Confident invention is the characteristic failure of this technology, and a vendor who has not thought hard about it has not built for it.

2. Can I watch it fail?

Ask for a live session where you type. Not a recorded demo, not a script. Then push: ask something oddly phrased, ask two things at once, ask about something adjacent that you do not offer.

You are not trying to break it for sport. You are finding out whether failure is graceful or embarrassing, because your customers will find the same edges within a week.

3. Where do the facts come from?

When it quotes a price or an availability, what is the source? A connected calendar and a maintained document are good answers. "It was trained on your business" is not an answer — it is a description of a process, and it leaves open whether the system is reading current data or reciting a stale snapshot.

4. What is it not allowed to do?

A good build has an explicit list. Ours refuses to give medical, legal, or financial advice, will not negotiate outside defined ranges, and will not confirm anything it cannot verify against a real source.

If a vendor cannot produce this list, one of two things is true: they have not defined the boundary, or there isn't one. Both are problems.

5. How does a human take over?

At some point someone needs a person. Find out what that path looks like, how fast it is, and — critically — whether the customer has to repeat themselves. Handing a person a conversation with no context is barely better than no handoff at all.

6. Who owns the configuration?

The knowledge base, the prompts, the logic — if you leave, do you take them? Can you export them? Can you read them without the vendor's software?

This one rarely comes up during a happy sale and matters enormously eighteen months later. Ask it early, while the answer is cheap to give.

7. What does a change cost?

Your prices change. Your hours change. You add a service. What is the process, what is the turnaround, and what is the invoice?

The healthy answer is that routine content changes are something you do yourself, quickly, without a ticket. If every small edit is billable work, the running cost is much higher than the quoted one.

8. What is it genuinely bad at?

Ask it directly and listen to the shape of the answer. Every real system has weak areas. A vendor who names theirs unprompted has thought about fit; one who claims uniform excellence is either inexperienced or managing you.

For the record, ours: deep post-purchase support is early and we do not sell it as a support-team replacement. Scheduling is solid for straightforward booking against a connected calendar and deliberately narrow beyond that — multi-provider routing and complex recurring appointments are not something we would promise today. Lead qualification is where the system is strongest and where we would point you first.

9. What happens to the data?

Where are conversations stored, for how long, who can read them, and what happens if a customer asks you to delete theirs? If you are in healthcare, legal, or finance, get this in writing rather than in conversation.

Be specific about compliance too. "We can handle compliance" is not a claim anyone should accept unexamined — ask exactly which framework, and what the vendor is actually attesting to.

10. Who fixes it at 2 AM?

The system runs when your team does not. That is the point of it. So when it stops working at 2 AM — and eventually it will — what happens?

Two things matter: whether anyone finds out automatically, and what the customer sees in the meantime. A system that fails silently into a dead end costs you exactly the leads it was bought to capture. A system that fails into "I can't reach my system right now, here's a booking link and an email address" costs you very little.

Ask what monitoring exists, and ask what the failure state looks like from the customer's side. The second question is the one most vendors have not considered.

A short note on how to use this

You do not need perfect answers to all ten. You need answers that are specific, and a vendor who does not flinch at the awkward ones.

Vagueness under direct questioning during a sales process does not improve after the invoice.