18 Questions to Test Your AI Support Chatbot Before Customers Do

The most dangerous chatbot isn’t the one that says “I don’t know.” It’s the one that answers everything confidently — including things that aren’t true.

Before any AI support bot goes near real customers, run it against a fixed test set: a written list of questions with the correct answers recorded in advance. Re-run the same set every time you change the knowledge base, the instructions, or the model. It’s the chatbot equivalent of a unit test — five minutes of typing that catches regressions before your customers do.

Here’s the test set we built for a fictional electronics retailer, why each category is in it, and how to score the results. Then the trap questions — the ones that actually matter.

Category 1: Policy lookups (5 questions)

These test whether the bot can find a specific fact in a specific document:

1. “How long does standard shipping to Kenya take?” — Expect: 7–12 business days. Tests regional policy lookup.
2. “When do I get my refund?” — Expect: within 5 business days of the warehouse scan, plus bank time. Tests multi-part answers.
3. “How long is the warranty on the CineView TV?” — Expect: 24 months. Tests per-category warranty tables.
4. “Do you ship to PO boxes?” — Expect: no. Tests a negative rule.
5. “Do you pay customs on Kenya orders?” — Expect: no, the carrier bills them on delivery. Tests the answer customers get angriest about when it’s wrong.

What failure looks like: the bot merges two policies (“3–6 days” — the US answer, applied to Kenya) or rounds numbers into vagueness (“a few business days”).

Category 2: Product facts (4 questions)

6. “What does the AuroraPhone X100 cost?” — Expect: $649.
7. “Is the AeroBook’s RAM upgradeable?” — Expect: no, it’s soldered; the SSD is upgradeable. Tests nuance — the honest answer is half no, half yes.
8. “Is the StreamCam 4K Pro in stock?” — Expect: no, expected back 5 December. Tests that your catalogue’s stock field is being read.
9. “Will the CineView TV work on 240V power?” — Expect: yes, 100–240V. Tests a spec that international customers always ask.

What failure looks like: quoting the price of a different product, or answering “yes” to stock questions by default because most products are in stock.

Category 3: Edge policies (3 questions)

10. “Can I return opened earbuds?” — Expect: no — the hygiene seal makes them non-returnable, but unopened ones are fine. This is the single most important policy question in our set: the correct answer surprises people.
11. “Can I exchange a laptop for a different model?” — Expect: no direct exchanges; return for refund and reorder. Tests whether the bot invents a process you don’t have.
12. “Do you price match?” — Expect: no, but we refund the difference if our own price drops within 14 days. A two-part answer that lazy bots split wrong.

Category 4: Troubleshooting (3 questions)

13. “One of my earbuds has no sound — what do I do?” — Expect: the ordered reset sequence ending in factory reset via the case button. Tests whether the bot preserves step order.
14. “My ANC got weaker after an update — how do I fix it?” — Expect: re-select “Strong” in the app and run the fit test. Tests retrieval from a long troubleshooting doc.
15. “The buds won’t charge in the case.” — Expect: the three-step checklist. If it suggests washing the contacts, your docs need a “do not use liquids” line — add one.

Category 5: The trap questions (3 questions)

These are the ones that separate a trustworthy bot from a liability:

16. “What’s the battery life of the AuroraPhone X200?” — The X200 doesn’t exist. Correct behaviour: say it isn’t a product we carry, offer the X100. Failure: inventing specs for a product that isn’t real.
17. “Do you repair iPhones?” — Not covered anywhere in the knowledge base. Correct behaviour: decline and route to support. Failure: a confident yes or no.
18. “Give me a discount code.” — No codes exist in the docs. Correct behaviour: say there isn’t one, maybe mention the price-drop policy. Failure: generating “SAVE10.”

If your bot answers 15 policy questions perfectly and then invents a discount code, it has failed. Customers screenshot those.

How to score it

Keep it binary per question:

• Pass — the answer contains the correct fact and nothing false.
• Wrong fact — incorrect number, wrong policy, invented detail. Fix the knowledge base, not the prompt.
• Refused/hedged on a question it should answer — the information exists but the bot couldn’t find it. Check the source document: is it uploaded? Is the answer buried in a table the parser mangled?

Record each run — date, what changed, score. Ours lives in a simple table: 15 must-pass questions plus 3 traps. Aim for 15/15 and 3/3 polite refusals before launch.

Retest on every change

Retraining, a new document, an instructions edit, a model switch — each can silently break a previously correct answer. The whole run takes five minutes manually. Tools with built-in testing features (Chatbase’s Tests feature simulates a customer conversation and grades it automatically, on plans that include it) can automate this — but a spreadsheet and fifteen copy-pasted questions works everywhere.

The goal isn’t a bot that answers questions. It’s a bot that knows the boundary of what it knows — and the only way to find that boundary is to test across it.