To test a chatbot, write down what it should do, turn each expectation into a test case with an input and a pass condition, run the set against every new version and score the failures by how much harm they could cause. The method is the same whether the bot follows scripts, recognises intents or generates text with a language model; only the way you judge a pass differs.
The wider practice of measuring a bot after launch is covered in testing and measuring chatbots. This page is the build-the-test-suite companion: what to write down before the first release.
Step 1: List what the bot is for
Start from the purpose in your design notes. Write a one-line job for each capability, such as “gives the status of an order” or “explains the returns policy”. Each line becomes a group of tests. If a capability has no line, nobody has decided what a correct answer is, which is a design problem rather than a testing one. The ideas in conversation design basics help here.
Step 2: Build a test matrix
A matrix keeps coverage visible. Put capabilities down the side and the types of test across the top:
| Capability | Typical request | Variant phrasing | Missing information | Out of scope | Hostile or odd input |
|---|---|---|---|---|---|
| Order status | “Where is my order?” | “Has my parcel shipped yet” | No order number given | Asking for a refund | Pasted code or long text |
| Returns policy | “Can I return this?” | “What if it doesn’t fit?” | No item mentioned | Legal advice request | Abuse, repeated characters |
| Human handover | “Let me talk to someone” | “agent please” | Unclear reason | Emergency situation | Empty message |
Each filled cell is a test case. Even a small bot with ten capabilities gets sixty cases from this simple grid.
Step 3: Write each case so a stranger can run it
A good test case has four parts:
- Input. The exact message or sequence of messages, including any earlier turns that set context.
- Starting state. Logged in or not, which language, what data exists.
- Expected behaviour. The pass condition, written as behaviour, not as an exact sentence, for example “asks for the order number” rather than a quoted reply.
- Severity. How bad a failure is: cosmetic, annoying, harmful or unsafe.
For rule-based and intent-based bots you can often check for the exact intent or path. For generated replies, describe what must be true (names the correct policy, does not invent a link, offers a human) and what must not occur (promises a refund, gives medical advice).
Step 4: Cover five families of tests
Happy paths
The common requests in the plainest words. They should all pass before anything else is considered.
Variation and mess
Typos, slang, mixed languages, very short and very long messages, several questions in one message, and changes of mind halfway through. The point is to test whether the bot recognises what the person is doing in the conversation. The idea of dialogue acts (greeting, request, confirmation, correction) gives a useful vocabulary for this.
Failure handling
Ask things the bot cannot answer. It should say so plainly, offer a next step and never loop. Count how many turns it takes to reach a human.
Safety and sensitive topics
Decide in advance how the bot should react to distress, threats, requests for medical, legal or financial advice, and personal data. Test those scripts, and check that the human route works.
Adversarial input
For language-model bots, include attempts to make the system ignore its instructions, reveal hidden prompts or leak data. The OWASP Top 10 for LLM Applications (2025 edition) lists prompt injection first, followed by sensitive information disclosure, and also names risks such as system prompt leakage and misinformation. Keep a standing set of these prompts and rerun it after every model or prompt change.
Step 5: Score results, not just pass or fail
A simple scale keeps debate short:
- 0, unsafe: gives dangerous advice or exposes private data. Blocks release.
- 1, harmful: wrong information on something important, such as a fee, a deadline or a legal right. Fix before release.
- 2, poor: unhelpful but safe, such as a dead end. Fix soon.
- 3, minor: awkward tone or formatting.
Report the number of failures at each level for each version. A rising count of severity 1 failures matters more than a falling count of severity 3.
Step 6: Test beyond the words
Interface and data matter too.
- Accessibility. The web accessibility guidelines WCAG 2.2, a W3C Recommendation dated 12 December 2024, include requirements such as making all functionality operable by keyboard. Try the chat window with a keyboard only and with a screen reader.
- Transparency. Tell people they are talking to software. The EU AI Act, in Article 50(1), requires providers of AI systems that interact directly with people to design them so that people are informed they are dealing with an AI, unless that is obvious from the context. If you operate in or serve the EU, take legal advice on how it applies to you.
- Privacy. Check what the bot stores, for how long and who can read transcripts. Avoid asking for sensitive details in free text.
- Performance. Test slow connections, long conversations and timeouts.
Step 7: Set release gates
Decide before you test what “ready” means. For example: all unsafe and harmful cases pass, happy paths pass completely, failure handling is reviewed by a person, and the adversarial set has been run on the final version. Keep the results with a date and the version tested.
Frequently asked questions
How many test cases does a chatbot need?
Enough to cover each capability across the main test types, then more wherever failures cluster. Start with the matrix and grow the set from real transcripts.
Can I automate chatbot testing?
Yes, for scripted checks, regression runs and intent matching. Generated replies are harder to automate and usually need people to review samples.
How often should I rerun tests?
After every change to content, prompts, models or integrations, and on a schedule if the underlying model can change without notice.
Who should test?
Both the builders and people who did not write the content. Fresh users find confusing wording that the team no longer sees. See how to choose a chatbot book for resources to learn more.
Keep the suite alive
A test suite is useful only if it grows. Every real failure becomes a new case, so the same mistake cannot return unnoticed. Treat the suite like the bot’s content: owned, dated and reviewed.