Testing Smart Answers helps you confirm that visitors get useful AI-generated answers for the questions they’re likely to ask. Instead of testing only keyword relevance, test the full answer experience: the question, generated answer, citations, source content, and follow-up tuning changes.
Use this workflow before launch and after meaningful Smart Answers or relevance changes.
Step 1: Identify Personas and Journeys
Start by identifying the visitors who are likely to use your search experience. For each persona, list the tasks or journeys they might complete on your site.
Ask questions such as:
- Who’s coming to the site?
- What are they trying to do?
- What information do they need before they can take the next step?
- What questions would they ask in their own words?
This helps you test questions that matter to visitors, not only questions that internal teams or domain experts might ask.
Step 2: Build a Question List
Create a spreadsheet for your Smart Answers test set. Add at least 10 to 20 questions that match the personas and journeys you identified.
Include different question types, such as:
- Process questions, such as How do I apply?
- Time-sensitive questions, such as When will the next conference take place?
- Fact questions, such as Who is the director?
For each question, leave columns for the generated answer, citation links, evaluation notes, tuning changes, and retest results.
Step 3: Run the Questions
Test each question on your search page or on the Hosted Search Page in the SearchStax Site Search dashboard.
- Open the search experience you want to test.
- Ask each question from your test sheet.
- Record the generated answer in the sheet.
- Record the citation links that appear with the answer.
- Note any questions that don’t generate a Smart Answer.
If you’re testing a recent tuning change, remember that Smart Answers responses are cached. For cache-specific testing guidance, see Tuning Smart Answers.
Step 4: Evaluate Each Answer
Review the answers one by one. For each answer, ask:
- Does the answer address the question?
- Are the citations correct?
- Is the answer factually correct?
- Does the answer come from the cited content, or does it appear to rely on general world knowledge?
- Does the answer reflect the domain expertise expected for your site?
- Does any part of the answer appear unsupported, inaccurate, or hallucinated?
Use the same evaluation criteria for each question so you can compare results before and after tuning.
Step 5: Tune and Retest
When an answer needs improvement, tune the search and Smart Answers settings that affect the source content and answer behavior.
| Control | How It Can Influence Smart Answers |
| Synonyms | Synonyms help search find relevant content when visitors use different words than your website content uses. This can improve the source material available for Smart Answers. |
| Promotions | Promotions can increase the visibility of important content in the search results used to generate answers. |
| Search Fields | Search Fields determine which indexed content fields can be searched and included in the result set that Smart Answers uses. |
| Data Filters and Smart Answers Content Exclusions | Data Filters and Content Exclusions control which content is eligible to contribute to answers for a search experience, audience, or topic. |
| Smart Answers Reasoning Controls | Reasoning Controls define which types of questions Smart Answers is allowed to answer. |
After each meaningful change, retest the full question set. Don’t test only the question you tried to fix. A change that improves one answer can lower answer quality for another important question.
Step 6: Launch, Review Analytics, and Iterate
When your highest-priority questions produce useful answers with accurate citations, you can prepare to launch Smart Answers for the search experience.
After launch, review analytics each month to identify what visitors are actually asking. Add common or high-value questions to your test sheet, tune the related settings, and repeat the testing workflow.
Continue iterating until Smart Answers covers the top questions for your search experience.
What to Do Next
For configuration details, see Smart Answers. For tuning guidance, see Tuning Smart Answers. For custom search pages, see Add the Smart Answers UX Widget to a Custom Search UI.