Testing Smart Answers helps you understand what visitors ask, what answers they receive, and whether those answers help them find the right information. Instead of testing only keyword relevance, test the full answer experience, including the question, generated answer, citations, and source content.
Use this workflow to build a question list, test Smart Answers, evaluate answer quality, and repeat testing after tuning changes.
Step 1: Identify Personas and User Journeys
Start by identifying who is likely to visit your website and what they’re trying to do.
- Identify the personas you expect to visit your site.
- Identify the journeys those personas might follow.
This helps you test questions that visitors are likely to ask, not only questions that internal teams or domain experts might ask.
Step 2: Build a Question List
Create a spreadsheet for your Smart Answers test set. Add at least 10 to 20 questions that match the personas and journeys you identified.
Include different question types, such as:
- Process questions, such as How do I apply?
- Time-sensitive questions, such as When will the next conference take place?
- Fact questions, such as Who is the CEO?
Use the same spreadsheet to record the answers during testing.
This approach helps you avoid focusing only on questions that reflect your internal expertise. Test questions that visitors are likely to ask in their own words.
Step 3: Run the Questions
- Open your search page, or open the Hosted Search Page in the SearchStax Site Search dashboard.
- Ask each question from your test sheet.
- Record the answers in the same sheet.
Step 4: Evaluate Each Answer
Review the answers one by one. For each answer, ask:
- Does the answer address the question?
- Are the citations correct?
- Is the information factually correct?
- Was the information pulled from the citations, or was it pulled from world knowledge?
- Does the answer reflect domain expertise, or is it generic?
- Does the answer include information that isn’t supported by the cited content?
Use the same evaluation criteria for each question so you can compare results before and after tuning.
Step 5: Tune and Retest
When an answer needs improvement, review the SearchStax Site Search dashboard settings that can affect search relevancy and answer quality. For configuration details, see Smart Answers. For tuning guidance, see Tuning Smart Answers.
| Control | How It Can Influence Smart Answers |
| Synonyms | Synonyms can help search find relevant content when visitors use different words than your website content uses. This can improve the source material available for Smart Answers. |
| Promotions | Promotions can increase the visibility of important content in search results that Smart Answers can use to generate answers. |
| Search Fields | Search Fields can affect which indexed content fields are searchable and available in search results. If important content isn’t searchable, Smart Answers might not be able to use it. |
| Data Filters and Smart Answers Content Exclusions | Data Filters and Smart Answers Content Exclusions can control which content is eligible for a search experience, audience, or topic. Use them to include or exclude content from the set of results Smart Answers can use. |
| Smart Answers Reasoning Controls | Smart Answers Reasoning Controls can define which types of questions Smart Answers is allowed to answer. Use them to align Smart Answers with questions your content can reliably support. |
After you make a change, test all the questions again. Check whether the change improves the target answer without reducing the quality of other answers.
Step 6: Review Analytics and Iterate
After you configure Smart Answers for your top questions, continue reviewing and improving the answers.
Review analytics each month to identify what visitors are asking. Then repeat this exercise with those questions.
After a few iterations, your test set should cover more of the traffic that comes through your search page.