This end-to-end walkthrough shows you how to index the SearchStax product documentation website using Site Search and its Crawler. This exercise takes about half an hour to complete.
Crawler Limitations
The Crawler feature has the following limitations:
- You can run only one crawl per day.
- Crawls are limited to 10,000 pages or 100,000 pages per crawl, depending on your contract.
Getting search results for your website is surprisingly easy. However, there are moments when you wonder what to do next. This discussion covers those moments.
We assume you've got a Site Search account from your SearchStax Onboarding Manager. Log in to the Site Search interface.
The exercise starts with setting up and running the Crawler. When the crawl is complete, we'll walk you through the Site Search features that create your site's first search experience.
Create the Site Search App
When you Create the Site Search App, configure it as a "Custom" application.
Configure and Run the Crawler
If your account has the crawler feature enabled, you'll find it in the navigation menu under Site Search > App Settings > Data Management > Crawler:
This link opens the Crawler list, which is initially empty.
A Search App can have multiple crawlers, depending on your contract terms. Each crawler can index a different website. Click Create a Crawler.
The next step is to provide your crawler with a name and a starting URL. After you enter the Start URL, select Validate URL to confirm that Site Search can reach it.
The crawler starts with the root URL and follows links within the same corporate domain. For this walkthrough, the optimum crawl depth is 2, though 0 also works.
If the site requires custom request headers, expand HTTP Headers and add the required header names and values before validating the Start URL. See HTTP Headers for more information about custom headers.
You can click Crawl Now after the Start URL validates successfully. However, we recommend you visit the list of crawler fields first. The crawler is limited to one run per day. We need to set up a special field before launching it.
Configure the Crawler Fields
Note: This is an optional step to demonstrate setting up a facet. The crawler imports a set of default fields from webpages (see Default Field Map for details). You'll find that your target website uses additional fields. Site Search lets you add these fields to the crawl.
The SearchStax website documentation pages contain a Products meta tag. This makes a simple facet demonstration:
<meta name="Products" content="Managed Search">We'd like the crawler to import the value of this tag into the index.
Open the Manage Fields for Search Index section of the crawler settings. You'll see the list of default fields.
These fields can be useful in your project and are harmless if unused. Click the Add Field button. The resulting dialog box appears on the page.
- Set the Custom Field Name to products. This is the label you'll see in the Site Search lists of fields.
- Select the Meta Tag Name option and enter Products. This is the meta tag name from the target page HTML.
- For a field that will become a facet, the string datatype is usually the best choice.
- We won't add any transformations now.
Click Add Field. The new field, labeled products_ss (your field name plus the string datatype), appears in the list.
When you're satisfied with the setup, scroll to the bottom and click Save Changes.
Crawling
To launch a crawl, return to the Crawler List. Select the checkbox beside the crawler name and click Crawl Now.
As the crawl proceeds, the History tab displays progress statistics.
Inspect the Document Fields (Optional)
This optional section presents some "tips and tricks" for inspecting the crawler's output. You can do this before configuring the Search Fields and Result Fields in Site Search. Although technical, the steps described here produce a convenient map of the indexed fields and their typical values.
Wait Five Minutes!
Due to search engine configuration settings, it may take up to five minutes for the crawl data to be written to the index. Until this happens, Site Search displays and query results will look the same as before the crawl.
Go to Site Search > APP Settings > All APIs > Search & Indexing. The Read-Only authentication token is near the bottom of the screen.
Copy the token to the clipboard and paste it into a text buffer temporarily.
Scroll back up and find the App's Select Endpoint.
Copy the endpoint to a text buffer and make these changes:
- Change emselect to select.
- Add ?q=*:*&wt=json&indent=true at the end following select.
Now you'll assemble a Curl command in the text buffer. Use this format:
curl -H "Authorization: Token <Read-Only Token>" "<Select Endpoint>/select?q=*:*&wt=json&indent=true"Paste this string into a Linux Bash command window or a PowerShell terminal on Windows and send it. It returns ten documents from your index. You'll see all the fields in use and their content. Notice the products_ss custom field near the bottom of this list.
{
"id":"https://support.searchstax.com/hc/en-us/articles/40048109595661",
"exif_tenant_id":"2",
"exif_crawlid":"2151",
"exif_crawl_definition_id":"43",
"exif_appid":"studio-1810",
"url":["https://support.searchstax.com/hc/en-us/articles/40048109595661"],
"paths":["docs / searchstudio / analytics-glossary"],
"document_type":["html"],
"date":"2024-06-24T02:36:12Z",
"title":["Analytics Glossary - SearchStax Site Search Docs"],
"headings1":["Analytics Glossary"],
"headings2":["Questions?"],
"description":["The SearchStax Site Search solution's Analytics Glossary is a summary of key terms and definitions used for analytics in Site Search."],
"products_ss":["Site Search"],
"content":["Analytics Glossary - SearchStax Site Search Docs Managed Search Site Search Help
<Most of the content was removed for clarity>
],
"_version_":1802708265532915712}If you've got difficulty with this, contact SearchStax Support for assistance.
This output is a convenient resource for the following steps.
Configure Search Fields
The webpage data is now in the index, but you can't search it yet. You need to choose which fields to search before you can search them.
Go to Site Search > Configurations > Basic Configurations > Search Fields. This screen tells the search index which fields to search.
Reload the Schema!
After a crawler run, and after waiting five minutes for the index to commit, click the Reload Schema button. This updates the list of potential search fields.
The left column shows the available fields in the schema. They may not all be present in the crawled documents. Click on a field to move it into the list of searchable fields.
The title and description fields are pre-selected for your convenience. Also select headings1, headings2, and headings3. Headings are rich in relevant keywords. The content field is pre-selected, but in this case, we suggest removing it from the search list.
To experiment with a facet list, also add the product_ss field to this list. Facets must be based on search fields.
Click Publish to re-issue the index. Publishing a small project like this one takes a couple of minutes.
Hosted Search Experience
In this exercise, we'll use the Site Search Hosted Search Experience to view and test your search settings. This feature is a fully functional search page that supports many features controlled by Site Search. You can configure a feature, publish it, and view the result a few seconds later.
You can do almost the same thing using the Site Search Preview screen. However, that environment is designed for engineers rather than website designers.
Configure Result Fields
The Search Fields (above) aren't ideal for display as Results Fields. You must show Site Search how to present results in the Hosted Search Experience.
Go to Site Search > Configurations > Basic Configurations > Results Fields.
The Results Fields screen lets you select fields from the index to display in the Hosted Search Experience's search results. You can map each field to a preformatted position in a results "card."
Reload the Schema!
After a crawler run and after waiting five minutes for the index to commit, click the Reload Schema button to update the list of potential display fields.
Choose a field from the Return Field list. Add a human-friendly Label if needed. Then map the field value to a Results Card Field, as explained on the Results Configuration page. The (+) icon on the right adds the configured field to the list of display fields (the lower red box). Don't overlook that step!
For this exercise, make the following mappings:
- Map the index's URL field to the result card's URL field. This is a default mapping. It makes the result items clickable and links them to the web pages they represent.
- Map the index's title field to the result card's Title field. This puts the page's title at the top of the result summary. This is a default setting.
- The index's description field maps by default to the Description field of the result card.
- Map headings1, headings2, and headings3 to "No mapping" on the results card. This lists the field values below the result item's description.
- Map the products_ss field to the result card's ribbon field. This displays the product name as a banner above the result item.
At this point, a typical result item in the Hosted Search Experience looks much like this:
Configure a Facet
The search page seems incomplete without at least one facet list on the side. How do you set that up?
Go to Site Search > Configurations > Basic Configurations > Faceting.
The Faceting page provides full instructions for using this screen. To begin, check the box that enables faceting.
The Facet Fields list lets you select an index field for a facet. If you don't see the field in the list, click the Reload Schema button again. Select products_ss. You can add a label to be the title of the facet list. In your example, the facet options will be ranked by count.
Click the (+) icon to add the facet to the Profile.
When finished, click Publish.
If you check the Hosted Search Experience at this point, you'll find a new facet on display.
Share Search Results
Experience shows that a search project often has many more stakeholders than developers. The project will need a public search portal. This lets stakeholders contribute their insights and requirements without logging into Site Search.
In the navigation menu, go to Site Search > Configurations > Search UI > Hosted Search Experience.
This screen provides a URL to a shareable search environment. You can View the page immediately or use the Copy icon on the right to share the URL with coworkers.
Click the Regenerate button to refresh the Hosted Search Experience after making changes.
Crawl Your Own Site?
You can use this guide to perform an initial crawl of your own website. Watch for these issues:
- Crawling time depends on the number of web pages and their size. The SearchStax site has about 1000 pages. Crawling time varies from site to site.
- The Crawler retrieves all fields of interest. Only some will be useful for searching. A few will make good facets. Others are for display only.
- See Field Types for advice about setting up text and string facets using the Crawler. The Crawler can index the same content into two index fields using different datatypes.