Crawler Content Field

The SearchStax Site Search solution's Crawler add-on has a catch-all system field called content. This field collects all the text on a web page and tokenizes it for keyword search.

Red circle highlights the xpath expression used to extract content from web pages during crawling.

The problem with the content field

Depending on the website, this field might collect more text than you want to index. For example, the SearchStax documentation pages (like this one) have a header, footer, and navigation pane (the left-side table of contents). The navigation pane captures every significant keyword in the corpus. As a result, every search matches every page.

Using a custom field instead

To sharpen search discrimination, you can delete the content field from the list of crawler fields. Then substitute a new paragraph field.

Red circle highlights an XPath selector used to extract paragraph content from web pages.

This is an xpath //p//text() text field configured to locate every

element on the page and capture its content for the index. This lets you search the page's unique payload without matching items in the Navigation Pane.

Note: The Crawler's default set of fields already captures the page's title and headings. These are likely to be the richest source of keywords that truly relate to the topic.

Articles in this section