Crawler Exclusions

Updated

The SearchStax Site Search solution's Crawler add-on explores the pages of a website starting at a start URL. It then follows the embedded links in the pages rather than following the hierarchical structure of the website.

Crawler Constraints

The Crawler follows these scope and exclusion rules:

  • Domain: The Crawler stays within the DNS domain specified in the start URL. For example, if the start URL is https://my.company.com/bios/, the Crawler stays within my.company.com.
  • Crawl depth: The Crawler follows links only to the configured crawl depth.
  • Exclusions: The Crawler skips URLs that match configured exclusion rules. Matching is case-sensitive, so /internal/ doesn't match /Internal/. Exclusion rules also apply to URLs listed in a sitemap without stopping sitemap discovery.

Note: Exclusions apply only to crawls that run after you save the rule. They don't remove documents indexed by earlier crawl events.

Find Documents from Earlier Crawls

If an excluded URL remains searchable, query the index for the URL and its crawl ID. If your Site Search App uses security tokens: 

curl -H "Authorization: Token <read-only token>" "https://searchcloud-1-us-west-2.searchstax.com/12345/crawler-1234/select?q=url:*&wt=json&indent=true&fl=url,exif_crawlid&rows=10&start=1"

If your Site Search App uses Basic Auth credentials: 

curl -u <read-only user>:<read-only password> "https://searchcloud-1-us-west-2.searchstax.com/12345/crawler-1234/select?q=url:*&wt=json&indent=true&fl=url,exif_crawlid&rows=10&start=1"

Run the /select query in a Linux terminal or a PowerShell terminal on Windows. The response includes each URL and its exif_crawlid value: 

{
  "response":{
    "numFound":368,
    "start":1,
    "numFoundExact":true,
    "docs":[
      {
        "url":"https://www.example.com/internal/example-page",
        "exif_crawlid":"12345"
      }
    ]
  }
}

Adjust the &rows and &start parameters to view different portions of the list.

Remove Documents from Earlier Crawls

The crawler's History tab doesn't display exif_crawlid, so you can't use this value to match a search result to a History entry. To remove content indexed by earlier crawls, delete the relevant earlier crawl events from History.

  1. Open the crawler and go to History.
  2. Select the earlier crawl events that contain the indexed content you want to remove.
  3. Use the delete action and confirm the deletion. The event moves to Deleting and then to Deleted when cleanup finishes.

For more information, see the History Tab section in the Crawler article.

Warning: Deleting a crawl event removes every indexed item associated with that event, not just the excluded URL.

Articles in this section