Troubleshoot Crawler Failures, Repeated Hits, and Host Slowdown

Updated

If a SearchStax Site Search crawl fails, appears to keep hitting your site, or causes host slowdown, use the checks below to confirm what happened and narrow the crawler scope.

For crawler setup details, see Crawler. For an end-to-end setup example, see Crawler Walkthrough.

What You May See

  • The crawler status shows a failed or incomplete crawl.
  • Your CDN, WAF, firewall, bot-protection service, or hosting provider blocks crawler requests.
  • Hosting logs show many requests during the crawl window.
  • The crawler appears to keep requesting the same host, section, or URL pattern.
  • Your site, hosting environment, or backend editing experience slows down while a crawl is running.

Check the Crawler Status and History

In Site Search, go to Site Search > App Settings > Data Management > Crawler. Open the crawler and review the History tab.

Check:

  • The crawl status.
  • The crawl start time and end time.
  • The crawl type, such as Full or Incremental.
  • The number of URLs crawled.
  • The number of items indexed.
  • Whether the crawl was still running when the host slowdown or blocked traffic occurred.

Scheduled crawler times are shown in the local time used by the Site Search interface. Crawl duration varies by site size, page size, site response time, crawl depth, and crawler scope. Use the History tab to confirm how long your crawler usually takes.

Confirm Whether the Traffic Is From the SearchStax Crawler

Before changing blocking rules, confirm that the requests in your hosting or security logs match SearchStax crawler traffic.

The SearchStax crawler IP address is 3.128.83.110. If your security provider needs additional crawler identity details, contact SearchStax Support.

If a request uses a different IP address, it may not be from the SearchStax crawler. Save the timestamp, request IP, requested URL, HTTP status code, and any User-Agent shown in your logs before contacting Support.

If the Crawl Fails or Is Blocked

A crawl can fail when the crawler can’t reach the Start URL, when the site requires authentication, or when a security tool blocks crawler traffic.

Check the following:

  • The Start URL is publicly reachable.
  • The Start URL returns the expected page, not a login page, error page, redirect loop, or bot-check page.
  • Your CDN, WAF, firewall, bot-protection service, or hosting provider allows requests from 3.128.83.110. For allowlisting guidance, see Crawler IP Allowlisting.
  • Any “under attack,” bot-protection, or challenge mode in your hosting/security platform isn’t blocking the crawler.
  • Your sitemap, if used as the Start URL, is publicly reachable and lists the URLs you expect the crawler to discover.
  • Your crawler settings were saved before the crawl ran.

If the crawler is blocked, ask your network, security, or hosting team to allow inbound requests from 3.128.83.110. If your security provider requires additional crawler details, contact SearchStax Support before making assumptions about crawler identity.

If the Crawler Appears to Keep Hitting Your Site

If hosting logs show crawler requests after a crawl appears to have failed or finished, collect the log details and contact SearchStax Support. A failed crawl isn’t expected to continue crawling indefinitely.

Include log samples that show:

  • Request timestamp, including time zone.
  • Request IP address.
  • User-Agent.
  • Requested URL.
  • HTTP status code.
  • Any CDN, WAF, firewall, bot-protection, or hosting rule that allowed or blocked the request.

If the requests are coming from the SearchStax crawler during an active crawl, review crawler scope. Repeated requests can happen when the crawler discovers many URL variants or enters a large section of the site.

Reduce Unnecessary Crawl Scope

Review your crawler settings if the crawler is reaching more pages than expected or spending too much time in sections that shouldn’t be indexed.

Check for:

  • Multiple URL versions of the same content, such as trailing-slash and non-trailing-slash URLs.
  • Filter, sort, calendar, search result, print, or tracking-parameter URLs.
  • Redirects that send the crawler through repeated paths.
  • Sitemap entries and page links that point to different versions of the same page.
  • Inclusion rules that are too broad.
  • Missing exclusion rules for sections you don’t want crawled.
  • A crawl depth that reaches more linked pages than you need.

Where possible, fix duplicate URL variants at the source by using one preferred URL pattern and redirecting variants such as www/non-www, trailing-slash/non-trailing-slash, or /index URLs to that preferred URL.

To narrow crawler scope, use inclusion rules for sections you want to include and exclusion rules for URL patterns you don’t want crawled. For details, see Crawler Exclusions.

If crawler history shows more items indexed than URLs crawled, that may be a separate reporting behavior rather than a repeated-hit problem. See Crawler Indexed More Pages Than It Crawled.

If Your Site or Host Slows Down During a Crawl

Crawl impact depends on your site, hosting environment, page count, page size, crawl depth, and crawler scope. SearchStax doesn’t provide a customer-facing setting for crawler throttling or per-domain crawl rate in the published crawler settings.

To reduce unnecessary crawler activity:

  • Use a sitemap Start URL when your sitemap reliably lists the pages you want indexed.
  • Lower crawl depth if the crawler is reaching more linked pages than needed.
  • Add inclusion rules when the crawler should stay within specific URL patterns.
  • Add exclusion rules for generated or low-value URL patterns, such as search pages, calendars, filtered listings, print pages, and tracking URLs.
  • Schedule crawls during a lower-traffic window when possible.
  • Use Incremental Crawl for scheduled crawls when it fits your site. Incremental crawls check for changes and update affected content during scheduled runs. A full crawl still runs periodically.

When to Contact Support

Contact SearchStax Support if the crawl still fails after these checks, if your security provider continues to block crawler traffic, if hosting logs show crawler requests after a failed or finished crawl, or if your site slows down during crawls after you narrow crawler scope.

Include:

  • Search App name.
  • Crawler name.
  • Start URL.
  • Crawl schedule.
  • Approximate time the issue happened, including time zone.
  • Crawler History screenshot showing status, start time, end time, crawl type, URLs crawled, and items indexed.
  • Recent crawler setting changes.
  • Recent CDN, WAF, firewall, bot-protection, or hosting changes.
  • Timestamped hosting logs that show blocked or repeated requests.
  • The IP address and User-Agent shown in your hosting or security logs.
Articles in this section