Crawler IP Allowlisting

Updated

If your firewall, proxy, web application firewall (WAF), or hosting provider blocks bot traffic, allowlist the SearchStax Site Search Web Crawler before you run a crawl.

The SearchStax crawler IP address is: 

3.128.83.110

For general crawler setup guidance, see Crawler. For an end-to-end setup example, see Crawler Walkthrough.

If your site requires custom request headers or authenticated crawler requests, configure HTTP Headers in the crawler settings. HTTP Headers are separate from IP allowlisting and don't replace firewall, WAF, or hosting-provider rules that require the crawler IP address.

When to Allowlist the Crawler IP

Allowlist the crawler IP when your site accepts requests only from approved sources or when your hosting provider blocks crawler traffic by default.

This is common when your site uses:

  • A firewall or WAF
  • Bot protection
  • Hosting-provider access rules
  • Security rules that block unknown automated traffic

If these controls block the crawler, crawls may fail, time out, or return fewer crawled URLs than expected.

Allowlist the SearchStax Crawler

Ask your network, security, or hosting team to allow inbound requests from this IP address: 

3.128.83.110

If your firewall supports comments or rule labels, use a label such as: 

SearchStax Crawler

After the rule is active, confirm that the crawler can reach the site’s Start URL.

Check the Crawler Schedule and History

In Site Search, go to Site Search > App Settings > Data Management > Crawler.

Open the crawler and check its schedule and crawl history. The schedule shows when the crawl starts. A crawl can continue after the scheduled start time while the crawler discovers and processes pages.

Use the crawler’s History tab to see how long previous crawls have taken. If your firewall rules use time-based access windows, make sure the window is long enough for the crawl to finish.

Verify Crawler Access

After you add the allowlist rule, confirm that the crawler can access your site.

  1. In Site Search, go to Site Search > App Settings > Data Management > Crawler.
  2. Open the crawler.
  3. If a crawl is available, run the crawler. Otherwise, wait for the next scheduled crawl.
  4. Open the crawler’s History tab.
  5. Confirm that the crawl completes and reaches the expected pages.

If you review server or hosting-provider logs, compare the request source IP with: 

3.128.83.110

If the request source IP doesn’t match 3.128.83.110, that request may not be from the SearchStax crawler.

If the Crawl Still Fails

If the crawler is still blocked or the crawl doesn’t complete after allowlisting 3.128.83.110, check the following:

  • The allowlist rule is active and published.
  • The rule applies to the correct domain, host, or environment.
  • The rule allows the crawler to reach the crawler’s Start URL.
  • Bot-protection or WAF rules also allow the crawler.
  • Any time-based access window is long enough for the crawl to finish.
  • The site is available to the crawler without a login, or the crawler is configured with the HTTP Headers required for access.

If the issue continues, contact Support. Include the crawler name, Start URL, crawl time, time zone, and any relevant firewall or hosting-provider log entries.

Articles in this section