Crawler IP Allowlisting

Updated

If your firewall, proxy, web application firewall (WAF), or hosting provider blocks bot traffic, allowlist the SearchStax Site Search Web Crawler before you run a crawl.

The SearchStax crawler IP addresses are: 

3.128.83.110
18.211.235.218

SearchStax uses 18.211.235.218 for preliminary checks. Allowlist both addresses so these checks and crawler requests can reach your site.

For general crawler setup guidance, see Crawler. For an end-to-end setup example, see Crawler Walkthrough.

If your site requires custom request headers or authenticated crawler requests, configure HTTP Headers in the crawler settings. HTTP Headers are separate from IP allowlisting and don't replace firewall, WAF, or hosting-provider rules that require the crawler IP addresses.

When to Allowlist the Crawler IP Addresses

Allowlist the crawler IP addresses when your site accepts requests only from approved sources or when your hosting provider blocks crawler traffic by default.

This is common when your site uses:

  • A firewall or WAF
  • Bot protection
  • Hosting-provider access rules
  • Security rules that block unknown automated traffic

If these controls block the crawler, crawls may fail, time out, or return fewer crawled URLs than expected.

Allowlist the SearchStax Crawler

Ask your network, security, or hosting team to allow inbound requests from both IP addresses: 

3.128.83.110
18.211.235.218

By default, SearchStax crawler requests use this User-Agent: 

SearchStax Crawler/1.0 (+https://www.searchstax.com)

Allow both source IP addresses in every security control that filters requests by IP address. If a security control also filters requests by User-Agent, allow the exact User-Agent shown above. Some environments require both an IP-address rule and a User-Agent rule before a request is accepted.

If your firewall supports comments or rule labels, use a label such as: 

SearchStax Crawler

After the allowlist is active, confirm that the crawler can reach the site’s Start URL.

Check the Crawler Schedule and History

In Site Search, go to Site Search > App Settings > Data Management > Crawler.

Open the crawler and check its schedule and crawl history. The schedule shows when the crawl starts. A crawl can continue after the scheduled start time while the crawler discovers and processes pages.

Use the crawler’s History tab to see how long previous crawls have taken. If your firewall rules use time-based access windows, make sure the window is long enough for the crawl to finish.

Verify Crawler Access

After you add the allowlist rules, confirm that the crawler can access your site.

  1. In Site Search, go to Site Search > App Settings > Data Management > Crawler.
  2. Open the crawler.
  3. If a crawl is available, run the crawler. Otherwise, wait for the next scheduled crawl.
  4. Open the crawler’s History tab.
  5. Confirm that the crawl completes and reaches the expected pages.

If you review server or hosting-provider logs, compare the request source IP with these addresses: 

3.128.83.110
18.211.235.218

If the request source IP doesn’t match either address, that request may not be from the SearchStax crawler.

If the Crawl Still Fails

If the crawler is still blocked or the crawl doesn’t complete after allowlisting both IP addresses, check the following:

  • The allowlist rules are active and published.
  • The rule applies to the correct domain, host, or environment.
  • The rule allows the crawler to reach the crawler’s Start URL.
  • Bot-protection or WAF rules also allow the crawler.
  • Any time-based access window is long enough for the crawl to finish.
  • The site is available to the crawler without a login, or the crawler is configured with the HTTP Headers required for access.

If the issue continues, contact Support. Include the crawler name, Start URL, crawl time, time zone, and any relevant firewall or hosting-provider log entries.

Articles in this section