Crawler Indexed More Pages Than It Crawled

The Crawler History Tab sometimes says that the Crawler indexed more pages than it crawled. What does this mean?

SearchStax Site Search Crawler history table shows 441 items indexed versus 418 URLs crawled for a one-time crawl event.

Why Crawler Sometimes Indexes More Pages Than It Crawled

This result sometimes appears when the workload was distributed across multiple parser processes. One of the processes failed before reporting which files it had indexed successfully.

Another process then picked up the task and reindexed a few of the files. This behavior is harmless.

Since the index records are identified by webpage URL, deduplication is automatic. No action is required.

Articles in this section