Clear a Site Search Index

Updated

Learn how to clear a Site Search index when you need to remove outdated or unwanted content.

Prerequisites

Before you start, you’ll need:

  • Access to the Dashboard for your Search App
  • The Update Endpoint URL and a Read & Write token (or admin credentials) for your Search App
  • The ability to re-ingest content after clearing the index

Understand How Crawler Deletions Work

The crawler automatically deletes pages that are no longer reachable. To prevent accidental data loss, it stops deletions if they exceed a built-in 20% threshold. Large, legitimate removals may require a manual clear or a one-time threshold change - in those cases, you can contact Support.

Option A: Clear the Index with the Update API

  1. In the Dashboard, go to App settings → All APIs → Search & Indexing. Copy the Update Endpoint URL and Read & Write token.
    Note: Delete operations require a Read & Write token. If you receive a 401 Unauthorized error, verify you're not using a Read Only token.
  2. From your terminal, send a delete-by-query request:
    Basic authentication:
curl -X POST "<update_endpoint>" \
  -H "Content-Type: application/json" \
  -d '{ "delete": { "query": "*:*" } }' \
  --basic --user <admin_username>:<admin_password>

Token authentication:

curl -H "Authorization: Token <read_write_token>" \
  -X POST "<update_endpoint>" \
  -H "Content-Type: application/json" \
  -d '{ "delete": { "query": "*:*" } }'
  1. Wait for deletes to commit, then repopulate the index using your preferred ingestion method.
  2. Open Preview and run a broad search (for example, *). Confirm that no results appear. It can take five to ten minutes for deletes to commit.

Warning: This removes all documents in the index. You can’t undo this action.

Delete Individual Documents

  1. In the Dashboard, open Preview and locate the document to remove. Copy its id value. In a crawler index, the id is usually the full page URL.
  2. Send a delete-by-query request to your Update Endpoint URL using the document id.

    Basic authentication:
curl -X POST "<update_endpoint>" \
  -H "Content-Type: application/json" \
  -d '{ "delete": { "query": "<document_id>" } }' \
  --basic --user <admin_username>:<admin_password>

Token authentication:

curl -H "Authorization: Token <read_write_token>" \
  -X POST "<update_endpoint>" \
  -H "Content-Type: application/json" \"
  -d '{ "delete": { "query": "id:"<document_id>" } }'
  1. In Preview, confirm the document no longer appears. It may take five to ten minutes for changes to reflect.

Tip: The JSON requires careful quoting and escaped double quotes around the id value. If quoting is incorrect, the request can fail with a 403 error.

Option B: Remove Crawled Content by Deleting the Crawler

  1. In the Dashboard, go to App Settings > Data Management > Crawler.
  2. Select the crawler you used for ingestion and delete it.
  3. After deletion, confirm in Preview that pages contributed by that crawler no longer appear. It can take up to ten minutes for changes to reflect.

Note: Deleting a crawler immediately removes the documents that crawler contributed. If you need to continue crawling with updated rules, create a new crawler and run it after you confirm the removal.

Option C: Ask Support to Allow Large Crawler Deletions

If you need to remove more content than the crawler’s safeguard allows, contact Support:

  1. Request a temporary adjustment to the crawler deletion threshold for your Search App. Include your App name, environment, and the crawler to change.
  2. Rerun the crawler after Support confirms the change.

When the crawl finishes, verify in Preview that the unwanted pages are removed.

Articles in this section