GetIndexedBot

GetIndexedBot is the crawler Get Indexed uses to fetch pages. It fetches sites submitted for an AEO audit, pages our customers ask the content optimizer to analyze, the public pages of business websites when we look up their contact details, and Yellow Pages listings when we check a business's directory entries. It renders a page's public content and never submits forms or runs transactions.

User agent

GetIndexedBot/1.0 (+https://getindexed.ai/bot)

Behavior

Crawls at low concurrency and fetches only up to a fixed page limit per audit.

robots.txt

GetIndexedBot fetches robots.txt to score it in the audit, but does not follow its rules yet. Support is in progress. In the meantime, email us and we'll add your domain to GetIndexedBot's exclusion list. It covers the domain and all of its subdomains; once applied, no GetIndexed audit or fetch will request any page on it.

Seeing unexpected traffic, or want it to stop? Email support@getindexed.ai.