What this check reads
Your robots.txt at https://your-domain/robots.txt, evaluated for the path / with the same parser CitedOS uses in its Foundation check. A token with no rule of its own falls back to the * group, and we say so.
Free tool
Which AI crawlers can read your site?
Enter a domain and see, crawler by crawler, whether its robots.txt allows or blocks it, with the line that says so. We read one public file and report what it says.
Read this first
Two kinds of AI crawler, two different choices.
Search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) fetch pages so an assistant can find and cite them, so blocking one takes your pages out of that engine's reach. Training crawlers (GPTBot, ClaudeBot, CCBot and similar) collect content to build models; blocking one is a content-rights choice, and for GPTBot, ClaudeBot and Applebot-Extended the vendors state it does not remove you from their search results. Google-Extended is the exception: Google says it also governs Gemini grounding, so we list it on its own. Many sites block the training kind on purpose and allow the search kind.
What it cannot see
robots.txt is a request, not enforcement. Firewall and CDN rules that let verified bots in or keep them out cannot be tested from outside, and some live-fetch agents may not follow robots.txt at all.
What it does not tell you
Whether any engine cites you. Access is a precondition, not a prediction: AI citations swing widely from month to month. llms.txt is optional, and no major engine has said it uses it, so we do not check it.