AI Search Visibility Checks
New checks in Website Toolkit show whether AI search tools can reach, read and quote your clients' sites.
I've added checks to Website Toolkit that show whether AI search tools can reach, read and quote your clients' sites. Every audit now includes a new AI Visibility section with eleven checks.
More and more people skip the search results page altogether. They ask ChatGPT, Perplexity or Claude, or read the AI Overview at the top of Google, and click whichever sources it cites. If those tools can't get at a site, it simply isn't in the answer.
The awkward part is that nothing on the site tells you. Pages load fine in a browser and rank fine in Google. Meanwhile a robots.txt line, a firewall toggle or a JavaScript-heavy theme is quietly keeping AI crawlers out. Those are the problems the new checks look for.
Why sites go missing from AI answers
Most of the time it's an accident: a setting someone switched on years ago, or a theme that builds the page in the browser. These are the four I see most often.
- robots.txt blocks the AI crawlers. A lot of sites added
Disallowrules for GPTBot and friends in 2023, aimed at stopping AI training. Some of those rules also block the crawlers that power AI search, such as OAI-SearchBot and PerplexityBot. - A firewall blocks them anyway. Cloudflare and other CDNs offer a one-click "Block AI bots" setting. It overrides robots.txt and leaves no trace on the site itself.
- The content needs JavaScript. GPTBot, ClaudeBot and PerplexityBot don't run JavaScript. If a page is built in the browser (a typical React or Vue app), they see an empty shell.
- Opt-out tags.
nosnippetandmax-snippet:0stop Google quoting a page in AI Overviews and AI Mode, not just in normal snippets. They often get added for an unrelated reason and stay forever.
The eleven new checks
Five checks are warnings: each one is a problem that stops AI tools reaching, reading or quoting a page. The other six are suggestions. They're worth doing, but they're optional or often deliberate.
| Check | Type | What it looks for |
|---|---|---|
| AI search crawlers blocked | Warning | robots.txt blocks crawlers that fetch pages for AI answers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User |
| AI crawlers blocked by firewall | Warning | The homepage loads for a normal browser but is refused when requested as GPTBot, ClaudeBot or PerplexityBot |
| JavaScript-only content | Warning | Under 50 words in the page's HTML, plus an empty app container or an "enable JavaScript" notice |
| Snippets blocked | Warning | nosnippet or max-snippet:0 in the robots meta tag or the X-Robots-Tag header |
| Invalid structured data | Warning | JSON-LD that fails to parse, so search and AI engines ignore it completely |
| AI training crawlers blocked | Suggestion | robots.txt blocks GPTBot, ClaudeBot, Google-Extended, CCBot and other training crawlers. Often deliberate, and that's fine |
| Structured data gaps | Suggestion | A homepage with no Organization or LocalBusiness markup, or an article with no author or date |
| llms.txt | Suggestion | No /llms.txt file, or one that's malformed or links to broken pages |
| Freshness signals | Suggestion | Sitemap entries with no lastmod date, or all stamped with the same date |
| AI opt-out tags | Suggestion | The non-standard noai and noimageai directives |
| Open Graph tags | Suggestion | Missing og:title, og:description or og:image, which chat apps and AI assistants use for link previews |
I deliberately kept two distinctions. Blocking a training crawler keeps content out of future AI models, which plenty of businesses choose to do. Blocking a search crawler keeps the site out of AI answers today. Only the second counts as a warning.
I also left out anything subjective. There are no "answer-readiness" or readability scores, because they'd add noise without giving you anything concrete to fix.
Where you'll see them
The new findings appear on the Issues tab alongside everything else, grouped by check, and go into the email report. Each finding explains what's wrong and what to change.
- Health score: only the five warnings count. A missing llms.txt or a deliberate training-crawler block won't drag a client's score down.
- Monitoring: scheduled monitors alert you when a warning appears, for example when a plugin update adds
nosnippetor someone switches on a firewall's AI block. Suggestions never trigger an alert. The first run after the update won't report every existing finding as new. - Cost: most checks reuse data the crawl already collects. Each audit makes about five extra requests: one for llms.txt, plus a few homepage fetches to test the firewall.
Quick fixes
Most of these take minutes to fix once you know they're there.
- Unblock the AI search crawlers in robots.txt. You can keep training crawlers out and still let search crawlers in:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
- Check the firewall. In Cloudflare, look under Security → Bots for the AI crawler setting. Other CDNs and security plugins have similar rules.
- Render content on the server. For JavaScript sites, server-side rendering or pre-rendering puts the content into the HTML that crawlers actually receive.
- Remove stray
nosnippettags unless a page really shouldn't be quoted. - Add basic structured data. Put Organization (or LocalBusiness) markup on the homepage, and give Article markup an author, datePublished and dateModified. Most SEO plugins do this once it's configured.
- Consider an llms.txt. It's a short Markdown file at the site root that lists the key pages (see llmstxt.org). The convention is still new, but it costs ten minutes.
Try it
The AI Visibility checks are switched on for every audit, with nothing to configure. Run an audit on one of your own sites and open the Issues tab. If a crawler is blocked, you'll find out today rather than when a client asks why a competitor keeps turning up in ChatGPT and they don't.