Every recommendation an AI engine makes starts the same way: a bot fetched your pages and either understood them or didn't. Before any content strategy, before any "GEO playbook", there's a plumbing question — can the machines that matter actually read you?
Meet the new crawlers
Your server logs in 2026 host a small zoo. The ones worth knowing (full landscape here):
- GPTBot — OpenAI's training crawler. Feeds future models.
- OAI-SearchBot — powers ChatGPT's live search answers and citations.
- ClaudeBot — Anthropic's crawler for Claude.
- PerplexityBot — Perplexity's index; heavily citation-driven.
- Google-Extended — a robots.txt token controlling whether your content trains Gemini. Blocking it does not remove you from Google Search or AI Overviews — those ride on regular Googlebot.
- Applebot-Extended, Bytespider, and a long tail of others.
Each one is a door you can open or close in robots.txt — and unlike
much of the AI gold rush, the major Western crawlers do respect it.
The strategic choice nobody can make for you
Blocking AI bots protects your content from training and answer-synthesis. It also removes you from the answers your buyers increasingly trust. A news publisher licensing its archive has a real case for blocking. A brand that wants to be recommended when someone asks "best accounting tool for freelancers"? Blocking OAI-SearchBot or PerplexityBot is unilateral disarmament.
Our default for commercial brands: allow the search and answer bots, make a deliberate, informed decision about the pure training bots — and write it down so it's a policy, not an accident.
A typical Seoh client configuration looks like this:
# robots.txt — open to answers, deliberate about training
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: GPTBot
Allow: / # or Disallow: / if training-use is a concern
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlllms.txt: the honest version
You've probably heard of llms.txt — a proposed markdown file at your
site root that gives language models a curated map of your most important content.
It has real momentum: an SE Ranking study of 300,000 domains found
about
10% adoption by mid-2026.
Now the cold water. Ahrefs measured actual bot behavior in May 2026 and found 97% of llms.txt files received zero requests from AI crawlers. Of the requests that did arrive, most came from SEO audit tools, not AI companies. Google's Gary Illyes has said plainly that Google doesn't support llms.txt and isn't planning to, and OpenAI's documented recommendation is robots.txt.
What actually moves the needle
While the industry argued about llms.txt, the boring fundamentals quietly decided who gets cited. These are the ones that show up in our audits again and again:
1. Server-side rendering
Most AI crawlers do not execute JavaScript. If your content only exists after a client-side framework hydrates, GPTBot and PerplexityBot see an empty shell. Googlebot renders JS; the AI fleet largely doesn't. Server-render or pre-render anything you want quoted. This is the single most common fatal flaw we find in otherwise excellent sites.
2. Clean, semantic HTML
Real headings in a logical hierarchy, real lists, real tables, main content high in the DOM. Retrieval systems chunk your pages; tidy structure means your key claims survive chunking with their context attached.
3. Structured data
Organization, Product, Article,
FAQPage schema — machine-readable statements of who you are and what
you're claiming. Cheap to add, compounding to maintain.
4. Speed and stability
AI crawlers run on budgets. Slow responses and flaky 5xxs mean shallower, rarer crawls — which means stale or missing information in the systems answering your buyers. Fast sites get read more, and more often.
5. Content parity
Whatever you show users behind tabs, accordions and "read more" buttons should exist in the HTML. If it only appears on click, assume machines never see it.
The 20-minute readability check
Want a quick self-audit? Do these five things: fetch your key page with
JavaScript disabled and see what's left; check robots.txt for which AI
bots you're blocking (on purpose or by accident); paste your page into a schema
validator; grep your server logs for GPTBot, OAI-SearchBot,
ClaudeBot and PerplexityBot to see who's already visiting;
and ask two AI engines a question your best page answers, then check whether you're
cited. Twenty minutes, and you'll know more about your AI readiness than most
agencies would tell you in a paid audit.
Or let us run the full version — crawl, rendering check, schema review, log analysis and an answer-share baseline — as part of a Boost sprint. Either way: fix the plumbing first. Everything else stands on it.
