We read what crawlers read
One request for robots.txt and one for llms.txt, parsed by the same RFC 9309 rules the crawlers follow: group matching, longest match, Allow wins ties.
// AI search visibility, checked in seconds
See which AI crawlers can read your site
Type a domain. We read its robots.txt the way GPTBot, ClaudeBot, PerplexityBot and 17 other AI crawlers do, and show you the line that decides each one.
Free, no sign-up. We fetch two public files once each. No domain handy? .
RFC 9309 parser20 AI crawlersNo cookiesEN / ES
yoursite.com
| Lamp | Crawler | Operator and product | Used for | robots.txt says |
|---|
The other 68 points: full report for yoursite.com
We audit up to 10 pages for llms.txt, the sitemap, noindex and snippet limits, text that loads without JavaScript, structured data, titles, canonicals and language tags. You get a 0-100 score with every weight shown and fixes ordered by points at stake, on a page you can share or save as PDF.
// how it works
From domain to fix list in three steps
You see every verdict
Each of the 20 AI crawlers gets a lamp: lit if it can read your homepage, dark if a rule blocks it, with the exact line that decides it.
You fix what costs points
The full report audits up to 10 pages and orders every fix by the points at stake, so the first change you make is the one that matters most.
// sample report
What the full report tells you
This is the top of a real report for example.com, the domain reserved for demos. It scored 52 of 100. Each fix names what is missing, what to change and how many points it is worth.
No structured data (JSON-LD or microdata) found
Add JSON-LD: Organization + WebSite on the homepage, Article/Product/FAQPage/BreadcrumbList where they apply.
Very little text in the server-rendered HTML (under 600 characters)
Add descriptive, crawlable text to the page (and make sure it is not injected by JavaScript).
No XML sitemap found at https://example.com/sitemap.xml
Publish /sitemap.xml and reference it with a "Sitemap:" line in robots.txt.
// scoring
How the score adds up
Every point comes from something you can check yourself. The free check covers the first three rows, 32 points. Training crawlers are left out on purpose: letting models train on your pages is a policy choice, and the score should not push you either way.
A score is a checklist. It does not predict rankings or promise that any AI product will cite you.
| What we check | Points |
|---|---|
| AI search crawlers allowed in robots.txt | 20 |
| Assistants allowed to fetch a page for a user | 8 |
| robots.txt reachable and well formed | 4 |
| Text present in the HTML the server sends | 14 |
| No noindex, nosnippet or tiny max-snippet | 12 |
| Structured data | 12 |
| Title, description, canonical, language, Open Graph, H1 | 10 |
| llms.txt present and in llmstxt.org format | 8 |
| Sitemap found and declared in robots.txt | 6 |
| HTTP status, HTTPS and response time | 6 |
// field data, october 2026
What we found on 751 sites in Mexico, the US and Canada
We ran this check on the most-linked domains in each country. Many sites that opt out of AI training also shut out AI search, usually with one broad rule. Here is how to avoid it.
Samples: top 300 .mx and top 300 .ca domains, and for the US a proxy of the top 200 .com, 50 .gov and 50 .edu, from the Majestic Million (CC BY 3.0) of 2026-10-20. 751 of 900 gave a usable answer. Only robots.txt and llms.txt were read, so blocks made by a CDN or firewall are not counted.
// pricing
Start free, pay once if you want the full picture
Free check
$0
- All 20 AI crawlers against your robots.txt
- The rule that decides each one
- llms.txt and sitemap presence
- 32 of the 100 score points
Full report
$19 USD, taxes at checkout
- Up to 10 pages audited
- 0-100 score with every weight
- Fixes ordered by points at stake
- English or Spanish, shareable page and PDF
- 14-day refund, no questions
Many sites or an API
$0.03 per site and up, on Apify
- The same audit as an Apify Actor
- Batch runs and schedules
- JSON output, MCP for AI agents
- Pay per site and per page
Card details go to Polar, our merchant of record, who also handles sales tax. We never see them.
// questions
Questions
Does blocking AI training crawlers remove my site from AI search?
No. OpenAI, Anthropic, Google and Apple use separate crawlers for training and for search. You can block GPTBot, ClaudeBot or Google-Extended and still allow OAI-SearchBot, Claude-SearchBot and Googlebot. Here is a robots.txt you can copy that blocks training and keeps search.
Does llms.txt help me show up in AI search?
Nobody has shown that it does. llms.txt is a community proposal, and no AI vendor has confirmed using it for ranking or citations. It is worth 8 of 100 points in our score and we list it as a low-severity fix.
Why can a crawler that fetches pages for a user show as blocked?
Some vendors say their user-initiated fetchers (ChatGPT-User, Perplexity-User and others) may ignore robots.txt. We report what your file says and add the vendor's note next to that crawler.
What can this check not see?
Blocks made by a CDN, firewall or bot-protection service, which happen before robots.txt is read. If AI crawlers are allowed in robots.txt but still cannot reach you, look at those settings.
Does the report run JavaScript?
No, on purpose. Most AI crawlers read the HTML your server sends, so that is what we measure. A page that needs JavaScript to show its text loses points for it.
How polite is the crawler?
It identifies itself as AtalayaBot, reads robots.txt first, skips anything it disallows, and requests one page at a time from each site.
Who runs Atalaya AI, and is AI involved?
Atalaya AI is a small software brand operated from Mexico. We use AI to help write code and drafts, and a person owns the project and approves what goes public. Vigía, the guide in the corner, is a scripted helper: it does not send anything you type anywhere. We are not affiliated with OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, DuckDuckGo or Common Crawl.
Your robots.txt is public. Find out what it says to AI.
It takes one domain and about two seconds.