Robots.txt: The Complete Guide for 2026 (Including AI Crawlers)

Robots.txt is a plain text file with a handful of lines in it, and it can remove an entire website from Google. The classic disaster happens after a launch: the staging site had Disallow: / to keep it out of search, someone copied the config to production, and organic traffic quietly drops to zero.

In 2026 robots.txt has a second job too. It's how you decide which AI companies can use your content, and the rules there changed: the big AI providers now run separate crawlers for model training and for AI search. Block the wrong one and you vanish from ChatGPT and Claude answers without meaning to.

This guide covers how robots.txt works, the syntax that matters, what it can't do, how to handle AI crawlers properly, the mistakes that cost traffic, and how to test your file.

What robots.txt does (and doesn't do)#

Robots.txt tells crawlers which URLs on your site they may crawl. It lives at the root of each host, for example https://example.com/robots.txt, and well-behaved crawlers read it before fetching anything else.

Three things it does not do:

• It doesn't keep pages out of Google's index. A URL blocked in robots.txt can still be indexed if other pages link to it. Google just shows it without a description. To keep a page out of search, use a noindex tag and let Google crawl the page so it can see that tag.
• It doesn't support noindex. Google stopped supporting a noindex rule inside robots.txt in 2019.
• It isn't security. The file is public, and it's a polite request. Anyone can read it, and bad bots ignore it. Never use it to "hide" private URLs. Password-protect them instead.

Where robots.txt goes#

• It must be at the root: example.com/robots.txt, not example.com/files/robots.txt.
• Each host and protocol needs its own file. shop.example.com doesn't follow the rules in example.com/robots.txt.
• Google reads up to 500 KiB of the file and ignores anything after that.
• The file must return a 200 status. If robots.txt returns a 404, Google assumes there are no restrictions. If it returns a server error (5xx), Google temporarily treats the whole site as blocked, which can stop crawling until the error is fixed.

Robots.txt syntax in 5 minutes#

A robots.txt file is made of groups. Each group starts with one or more User-agent lines, followed by the rules for those crawlers.

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

• User-agent names the crawler the group applies to. * means every crawler that doesn't have its own group.
• Disallow blocks paths that start with the given value. Disallow: /admin/ blocks everything under /admin/.
• Allow makes an exception inside a blocked path.
• Sitemap points to your XML sitemap. Use the full URL.

A few rules that trip people up:

• Paths are case-sensitive. Disallow: /Private/ does not block /private/.
• Two wildcards are supported: * matches any characters, and $ marks the end of a URL. Disallow: /*.pdf$ blocks every URL ending in .pdf.
• The most specific rule wins. When an Allow and a Disallow both match, Google uses the rule with the longest path. If they're equally long, Allow wins.
• A crawler follows only one group: the most specific one that matches its name. If you create a User-agent: Googlebot group, Googlebot ignores the * group entirely, so repeat any rules it should still follow.
• Google ignores Crawl-delay. Some other search engines respect it.

Robots.txt and AI crawlers in 2026#

This is where most robots.txt files are out of date. The major AI companies now split their crawlers by purpose: one bot collects content to train models, and others fetch pages so the AI can cite your site in answers.

The key point: these are independent. Blocking GPTBot does not block OAI-SearchBot, and blocking ClaudeBot does not block Claude-SearchBot. Blocking Google-Extended has no effect on Googlebot or your Google rankings.

That lets you make a deliberate choice. A common setup in 2026 is to opt out of training while staying eligible to be cited in AI answers:

# Opt out of AI model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Stay visible in AI search answers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Everyone else, including Googlebot
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Two caveats. First, the fetchers that act on a user's direct request (like ChatGPT-User) may not be governed by robots.txt the same way as automated crawlers. Second, robots.txt only works for bots that respect it. Some don't, and those have to be blocked at your firewall or CDN. If you use a CDN's "block AI bots" setting, check it isn't also blocking the search crawlers you meant to allow: a firewall rule wins over anything in robots.txt.

Common robots.txt setups#

WordPress. WordPress serves a virtual robots.txt that blocks the admin area but keeps the AJAX endpoint open, which many themes and plugins need:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Online stores. Block internal search results and endless filter combinations that create thousands of near-duplicate URLs, but never block the product and category pages you want to rank:

User-agent: *
Disallow: /search
Disallow: /*?*sort=
Disallow: /cart
Disallow: /checkout

Staging sites. Don't rely on robots.txt alone. Password-protect staging. If you do use Disallow: / there, make it part of your launch checklist to confirm production doesn't have it.

The robots.txt mistakes that cost traffic#

Disallow: / on the live site. Two lines that block every crawler from everything. Usually copied from staging during a launch or a migration.

Blocking CSS and JavaScript. Google renders pages like a browser. If it can't load your CSS and JS, it may see a broken page and misjudge its content and mobile-friendliness.

Using robots.txt to remove pages from Google. If a page is already indexed, blocking it stops Google from crawling it, which also stops Google from seeing the noindex tag you added. Remove the block, add noindex, and let Google recrawl.

Blocking AI search bots by accident. A "block all AI" rule copied from a 2023 blog post can remove you from ChatGPT and Claude search answers. Decide training and search separately.

Relative sitemap URLs and typos. Sitemap: /sitemap.xml is not a valid sitemap line, and Dissalow is silently ignored. Crawlers skip rules they don't understand.

Forgetting subdomains. Your blog on blog.example.com or shop on shop.example.com uses its own robots.txt.

How to test your robots.txt#

1. Check the file loads. Open yourdomain.com/robots.txt in a browser. It should load as plain text with a 200 status.
2. Test specific URLs. MyKavo's free Robots.txt Tester tells you whether a URL is blocked for Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers, and shows the exact rule that decides it.
3. Use Search Console's robots.txt report. Google replaced its old robots.txt Tester with the robots.txt report in late 2023. It shows the robots.txt files Google found for your site, when they were last crawled, and any parsing errors or warnings. You can also request a recrawl after you fix the file.
4. Inspect important URLs. Search Console's URL Inspection shows whether Googlebot is allowed to crawl a specific page.

Watch your robots.txt for changes#

Robots.txt changes without anyone deciding to change it: a plugin rewrites it, a migration copies the wrong file, a developer tests something and forgets to revert it. Nothing errors. Pages just stop being crawled, and you find out weeks later from a traffic graph.

MyKavo checks your robots.txt on every scan and alerts you when its content changes, when it disappears, and, as a critical alert, when it starts blocking all crawlers. For the other way a page silently drops out of search, see our guide to detecting an accidental noindex.

The bottom line#

Keep robots.txt simple: block what crawlers genuinely shouldn't crawl, point to your sitemap, and never block CSS, JavaScript or pages you want to rank. Make a deliberate decision about AI crawlers, treating training and search separately. Test every change, and watch the file, because the most damaging robots.txt changes are the ones nobody meant to make.

Test your file now with the free Robots.txt Tester, then start free with MyKavo so you know the day your robots.txt changes.