How to Block AI Crawlers in WordPress (and Still Get Cited)

AI companies send crawlers to WordPress sites every day. Some collect pages to train language models. Others fetch pages so an AI search engine can quote and link to you, or because a person just asked a chatbot about your page. Blocking all of them is easy. Blocking the right ones takes a little more thought.

This guide covers which AI crawlers visit WordPress sites, how to block the training ones while staying visible in AI search, what robots.txt cannot do, and how to check your rules work.

Which AI crawlers visit WordPress sites#

AI crawlers identify themselves with a name in the User-Agent header of each request. That name is what robots.txt rules match against. They fall into three groups.

Training crawlers#

These collect pages in bulk to build datasets for training AI models. Blocking them does not remove you from any search results.

AI search crawlers#

These build the index that AI search answers are drawn from. If they cannot read your pages, those engines have less reason to cite or link to you.

User-triggered fetchers#

These visit when a person asks an AI assistant something that needs your page, for example "summarize this article" or a question the assistant answers by opening your site.

Some operators state that their user-triggered fetchers may not follow robots.txt in the same way as their automated crawlers, because a person started the request. Check each operator's own crawler documentation if this matters to you.

You may also see other AI agents, such as Amazonbot, DuckAssistBot and MistralAI-User. The same thinking applies: find out what the operator says the bot is for, then decide.

Google and Apple work differently#

Google does not use a separate crawler for Gemini training. Instead it reads a robots.txt token called Google-Extended. Disallowing Google-Extended tells Google not to use the content it crawls for training Gemini models and for grounding in some Gemini products. It does not change how Googlebot crawls your site, and it does not affect your ranking in Google Search. AI Overviews are part of Google Search, so Google-Extended is not a way to opt out of them.

Apple has a similar token, Applebot-Extended, which controls whether pages crawled by Applebot are used to train Apple's AI models.

The decision: block training, allow AI search#

For most business sites, agencies and publishers, a sensible default is:

• Block training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, plus the Google-Extended and Applebot-Extended tokens). You get nothing back when your content goes into a training set.
• Allow AI search crawlers and user-triggered fetchers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User). These are the visits that can lead to a citation, a link, and a real person clicking through to your site.

There are exceptions. A site selling original content, such as a paid newsletter, may want to block AI search too. The point is to choose on purpose.

Do not block every AI bot by reflex: if a search crawler cannot read a page, it has nothing to quote. For the wider picture of how WordPress sites get found, see our WordPress SEO guide.

Where WordPress keeps robots.txt#

WordPress handles robots.txt in a way that confuses many site owners.

The virtual robots.txt#

A fresh WordPress install has no robots.txt file on disk. When something requests /robots.txt, WordPress generates one on the fly. On a typical public site it looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

You will not find this file over FTP. Plugins and themes can change it through the robots_txt filter.

A physical robots.txt file#

If you upload a real file named robots.txt to the site's root folder (the folder that holds wp-config.php in most installs), the web server serves that file directly and WordPress never runs. The virtual version, and anything plugins add to it, is ignored.

If you changed robots.txt in your SEO plugin and nothing happened, a physical file in the root folder is usually winning.

How to block AI crawlers in WordPress#

Pick one of these three methods. Do not mix them, or you will end up with rules in two places.

Method 1: your SEO plugin's robots.txt editor#

Most SEO plugins include an editor:

• Yoast SEO: Yoast SEO > Tools > File editor.
• Rank Math: Rank Math > General Settings > Edit robots.txt. Rank Math edits the virtual file, so the editor is locked if a physical file exists.
• All in One SEO: All in One SEO > Tools > Robots.txt Editor.

Method 2: edit or upload a physical file#

Create a plain text file called robots.txt, add your rules, and upload it to the site root. WordPress will no longer generate anything for you, so keep the default WordPress lines and your sitemap line.

Method 3: the robots_txt filter#

Developers can add rules to the virtual file from a small must-use plugin in wp-content/mu-plugins/, so a theme change does not remove them:

<?php
// wp-content/mu-plugins/ai-crawlers-robots.php
add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "\nUser-agent: GPTBot\n";
    $output .= "User-agent: ClaudeBot\n";
    $output .= "User-agent: CCBot\n";
    $output .= "User-agent: Bytespider\n";
    $output .= "User-agent: Meta-ExternalAgent\n";
    $output .= "User-agent: Google-Extended\n";
    $output .= "User-agent: Applebot-Extended\n";
    $output .= "Disallow: /\n";
    return $output;
}, 10, 2 );

This only works when there is no physical robots.txt file.

The rules to use#

Whichever method you choose, the finished file for "block training, allow AI search" looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# AI training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/wp-sitemap.xml

How this works:

1. Several User-agent lines in a row form one group, and the Disallow: / under them applies to all of them.
2. A crawler follows the most specific group that names it, and ignores the * group. So GPTBot reads only "Disallow: /".
3. OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered fetchers are not named, so they follow the * group and can read everything except /wp-admin/.

If you add a group for a search crawler, for example to allow it explicitly, remember that it will then ignore the * group. Repeat any lines you still want it to follow, such as Disallow: /wp-admin/.

Replace the sitemap address with your own. If you use Yoast SEO or Rank Math, the sitemap is usually /sitemap_index.xml rather than the core /wp-sitemap.xml.

What robots.txt cannot do#

robots.txt is a request, not a lock. A well-behaved crawler reads it and follows it. A badly behaved one ignores it, and nothing in robots.txt can stop it. It also does nothing about pages that were already collected before you added the rule.

If you need enforcement, block at the server, CDN or firewall level.

Block by user agent in .htaccess (Apache and LiteSpeed)#

Add this above the # BEGIN WordPress block in .htaccess, so WordPress does not overwrite it:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent) [NC]
RewriteRule .* - [F,L]
</IfModule>

Matching requests get a 403 Forbidden response. Nginx servers need an equivalent rule in the server configuration, which most managed hosts will add on request. Test carefully: a broken .htaccess file can take the whole site down with a 500 internal server error. Keep a copy of the working file before you edit it.

Note that Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers, so there is nothing to match in a server rule. They only work in robots.txt.

Use your CDN or firewall#

Many CDNs and web application firewalls, Cloudflare among them, offer settings to block or challenge known AI crawlers. These are often better than .htaccess rules because they can check whether a request really comes from the operator it claims to be, rather than trusting the User-Agent header. Anyone can send a request that says "GPTBot", so user agent matching alone can be fooled in both directions.

Security plugins can also block by user agent. Keep their rules consistent with robots.txt. Our WordPress security guide covers how these layers fit together.

Do not block search crawlers by accident#

Server-level blocks are where mistakes happen. A broad pattern such as bot or GPT can catch OAI-SearchBot, ChatGPT-User or even Googlebot. If AI search tools stop citing you after a change, check your server rules and firewall first.

How to check that your rules work#

After any change, verify it from the outside.

1. Open https://yoursite.com/robots.txt in a private browser window. Confirm you see the new rules, not the old file. If you use a CDN, purge its cache for robots.txt, since a cached copy can stay in place after you edit it.
2. Test specific crawlers against specific URLs. Our free robots.txt tester fetches your live robots.txt and tells you whether a URL is blocked for Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and CCBot, along with the exact rule that decides it. Test your homepage and one important page with GPTBot or CCBot (should be blocked) and with PerplexityBot, an AI search crawler (should be allowed).
3. Check Googlebot too. Make sure your new rules have not blocked normal search by mistake.
4. Check server blocks with a request. If you added an .htaccess rule, a developer can test it with a command like curl -A "GPTBot" -I https://yoursite.com/ and expect a 403 response, then repeat with a normal browser user agent and expect a 200.
5. Recheck after plugin updates. An SEO or security plugin update can change these rules.

See which AI bots actually visit your WordPress site#

Rules tell you what you asked for. They do not tell you who actually comes. For that you need to count the visits.

The free MyKavo WordPress plugin has an AI crawlers screen that shows:

• Who visits. Visits from OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Amazonbot, DuckAssistBot, Meta-ExternalAgent, MistralAI-User, CCBot and Bytespider, with the reason each one visits: AI search answers, a person asking about your page, or training.
• Which pages AI reads most, and visits per day for the last 30 days.
• Errors served to AI. Pages that answered a crawler with a 404 or 5xx error. Those are pages AI tools cannot quote, so they are worth fixing first. Our guide to the 404 error covers the usual causes.

Crawlers are counted by the name they send in the User-Agent header. The plugin records no IP addresses, no cookies and nothing about human visitors, and you can switch counting off on that screen. On the public site, the only work it does is a text match on the User-Agent header.

After you block training crawlers, you can see whether AI search bots keep visiting. And if a training crawler keeps showing up after you blocked it, that is your signal to add a server or firewall rule.

MyKavo also watches the pages that matter on your site and tells you when something important changes, such as a page switching to noindex, a canonical tag changing, or a page starting to return errors. If a plugin update changes your SEO settings and a page suddenly carries a noindex tag, that shows up as a change with the old and new values side by side. Read more on the MyKavo WordPress plugin page.