Quick answer: You can block AI crawlers with robots.txt by adding a group for each crawler name and disallowing the paths you want to protect. This works best for crawlers that respect the file. It does not work as a security tool, and it can cost you visibility in AI answers when you block too much.
What Robots.txt Actually Controls
Robots.txt is a small text file that lives at the root of your website. Crawlers read it before they fetch your pages, and it tells them which parts of your site they may request. It is a request, not a lock. A polite crawler follows it. A rude crawler can ignore it. A human visitor never sees it unless they look for it.
Blocking a page in robots.txt also does not remove it from search results on its own. If other sites link to that page, a search engine may still show its address even when it cannot fetch the content. If you want a page kept out of an index, you usually need a different tool, such as a noindex tag on the page itself. Robots.txt controls fetching. It does not control every form of listing, quoting, or reuse. Keep that limit in mind before you treat a robots.txt change as a privacy fix.
Which AI Crawlers Should You Know
AI crawlers do not all do the same job, so a single blanket block is often too blunt. Some crawlers collect training data. Others fetch pages in real time to answer a question. Others check facts or fetch a page when a user shares a link. Knowing the job helps you decide whether to block, allow, or limit access.
- GPTBot: OpenAI crawler used for training and related data collection. Blocking it is a common choice for publishers who do not want their content used to train models.
- OAI SearchBot: OpenAI crawler used for search features. Blocking it can reduce your presence in ChatGPT search style answers.
- ClaudeBot: Anthropic crawler. It is often associated with training and data collection for Claude models.
- CCBot: Common Crawl bot. It collects a public web archive used by many researchers and AI builders, so blocking it has a wider effect than blocking one company.
- Applebot Extended: Apple crawler token used for AI training opt out, separate from the main Applebot used for Siri and Spotlight search.
- PerplexityBot: Perplexity crawler used to fetch pages for answers. Blocking it can remove you from Perplexity answers while leaving other search visibility intact.
- Google Extended: Google token that lets you opt out of AI training uses without blocking Google Search crawling. This is useful when you want search traffic but not training use.
How To Block AI Crawlers With Robots.txt
Start with a backup of your current robots.txt file. Then add one group per crawler you want to block. A group names the crawler and then states the rule. For a full block of GPTBot, the group looks like this in plain text.
User agent: GPTBot
Disallow: /
That pair tells GPTBot not to fetch any path on your site. Repeat the same pattern for each crawler you want to block, with its own user agent line. You can also block a crawler from only part of your site. For example, you might allow it to read your free guides but block it from a members area or a paid course section. In that case, use a more specific disallow path, such as the folder for that section, and leave the rest open.
When You Should Not Block AI Crawlers
Do not block AI crawlers just because AI search feels new or uncertain. For many brands, AI answers are becoming a discovery channel, much like search results and social feeds. If your content answers clear questions, explains a process, or compares options, an AI answer can send you curious visitors who already understand the topic. Blocking every AI crawler removes that chance before you know whether it helps you.
Our guide to how to get cited by Meta AI shows the other side of this choice. Citation depends on clear facts, credible sourcing, and pages that answer engines can read and trust. A site that blocks all AI access cannot be cited, no matter how good its content is. If your goal is brand visibility, selective access usually beats a total block.
Smarter Alternatives To A Total Block
A total block is simple, but it is rarely the smartest option. Start by asking what you actually want to protect. If the worry is training use, use training specific opt outs where they exist, such as Google Extended, instead of blocking search style crawlers. If the worry is a paywall, block only the paid section and keep free previews open. If the worry is scraping speed, use rate limits, caching, and server rules in addition to robots.txt.
Measuring the effect is just as important as making the change. Our guide on how to measure AI search traffic explains practical ways to track referrals, branded searches, and answer citations over time. Before you block, record a baseline for traffic from AI tools, direct visits, and branded search. After you block, compare the same numbers for at least four weeks. If referrals fall and no other benefit appears, you have evidence to reverse the decision.
A Simple Decision Framework
Use this framework when a team member asks for a blanket AI block. First, name the asset at risk and the harm you fear. Second, check whether a narrower rule solves the same problem. Third, estimate the visibility you will give up. Fourth, set a review date. Many sites block in a panic during a news cycle and never revisit the choice. A review date forces a calm second look once the headlines pass.
How To Monitor And Maintain Your Rules
Robots.txt is not a set and forget file. AI companies rename crawlers, add new ones, and change their documentation. A rule that worked last year may miss a new crawler this year. Review your file at least once a quarter, and review it again after any major AI announcement that affects your niche. Keep a short change log so future team members can see who changed what and why.
Server logs are your best source of truth. Look for user agents that fetch a large number of pages, ignore your rules, or arrive in bursts at odd hours. Some bots claim to respect robots.txt but still fetch disallowed paths. For those, you may need stronger controls at the server or firewall level, such as blocking an IP range or requiring a challenge for suspicious traffic. Be careful with aggressive firewall rules, because they can also block real users and useful tools.
Also watch for unintended side effects. If search traffic drops after a robots.txt change, check whether you blocked a search crawler by accident. If a partner tool stops working, check whether it uses a crawler name you just blocked. If an AI answer that used to cite you disappears, check whether the change was caused by your block or by a content change on the page. Correlation is not proof, so test one change at a time when you can.
Building a clear brand identity helps answer engines recognise you even when competition is strong. Our guide to entity SEO for AI search explains how consistent names, facts, and relationships across the web help machines understand who you are. That work pairs well with a thoughtful robots.txt policy, because it gives AI systems clean, trustworthy material to cite when you choose to allow access.
Frequently Asked Questions
Does robots.txt stop AI companies from using my content completely? No. It asks polite crawlers not to fetch your pages, but it does not delete content already collected, and it does not stop a crawler that ignores the file. For stronger protection, combine robots.txt with access controls, licensing terms, and direct requests where needed.
Can I block training crawlers but keep AI search visibility? In many cases, yes. Use training specific tokens where they exist, such as Google Extended, and block training crawlers by name while allowing search style crawlers. Check each platform’s documentation, because the tokens and their effects differ.
Will blocking AI crawlers hurt my Google rankings? Blocking AI only crawlers should not hurt classic Google Search rankings if you leave Googlebot untouched. Trouble starts when a wildcard rule or a typing error blocks Googlebot as well, so test the file carefully after every change.
How long does it take for a block to take effect? Polite crawlers usually respect a new robots.txt file on their next visit, which can be hours or days depending on crawl frequency. Pages already collected may remain in use for longer, so do not expect an instant removal from every AI system.
Should small sites bother with AI crawler rules? Yes, but keep it simple. A short, named list of major AI crawlers is enough for most small sites. Revisit the file when your content becomes more valuable, when you launch paid material, or when you see unusual crawl activity in your logs.
Work With SEO Spider
If you are unsure whether to block AI crawlers or open your site to AI answers, talk to SEO Spider. Our GEO services help you protect valuable content while building visibility in AI search, so you do not have to choose between safety and discovery. Contact SEO Spider today for a clear review of your robots.txt file, your AI visibility, and the next steps that fit your business.