Bitdoze Logo

How to Block AI Crawlers & Safeguard Your Website (2026)

Block AI crawlers from stealing your website content. Complete guide: Cloudflare one-click blocking, robots.txt setup, Nginx rules, and AI Labyrinth. Updated for 2026.

DragosDragos31 min read
How to Block AI Crawlers & Safeguard Your Website (2026)

AI crawlers now account for over 20% of all web traffic, and with 170+ AI bots actively scraping content, blocking AI crawlers matters for every website owner. Whether your content is training AI models without compensation, you’re losing referral traffic to AI-powered search, or your VPS bandwidth is disappearing, this guide covers every method available in 2026, from Cloudflare’s free one-click toggle to server-level Nginx rules and the AI Labyrinth defense.

The approach that works is layered defense. No single method stops all bots. I’ll walk through seven methods, from the simplest (one click in Cloudflare) to the most granular (server-level configs and new web standards), and show you how to verify each one actually works.

Why block AI crawlers in 2026?

Three reasons this matters more now than it did in 2024:

Bandwidth and cost. AI bots are responsible for over 20% of all HTML requests across Cloudflare’s network. Aggressive crawlers like ByteDance’s Bytespider accessed over 40% of Cloudflare-protected sites. Anthropic’s ClaudeBot saw an 800% volume increase in late 2025. If you’re running a VPS with bandwidth caps, these bots are eating into your allocation for zero benefit.

Content ownership. Your content is being used to train commercial AI models without compensation or attribution. The visitors don’t come to your site, don’t subscribe to your newsletter, don’t click your affiliate links. The AI company profits; you get the bandwidth bill.

Legal context. The EU AI Act (Article 53, in force August 2, 2025) now requires General-Purpose AI providers to implement copyright compliance policies and respect machine-readable opt-outs like robots.txt. If you’re in the EU, your robots.txt file now carries legal weight. More on that in Method 6.

Not all AI bots are bad

Not all AI crawlers are harmful. Search-indexing bots like OAI-SearchBot and Claude-SearchBot may drive traffic to your site through AI-powered search results. Training crawlers like GPTBot and ClaudeBot won’t send you a single visitor. This guide helps you decide what to block and what to allow.

The AI crawler landscape in 2026: who’s crawling and why

The AI bot ecosystem has exploded since 2024. The community-maintained ai-robots-txt repository (4,000+ GitHub stars) now tracks 170+ AI crawlers. For practical purposes, you need to know about three categories, which map directly to Cloudflare’s new classification system:

The Perplexity stealth crawler warning

This is the most important cautionary tale for anyone relying solely on robots.txt: In August 2025, Cloudflare caught Perplexity using stealth, undeclared crawlers that masqueraded as Chrome on macOS. When blocked via their declared PerplexityBot user-agent, they switched to a generic browser fingerprint, used rotating IPs across multiple ASNs, and in some cases didn’t even fetch robots.txt. Cloudflare observed 3-6 million daily stealth requests and de-listed Perplexity as a verified bot.

If you’re interested in how Perplexity’s AI search works and its approach, check out our article on Perplexity’s approach to AI search.

The lesson: robots.txt is a polite request, not a security barrier. Technical enforcement at the WAF level is the only thing that actually stops bad actors.

This is the simplest and most effective method. Cloudflare introduced a one-click toggle in July 2024 that blocks known AI scrapers and crawlers automatically, no WAF rule creation needed. It’s available on the Free plan and Cloudflare updates the bot fingerprints as new crawlers emerge.

  • Cloudflare account (free tier works)
  • Domain added to Cloudflare with DNS managed there
  • Proxy enabled (orange cloud) on your DNS records

Step 1: Enable the Cloudflare proxy

Make sure your site’s DNS records have the orange cloud (proxy) enabled, not just DNS-only. If you need help setting up Cloudflare for the first time, see how to deploy your site on Cloudflare or set up a blog on Cloudflare’s free tier.

Step 2: Navigate to Security > Bots

In the Cloudflare dashboard, go to Security > Bots. You’ll see the “AI Scrapers and Crawlers” section.

Step 3: Toggle “Block AI bots” to ON

Enable the toggle. That’s it, Cloudflare handles the rest.

Step 4: Verify it works

Wait 5 minutes, then test:

curl -s -o /dev/null -w "%{http_code}" -A "GPTBot/1.0" https://yoursite.com
# Should return 403 or 406

You can also check the Security > Analytics dashboard after a few hours to see how many AI requests have been blocked.

Cloudflare has blocked over 416 billion AI bot requests since July 2025, and 2.5 million websites have enabled this protection. This is the fastest path: one click and you’re covered. Cloudflare automatically updates bot signatures as new crawlers appear.

Method 2: Cloudflare granular AI bot policies (Search, Agent, Training)

If you want fine-grained control over which types of AI bots can access your site, Cloudflare’s new three-category system (launched July 2026) gives you exactly that. This replaces the older “Block AI bots” managed preset.

Navigate to Security Settings > Configure AI bot policies in your Cloudflare dashboard. You’ll see three categories:

Category What it controls Default (from Sept 15, 2026)
Search Crawlers indexing content for AI search (OAI-SearchBot, etc.) Allow
Agent User-triggered agents acting in real time (ChatGPT-User, etc.) Block on pages with ads
Training Crawlers collecting data for model training (GPTBot, ClaudeBot, etc.) Block on pages with ads

For each category, you can choose:

  • Block: block on all pages
  • Block on pages with ads: middle ground (new default for Training and Agent)
  • Allow: let them through

The “Block on pages with ads” option is clever. It lets AI bots access your content pages (which may drive traffic) while blocking them on monetized pages (where they’d cost you ad revenue without contributing).

To check your results, go to Security > Analytics after 24 hours and review blocked vs. allowed requests per category.

The old “Block AI bots” preset deprecates on September 15, 2026. If you’re currently using it, migrate to the new three-category system before that date.

What happened to the old WAF custom rule method?

The old method used a WAF custom rule with the expression (cf.verified_bot_category eq "AI Crawler"). This still works and doesn’t consume your custom rule slots differently, but it’s being superseded by the new three-category system which offers finer control (block training separately from search). If you already have this rule deployed, it’s fine to keep it, but the new system at Security Settings > Configure AI bot policies is the recommended path going forward.

Method 3: AI Labyrinth: waste bot resources instead of blocking

Cloudflare’s AI Labyrinth (launched March 2025) takes a different approach: instead of just blocking bots, it serves them AI-generated decoy content through hidden links that no human would ever click. This does three things:

  1. Wastes the crawler’s compute resources processing fake content
  2. Acts as a honeypot: if a crawler follows the hidden links, Cloudflare knows it’s a bot
  3. Feeds Cloudflare’s ML models to identify new bot patterns faster

It’s available on the Free plan as an opt-in toggle at Security > Bots (same page as the AI Scrapers toggle from Method 1).

AI Labyrinth works alongside blocking, it’s not a replacement. Enable it after turning on Method 1 for maximum defense-in-depth. The hidden links are invisible to human visitors, so there’s zero impact on your site’s user experience.

This is the kind of defense I like: it doesn’t just reject bad traffic, it actively makes the attacker’s job harder and more expensive.

Method 4: Block AI crawlers using robots.txt (complete list)

robots.txt is the web standard for declaring which crawlers can access your content. It’s now legally significant in the EU under the AI Act (more on that in Method 6). But keep in mind: robots.txt is a request, not a barrier. Well-behaved bots respect it. Bad actors don’t. Always combine this with Cloudflare or server-level blocking.

Here’s a purpose-annotated robots.txt that covers the major AI crawlers:

The ai-robots-txt community repository (4,000+ stars) maintains an even more comprehensive list with 170+ bots. It also provides ready-made configs for Nginx, Apache, Caddy, and HAProxy.

robots.txt is a polite request, not a security barrier. Perplexity was caught ignoring it entirely and spoofing browser fingerprints. Always combine with Cloudflare (Methods 1-3) or server-level blocking (Method 5) for actual protection.

My robots.txt changes aren't taking effect

Common causes:

  • Crawlers cache robots.txt — it can take 24-48 hours for major crawlers to re-fetch and respect changes
  • File not at the root URL — robots.txt must be at https://yourdomain.com/robots.txt, not in a subdirectory
  • Syntax errors — a malformed file may be ignored entirely. Use Google’s robots.txt tester in Search Console to validate
  • Your robots.txt is blocked — if you have a WAF rule blocking all bots, they can’t read the file either (the Cloudflare toggle handles this correctly — it allows robots.txt access)

Method 5: Server-level AI crawler blocking (Nginx, Apache, Caddy)

For self-hosters running their own VPS, server-level user-agent blocking adds defense-in-depth below the WAF layer. If you’re self-hosting on a VPS like Hetzner Cloud, this is your last line of defense when bots bypass or don’t go through Cloudflare.

The ai-robots-txt community repo provides ready-made config files for all three major web servers. To secure your VPS against malicious traffic more broadly, combine this with tools like CrowdSec.

Server-level blocking is your last line of defense. Bots can spoof user-agents, so this shouldn’t be your only method. Combine with Cloudflare (Methods 1-3) for complete protection.

Nginx blocking isn't working

Check these common issues:

  • Run nginx -t to verify your config has no syntax errors
  • Ensure the if block is inside the correct server {} context (not inside a location {} block)
  • Check for conflicting location blocks that might override the server-level rule
  • Verify nginx actually reloaded: systemctl status nginx
  • Test locally first: curl -A "GPTBot/1.0" http://localhost before testing through Cloudflare

Method 6: TDM Reservation Protocol and machine-readable opt-outs

Several new web standards have emerged to give publishers machine-readable ways to declare AI usage rights. These complement robots.txt and carry increasing legal weight, especially in the EU.

TDM Reservation Protocol (W3C standard)

The TDM Reservation Protocol lets you declare rights reservations via HTTP response headers. Add this to your Nginx config:

# Add to your server {} or location {} block
add_header tdm-reservation "1";
add_header tdm-policy "https://yourdomain.com/tdm-policy.json";

For Apache:

Header set tdm-reservation "1"

This is legally significant under the EU AI Act — GPAI providers must detect and respect these signals.

Cloudflare Content-Signal

If you use Cloudflare’s managed robots.txt, it now automatically prepends a Content-Signal directive:

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

The use= parameter signals content usage levels: immediate, reference, or full. This is Cloudflare’s way of making opt-outs machine-readable without requiring manual config.

Other emerging standards

  • ai.txt — Proposed standard for AI usage permissions (no-training, no-inference, Allow-RAG). Placed at site root or /.well-known/ai.txt
  • llms.txt — A Markdown file at site root that gives LLMs a curated summary of key content. Not a blocking tool — it’s an AI SEO tool for controlling how LLMs understand your site
  • RSL 1.0 (Really Simple Licensing) — Open standard (December 2025) supplementing robots.txt with licensing categories and contribution payment models. Backed by Yahoo, Ziff Davis, and O’Reilly Media

EU AI Act: robots.txt now has legal weight

If you’re in the EU, a properly configured robots.txt carries legal weight under AI Act Article 53 (in force since August 2, 2025). The Hamburg Higher Regional Court confirmed in December 2025 that natural-language opt-outs in terms of use are insufficient — opt-outs must be machine-readable. In the US, robots.txt has no direct legal enforceability under current law. Either way, technical enforcement matters more than legal threats.

Method 7: IP blocking and supplementary methods

IP blocking is the oldest trick in the book but the least practical at scale — AI companies use cloud IPs that rotate frequently. Still, it has a place as a supplementary method.

Blocking AI crawler IPs with ufw

If you’re seeing heavy traffic from specific IPs, you can block them at the firewall level. Check OpenAI’s bot documentation for current IP ranges (they change over time).

# Example — verify current ranges before running
sudo ufw deny proto tcp from 23.98.142.176/28 to any port 80
sudo ufw deny proto tcp from 23.98.142.176/28 to any port 443
sudo ufw deny proto tcp from 40.84.180.224/28 to any port 80
sudo ufw deny proto tcp from 40.84.180.224/28 to any port 443

Caveat: IP ranges change. Check the docs for current values. Also note that if you’re running Docker, containers may bypass ufw rules — see our guide on firewall rules that actually work with Docker.

Identifying the IPs: Check your web server logs for AI crawler traffic:

grep -E "GPTBot|ClaudeBot|CCBot|Bytespider" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn
Cloudflare WAF IP Access Rules

If you’re on Cloudflare, go to Security > WAF > Tools to add IP Access Rules. You can block specific IPs or ranges without consuming WAF rule slots. Enter the IP, select “Block,” and add a note for why.

This is useful for blocking specific abusive IPs you’ve identified in your logs, rather than trying to block entire AI company IP ranges.

Other supplementary methods

Meta tags. The noai and noimageai meta tags are still valid. Add them to your HTML <head>:

<meta name="robots" content="noai, noimageai">

Compliant AI companies check for these, but they’re easy to ignore technically.

WordPress plugins. If you’re on WordPress:

  • Block AI Crawlers (by bobmatyas, updated November 2025) — generates robots.txt blocking common AI crawlers and adds noai/noimageai meta tags
  • Known Agents (formerly Dark Visitors) — tracks the AI crawler landscape and auto-generates your robots.txt

For more WordPress-specific options, see our guide on WordPress anti-scraping plugins.

Netlify users. Netlify offers a User Agent Blocker extension (Edge Function-based) that blocks AI crawlers from the project dashboard — worth checking if you host static sites there.

DNS-level blocking. You can also block unwanted traffic at the DNS level with NextDNS, which adds another layer of protection before requests even reach your server.

IP blocking isn't catching all bots

This is expected. IP blocking has fundamental limitations:

  • AI companies use cloud infrastructure (AWS, Azure, GCP) with IPs that rotate
  • Bots can use residential proxies or VPNs to change IPs
  • User-agent spoofing means the same IP can appear as any browser
  • Blocking IP ranges may accidentally block legitimate cloud services

This is why IP blocking is supplementary — not primary. Use Cloudflare (Methods 1-3) for reliable blocking.

How to verify your AI crawler blocks are working

Don’t just enable protections and assume they work. Here’s how to confirm:

Test with curl using AI bot user-agents:

# Test if GPTBot is blocked
curl -s -o /dev/null -w "%{http_code}" -A "GPTBot/1.0" https://yoursite.com
# Should return 403 or 406

# Test if robots.txt is still accessible (it should be)
curl -s -A "GPTBot/1.0" https://yoursite.com/robots.txt
# Should return your robots.txt content

# Test ClaudeBot
curl -s -o /dev/null -w "%{http_code}" -A "ClaudeBot/1.0" https://yoursite.com
# Should return 403

Quick verification

If the curl commands return 403, your blocking is working. If they return 200, something isn’t configured correctly — check your Cloudflare toggle, Nginx reload status, or robots.txt syntax.

Check Cloudflare Security Analytics:

Go to Security > Analytics in the Cloudflare dashboard. After 24 hours you should see blocked requests in the “Bot Traffic” section. Filter by bot category to see training, search, and agent traffic separately.

Search server logs:

# Check for AI crawler hits in the last 24 hours
grep -E "GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot" /var/log/nginx/access.log | tail -20

# Count blocks vs. gets
grep -E "GPTBot|ClaudeBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Note: If you updated robots.txt, compliant crawlers cache it for 24-48 hours before re-fetching. Don’t expect immediate results from robots.txt changes.

The future: Pay Per Crawl and AI content monetization

Cloudflare introduced Pay Per Crawl (private beta, July 2025) as a third option beyond “allow” or “block” — monetize. Instead of blocking AI crawlers outright, publishers can charge per request. It uses HTTP 402 (Payment Required), Web Bot Auth with Ed25519 signatures, and Cloudflare acts as Merchant of Record.

The idea: AI companies get access to quality content, publishers get paid, and Cloudflare handles the payment infrastructure. For solo operators who’ve spent years building content libraries, this could eventually become a revenue stream.

Pay Per Crawl is in private beta as of July 2026. You can configure it from your Cloudflare dashboard, but widespread adoption depends on AI companies signing up. We’ll update this guide when it becomes generally available.

Conclusion: build a layered defense against AI crawlers

No single method blocks all AI crawlers. The Perplexity stealth crawler scandal proved that even well-configured robots.txt can be ignored. Here’s the defense stack I recommend:

  • Cloudflare one-click toggle (Method 1) — enable this first, it’s one click and free
  • Granular AI bot policies (Method 2) — fine-tune Search vs. Agent vs. Training access
  • AI Labyrinth (Method 3) — opt-in, wastes bot resources for extra defense-in-depth
  • robots.txt (Method 4) — legal compliance and legitimate bot guidance
  • Server-level blocking (Method 5) — Nginx/Apache/Caddy for self-hosters who want a safety net below Cloudflare
  • TDM Reservation headers (Method 6) — especially important if you’re in the EU
  • Verify your blocks — test with curl, check analytics, grep your logs

Combining Cloudflare (Methods 1-3) with robots.txt (Method 4) covers 99% of threats for most sites. Self-hosters on a VPS should add server-level blocking (Method 5) as a safety net.

If you’re serious about taking control of who accesses your content, the layered approach works. For broader protection of your online presence, you can also block unwanted traffic at the DNS level and explore self-hosted privacy solutions that keep your data under your control.

Deploy Your Site on Cloudflare

FAQ

Will blocking AI crawlers hurt my SEO?

No. Search engine crawlers (Googlebot, Bingbot) are completely separate from AI training crawlers. The methods in this guide target AI-specific bots. Google’s search crawler has its own user-agent and is not affected by blocking GPTBot, ClaudeBot, or any other AI training crawler. The only overlap is Google-Extended, which controls Gemini/Vertex AI training — blocking it does not affect Google Search indexing.

Should I block all AI bots or just training crawlers?

It depends on your goals. Training crawlers (GPTBot, ClaudeBot, CCBot) provide zero referral traffic — block these without hesitation. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) may drive visitors to your site through AI-powered search results — consider allowing these. Agent bots (ChatGPT-User, Claude-User) are user-triggered and not bulk crawling — these are generally safe to allow. The recommended robots.txt in Method 4 is annotated to help you decide.

Does robots.txt actually stop AI crawlers?

Only for compliant bots. Major companies (OpenAI, Google, Anthropic, Apple) generally comply with robots.txt — they have legal teams and public commitments. But Perplexity was caught in August 2025 ignoring it entirely, using stealth crawlers that spoofed Chrome’s user-agent. Always combine robots.txt with Cloudflare or server-level blocking for real protection.

Is Cloudflare's AI bot blocking really free?

Yes. The one-click “AI Scrapers and Crawlers” toggle and all three AI category policies (Search, Agent, Training) are available on Cloudflare’s Free plan. No WAF rule slots are consumed. AI Labyrinth is also free (opt-in). The only paid feature is Pay Per Crawl, which is in private beta.

What about the EU AI Act? Do I need to do anything special?

If you’re in the EU, ensure your robots.txt is properly configured — it now has legal weight under AI Act Article 53 (in force since August 2, 2025). Adding TDM Reservation headers (Method 6) provides additional legal protection. The Hamburg court ruling in December 2025 confirmed that natural-language opt-outs in terms of use are not sufficient — opt-outs must be machine-readable (robots.txt, TDM headers). In the US, robots.txt has no direct legal enforceability under current law, but technical enforcement still matters.

#Ai#Cloudflare#Security

Related Posts