Robots.txt Generator & AI Bot Blocker
Generate a compliant RFC 9309 robots.txt file, selectively block AI training crawlers like GPTBot and ClaudeBot without hurting SEO, and test URL rules live.
Robots.txt Generator & AI Bot Blocker: the tool
AI Crawler Control Grid (6/8 Blocked)
General Crawler Rules (`User-agent: *`) & Sitemap
Generated robots.txt
# Generated with DEESU Robots.txt & AI Bot Blocker
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
# AI Crawlers — Blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xml
Live URL Crawl Tester (RFC 9309)
Path "/blog/how-to-optimize-seo" did not match any Disallow rule in the * (Wildcard Fallback) group.
Need technical SEO architecture, automated sitemaps, or AI search readiness?
DEESU builds fast, crawl-optimized web applications with clean structured data, SSR/prerendering, and disciplined crawler governance.
How it works
How RFC 9309 Robots.txt Matching Actually Works
The Robots Exclusion Protocol—formally standardized in 2022 as RFC 9309—governs how compliant search engines and AI bots discover which URLs on your origin they are permitted to fetch. When a crawler visits your domain, it requests /robots.txt over HTTPS and looks for the most specific User-agent group matching its product token (for instance, GPTBot or Googlebot). If no dedicated group exists, it falls back to the wildcard User-agent: * block.
Crucially, modern crawlers do NOT evaluate Allow and Disallow rules top-to-bottom. Instead, RFC 9309 specifies longest-prefix matching: the rule with the longest matching path pattern wins regardless of line order. If an Allow rule and a Disallow rule match with the exact same character length, Allow wins. That is why WordPress sites can write Disallow: /wp-admin/ alongside Allow: /wp-admin/admin-ajax.php and safely expose the AJAX endpoint while blocking the rest of the admin dashboard.
Separating Search Indexing from AI Model Training Crawlers
Historically, blocking search bots meant disappearing from Google or Bing. Today, major AI labs operate separate user-agents for search retrieval versus foundation model training so publishers can stay visible in search results without donating proprietary content to training datasets:
- OpenAI splits crawling between
GPTBot(used to train future foundation models) andOAI-SearchBot(used to surface live citations in ChatGPT Search). BlockingGPTBotwhile allowingOAI-SearchBotpreserves search referrals. - Google uses
Googlebotfor Search indexing andGoogle-Extendedas an independent opt-out control for Gemini and Vertex AI training—blockingGoogle-Extendeddoes not impact your Google Search rankings. - Anthropic's
ClaudeBot, Common Crawl'sCCBot(which feeds dozens of open-weight LLM datasets), ByteDance'sBytespider, and Apple'sApplebot-Extendedcan each be targeted individually via explicitUser-agentblocks.
Common Robots.txt Mistakes That Hurt SEO and Crawl Budget
A single typo in /robots.txt can deindex an entire site overnight. The most frequent production incident occurs when a staging environment's Disallow: / file is accidentally deployed to production, cutting off organic traffic within days. Always test your generated file against critical landing pages before pushing to main.
Another common misconception is using robots.txt to hide private data or remove already-indexed pages from Google. Remember that robots.txt is publicly readable at https://yourdomain.com/robots.txt—listing secret URLs like /secret-investor-deck/ exposes them to attackers. Moreover, if external sites link to a disallowed URL, Google may still index the bare URL without a snippet. Use proper authentication (401/403) for private pages and <meta name="robots" content="noindex"> or X-Robots-Tag: noindex headers to deindex public URLs.
Declaring XML Sitemaps and Optimizing Crawl-Delay
Including an absolute Sitemap: https://yourdomain.com/sitemap.xml directive at the bottom of your robots.txt gives Googlebot, Bingbot, and AI search crawlers an immediate discovery path to every canonical URL without relying solely on link discovery. Note that Google ignores Crawl-delay (use Google Search Console for crawl rate), whereas Bingbot, DuckDuckBot, and Yandex honor it—so only add Crawl-delay if non-Google bots are straining your origin server.
Questions, answered
Googlebot. Blocking Google-Extended, GPTBot, ClaudeBot, CCBot, or Bytespider only opts your site out of AI training crawlers and has zero negative effect on Google Search indexing or rankings.https://example.com/robots.txt. In Next.js, Vite, or React static sites, place it inside your public/robots.txt folder (or use app/robots.ts in Next.js App Router). Subdomains such as docs.example.com require their own separate /robots.txt.User-agent group matching its own name (such as User-agent: GPTBot), it exclusively follows that specific block and ignores the generic User-agent: * group.robots.txt is a voluntary protocol respected by reputable search engines and major AI labs, not a security firewall. Malicious scrapers ignore it. Always enforce server-side authentication, WAF/Cloudflare Bot Fight rules, and rate limiting for sensitive endpoints.* to match any sequence of characters (e.g., Disallow: /*?sort=) and $ to anchor the end of a URL (e.g., Disallow: /*.pdf$). Our Live URL Rule Tester supports both * and $ anchors accurately.Related reading and help
Web DevelopmentRobots.txt & AI Crawlers in 2026: How to Control GPTBot, ClaudeBot & Perplexity Without Hurting Google SEO
Control AI scraping bots like GPTBot, ClaudeBot, and Bytespider via robots.txt in 2026 while preserving Google Search, AI Overviews, and crawl budget.
Nov 18, 202610 min read- Web Development
Open Graph & Social Meta Tags in 2026: The Complete Guide to Bulletproof Link Previews
Why social link previews break on LinkedIn, X, and Slack, how edge dynamic OG image generation works, and how to build metadata that maximizes clicks.
Oct 11, 20268 min read
Web DevelopmentNext.js 15 & 16 Caching Architecture: Mastering Data Cache, Request Memoization & Server Actions Invalidation
Master the 4-tier caching architecture in Next.js 15 and 16. Learn Data Cache, Router Cache, revalidateTag, unstable_cache, and zero-stale Server Actions.
Nov 8, 20266 min read- DEESU serviceWeb DevelopmentWeb applications built to be found and built to convert — SEO-minded from the first commit, from storefronts to internal portals.See the service
More free tools
- Open Graph & social meta tag generatorGenerate and test Open Graph, Twitter Card, and SEO meta tags with live previews for Google, X, LinkedIn, and Facebook.
- Google Play & App Store fee calculatorCalculate your exact take-home payout, compare standard 15% vs 30% cuts, and test if external billing saves money.
- App cost calculatorEstimate a price range for an Android app or website from its features, in your local currency.
- Google Play launch checklistAn interactive checklist covering the store listing, policies, quality checks and release steps.
- Project brief generatorAnswer a few questions and get a clear, copyable brief you can send to any development team.
- App privacy policy generatorAnswer a few questions and get a plain-English privacy policy and a delete-account page for your Android or iOS app.
- App icon generatorUpload one image and download every Android, iOS and favicon size, with adaptive icon safe-zone checks.
- Play Store listing optimizerCheck your title, short and full description against keyword, readability and Google Play policy rules.
- LLM token & cost calculatorForecast production AI API expenses across OpenAI, Anthropic & DeepSeek. Model prompt caching savings and routing arbitrage.
- JWT debugger & security inspectorDecode, verify, and audit JSON Web Tokens for alg:none bypasses, key confusion attacks, expiry status and leaked secrets.
- Cron expression generator & explainerBuild, validate and translate crontab schedules for Linux, AWS EventBridge, GitHub Actions and Kubernetes.
- JSON to TypeScript & Zod converterInstantly convert raw API JSON payloads into clean TypeScript interfaces and production Zod v3 validation schemas.
- Docker Run to Docker Compose converterConvert raw docker run commands into production Docker Compose v2 YAML with automated security hardening and healthchecks.
- cURL to Code converter (Fetch, Axios, Python, Go)Translate cURL commands into JavaScript fetch, Axios, Python requests, Go, Rust, and Node.js with 100% in-browser privacy.
- SVG optimizer & React JSX converterMinify and clean vector SVGs, strip Figma metadata, and convert to production React JSX, React Native, or Vue components.
- Regex tester, debugger & ReDoS auditorTest regular expressions live with capture groups, replace previews, plain-English explanations, and catastrophic backtracking detection.
- JSON formatter, validator & diff viewerBeautify, validate, minify, sort keys, and compare two JSON payloads side by side with nested JSONPath diff inspection.
- SQL formatter & query linterBeautify PostgreSQL, MySQL, SQLite, and T-SQL queries and catch SELECT *, missing WHERE clauses, and slow non-sargable scans.
- CSS to Tailwind v4 converter & OKLCH paletteConvert raw CSS declarations to Tailwind v4 utility classes and generate accessible 50–950 OKLCH color scales with @theme export.
- Base64, URL, HTML & SHA-256 hash toolkitEncode/decode Base64 (UTF-8 & URL-safe), URLs, HTML entities, Hex, and generate SHA-256, SHA-512 & HMAC signatures offline.
- UUID v7, UUID v4, ULID & NanoID generatorBulk generate cryptographically secure RFC 9562 UUID v7, UUID v4, ULID, and custom NanoIDs and decode embedded timestamps.
- Linux chmod permissions calculatorCalculate Linux octal and symbolic file permissions (755, 644, 600), configure SetUID/SetGID/Sticky bits, and export chmod commands.
- CSP & HTTP security headers generatorGenerate strict Content-Security-Policy, HSTS, and Permissions-Policy configs for Next.js, Vercel, Nginx, Apache, and Cloudflare.
- Markdown table & GitHub README builderCreate formatted Markdown tables visually, import CSV/TSV spreadsheets, and build GitHub alert callouts and Shields.io badges.
Tell us what you're building.
Send the problem, not a polished brief. We'll tell you what it actually takes to ship it: stack, timeline and cost, before you commit to anything.