Robots.txt Generator
Create a custom robots.txt file to guide search engine bots on how to crawl and index your website effectively. Prevent indexing of sensitive directories.
User-agent: * Allow: / Disallow: /wp-admin/ Disallow: /temp/
Warning: Incorrect instructions in robots.txt can lead to your entire website being removed from Google search results. Double check your disallow paths.
About the Robots.txt Generator
The robots.txt file is your website's traffic controller for search engines: it tells Google, Bing, and AI crawlers which parts of your site to crawl and which to skip. A missing file wastes crawl budget on admin pages and duplicate URLs; a misconfigured one can silently remove your entire site from Google — we've seen a single wrong line cost a business every ranking it had.
This generator builds a correct robots.txt for your situation: allow or block specific bots (including AI crawlers like GPTBot and ClaudeBot), protect admin and private paths, and point crawlers at your sitemap. Copy the output and upload it to your site's root.
How to Create Your Robots.txt
- 1Choose your default policy — most business sites should allow all crawlers.
- 2Add paths to block, such as /admin, /cart, or internal search results.
- 3Decide how to treat AI crawlers (GPTBot, ClaudeBot, PerplexityBot) — allowing them makes you citable in AI answers.
- 4Add your sitemap URL, copy the file, and upload it to yourdomain.com/robots.txt.
Frequently Asked Questions
What is a robots.txt file?
A plain-text file at your domain root (yourdomain.com/robots.txt) that gives crawling instructions to search engine bots. It controls what gets crawled — though not necessarily what gets indexed; for that you also need meta robots tags or X-Robots-Tag headers.
Can robots.txt hurt my SEO?
Badly, if misconfigured. 'Disallow: /' blocks your whole site from crawling — a surprisingly common leftover from development. Blocking CSS/JS files can also hurt, because Google needs them to render your pages. Always test changes in Google Search Console's robots.txt report.
Should I allow AI crawlers like GPTBot and ClaudeBot?
For most businesses, yes. Allowing AI crawlers means ChatGPT, Claude, and Perplexity can cite your business when Ugandans ask them for recommendations — that's Answer Engine Optimisation (AEO), and it's where a growing share of discovery happens. Block them only if you have proprietary content you don't want in AI training data.
What should a typical business website block?
Admin areas (/admin, /wp-admin), checkout and cart pages, internal search results, and thank-you pages. Everything customers should find — products, services, blog — should stay crawlable, and your sitemap should be listed at the bottom of the file.
How do I know if my robots.txt is working?
Visit yourdomain.com/robots.txt to confirm it's live, then use Google Search Console's robots.txt tester. GSC's Page Indexing report also shows pages 'Blocked by robots.txt' so you can verify only intended paths are affected.