Robots.txt generator
Build a clean robots.txt that tells search engines what to crawl. Add your sitemap, keep admin pages out of crawlers’ way and optionally block AI training bots like GPTBot and CCBot.
How to use the Robots.txt Generator
- Choose the default rule — almost every site should allow all crawlers.
- List paths to keep out (admin, cart, internal search) and add your sitemap URL.
- Download robots.txt and upload it to the root of your domain (example.com/robots.txt).
Features
- Allow or block all crawlers
- Custom disallow paths and allow rule
- Sitemap line and crawl-delay
- One-click AI training bot blocking
Who is the Robots.txt Generator for?
A correct robots.txt keeps crawlers away from admin and cart pages and points Google to your sitemap — important for WordPress, Shopify and custom sites.
Shared premium SEO tools from ৳300/month — instant access, bKash & Nagad, 48-hour refund.
Frequently asked questions
Does robots.txt hide a page from Google?
No. It only asks crawlers not to visit. A blocked page can still be indexed if other sites link to it. To keep a page out of Google, use a noindex meta tag instead and let Google crawl it.
Should I block AI bots?
It is your choice. Blocking GPTBot, CCBot and Google-Extended stops those companies from using your content for AI training. It does not affect your normal Google rankings.
Where do I put the file?
In the top folder of your domain, so it opens at yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored.