Robots.txt Generator

Create a correct robots.txt in seconds — with presets for WordPress, WooCommerce and Blogger, and simple switches for AI crawlers.

FreeWordPress & Blogger presetsAI bot controls

AI crawlers

Ticked = blocked. Robots.txt is a request; most large AI companies say they follow it.

Your robots.txt

Robots.txt rules that cause the most SEO problems

Page still not showing in Google? Read how to fix "Discovered – currently not indexed", and check any URL with the index checker. Want AI assistants to understand your site? Create an llms.txt file.

Frequently asked questions

What is a robots.txt file?

robots.txt is a plain text file at the root of your site (example.com/robots.txt) that tells crawlers which paths they may or may not crawl. Well-behaved bots follow it; it is not a security tool, so never use it to hide private data.

Does blocking a page in robots.txt remove it from Google?

No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.

What is the best robots.txt for WordPress?

Keep it short: block /wp-admin/ but allow /wp-admin/admin-ajax.php, and add your sitemap URL. Do not block /wp-content/ or /wp-includes/ — Google needs CSS and JavaScript files to render your pages.

What is the default robots.txt for Blogger?

Blogger blocks /search (label and search result pages) for all bots, allows everything else, gives the AdSense crawler (Mediapartners-Google) full access, and lists the sitemap. You can set a custom one in Settings → Crawlers and indexing.

Should I block AI bots like GPTBot?

It depends on your goal. Blocking training crawlers (GPTBot, Google-Extended, CCBot) asks those companies not to use your content for training. Blocking AI search crawlers (OAI-SearchBot, PerplexityBot) can also stop your pages being cited in AI answers, which may cost you traffic. This generator lets you choose each one.

Does Google-Extended affect Google Search rankings?

No. Google-Extended is a control token for use of your content in Gemini models. Blocking it does not change how Googlebot crawls or ranks your site in Search.

Write articles that rank, not just exist

SerpNuts researches search intent, competitors and content gaps, writes the article with AI and checks it against Rank Math, Yoast and AIOSEO rules. 10 optimizations + 1,500 AI words free every month — no card needed.

Start free