Free SEO tool
Robots.txt Generator
Build a robots.txt file that tells search engine crawlers which parts of your site they may crawl. The tool checks for common mistakes as you go and lets you test any path before you upload the file.
- Free, no signup
- Built-in path tester
- Nothing you enter is stored
Start from a template (optional)
Templates are starting points, not recommendations for every site. Only block paths that you understand and have checked.
Groups and rules
Each group lists one or more crawlers (user-agents) and the paths they may or may not crawl. Use * for all crawlers. In paths, * matches anything and $ marks the end of the URL. Paths are case-sensitive.
Check, test and download
Checks
Test a path
The tester follows Google's matching rules and uses the group whose user-agent exactly matches the crawler you choose, otherwise the * group. Some crawlers, such as Google's specialised crawlers, may fall back to a more general group; check their documentation if that matters to you.
How it works
How to create and publish a robots.txt file
- Choose a starting point
Start with "Allow all" or a template, then adjust it to your site's real structure.
- Add groups and rules
Add only the paths you need to block, and your sitemap URL.
- Fix issues and test
Resolve any errors and test important pages to make sure they are allowed.
- Upload and verify
Upload robots.txt to your site's root, then check it in Google Search Console's robots.txt report.
Worked example
Example: a typical WordPress site
robots.txt
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.example.com/sitemap.xml
What it means
- /wp-admin/options.php
- Blocked by
Disallow: /wp-admin/ - /wp-admin/admin-ajax.php
- Allowed: the Allow rule is longer, so it is more specific
- /blog/my-post/
- Allowed: no rule matches
admin-ajax.php stays open because some themes and plugins use it to load content on public pages. The sitemap line helps crawlers find your pages. These results are what the tester above reports for this file.
Practical guidance
What robots.txt can and cannot do
robots.txt is a set of crawling instructions for well-behaved bots. It is useful for keeping crawlers away from areas that offer no value in search, such as admin screens, internal search results or endless filtered URLs, so they spend their time on your important pages.
Common mistakes to avoid
- Blocking the whole site by accident, often by copying a staging site's
Disallow: /to the live site. - Using robots.txt to hide pages from search. A blocked page can still be indexed if it is linked from elsewhere, and Google cannot see a noindex tag on a page it is not allowed to crawl.
- Blocking CSS, JavaScript or images that search engines need to render your pages.
- Listing secret paths. The file is public; listing a private folder tells everyone it exists.
- Forgetting that paths are case-sensitive:
/Private/and/private/are different.
After uploading, Google usually picks up changes within a day, as it caches robots.txt for up to 24 hours. You can check the file Google has fetched in the robots.txt report in Google Search Console. If you are not sure why pages are missing from Google, our guide on why pages are not indexed covers the most common causes.
What this tool does not do
- It does not fetch or edit the robots.txt file on your website.
- It cannot guarantee how every crawler interprets the file; behaviour varies, and bad bots ignore it.
- It does not remove pages from search results. Use noindex or Search Console's removals tool for that.
Robots.txt: common questions
More free tools
Related tools
Technical SEO
Not sure what your site should block?
We review robots.txt, sitemaps and indexing as part of technical SEO, and fix crawl problems that keep pages out of Google. Ask us for a free website and SEO review.