Free SEO tool

Robots.txt Generator

Build a robots.txt file that tells search engine crawlers which parts of your site they may crawl. The tool checks for common mistakes as you go and lets you test any path before you upload the file.

  • Free, no signup
  • Built-in path tester
  • Nothing you enter is stored

Start from a template (optional)

Templates are starting points, not recommendations for every site. Only block paths that you understand and have checked.

Groups and rules

Each group lists one or more crawlers (user-agents) and the paths they may or may not crawl. Use * for all crawlers. In paths, * matches anything and $ marks the end of the URL. Paths are case-sensitive.

Use full URLs. Sitemaps apply to all crawlers, whichever group they are in.

Check, test and download


                

Checks

    Test a path

    The tester follows Google's matching rules and uses the group whose user-agent exactly matches the crawler you choose, otherwise the * group. Some crawlers, such as Google's specialised crawlers, may fall back to a more general group; check their documentation if that matters to you.

    How it works

    How to create and publish a robots.txt file

    1. Choose a starting point

      Start with "Allow all" or a template, then adjust it to your site's real structure.

    2. Add groups and rules

      Add only the paths you need to block, and your sitemap URL.

    3. Fix issues and test

      Resolve any errors and test important pages to make sure they are allowed.

    4. Upload and verify

      Upload robots.txt to your site's root, then check it in Google Search Console's robots.txt report.

    Worked example

    Example: a typical WordPress site

    robots.txt

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    
    Sitemap: https://www.example.com/sitemap.xml
    

    What it means

    /wp-admin/options.php
    Blocked by Disallow: /wp-admin/
    /wp-admin/admin-ajax.php
    Allowed: the Allow rule is longer, so it is more specific
    /blog/my-post/
    Allowed: no rule matches

    admin-ajax.php stays open because some themes and plugins use it to load content on public pages. The sitemap line helps crawlers find your pages. These results are what the tester above reports for this file.

    Practical guidance

    What robots.txt can and cannot do

    robots.txt is a set of crawling instructions for well-behaved bots. It is useful for keeping crawlers away from areas that offer no value in search, such as admin screens, internal search results or endless filtered URLs, so they spend their time on your important pages.

    Common mistakes to avoid

    • Blocking the whole site by accident, often by copying a staging site's Disallow: / to the live site.
    • Using robots.txt to hide pages from search. A blocked page can still be indexed if it is linked from elsewhere, and Google cannot see a noindex tag on a page it is not allowed to crawl.
    • Blocking CSS, JavaScript or images that search engines need to render your pages.
    • Listing secret paths. The file is public; listing a private folder tells everyone it exists.
    • Forgetting that paths are case-sensitive: /Private/ and /private/ are different.

    After uploading, Google usually picks up changes within a day, as it caches robots.txt for up to 24 hours. You can check the file Google has fetched in the robots.txt report in Google Search Console. If you are not sure why pages are missing from Google, our guide on why pages are not indexed covers the most common causes.

    What this tool does not do

    • It does not fetch or edit the robots.txt file on your website.
    • It cannot guarantee how every crawler interprets the file; behaviour varies, and bad bots ignore it.
    • It does not remove pages from search results. Use noindex or Search Console's removals tool for that.
    Frequently asked questions

    Robots.txt: common questions

    Not reliably. robots.txt controls crawling, not indexing. A blocked page can still appear in search results, usually without a description, if other pages link to it. To keep a page out of search results, allow it to be crawled and add a noindex meta tag or X-Robots-Tag header, or protect it with a password.

    No. The file is public, so anyone can read it, and it can reveal the paths you list. Well-behaved crawlers follow it voluntarily; others can ignore it. Protect private areas with a login, not robots.txt.

    At the root of your domain, so it is available at an address such as https://www.example.com/robots.txt. It only applies to that exact host and protocol, so each subdomain needs its own file.

    Google uses the most specific rule, meaning the one with the longest matching path. If two matching rules are equally specific, Google uses the Allow rule. Other crawlers may handle conflicts differently, so avoid ambiguous rules where you can. The tester on this page follows Google's approach.

    No. A crawler follows only the most specific group that names it. If you add a group for Googlebot, Googlebot ignores the rules under "User-agent: *", so repeat any shared rules in its group.

    Generally no. Search engines need CSS and JavaScript to render pages and understand how they look on mobile. Blocking them can affect how your pages are understood.

    Google does not support the Crawl-delay rule, and other search engines treat it differently. Google adjusts its crawl rate automatically based on how your server responds.

    Blocking Google-Extended does not affect how your site appears in Google Search, according to Google. Blocking Googlebot itself would. Note that following robots.txt is voluntary, and blocking a crawler does not remove content it has already collected.

    No. The file is generated in your browser and nothing you enter is sent to or saved on our server. Like the rest of our website, this page uses Google Tag Manager and Microsoft Clarity for usage analytics; the tool area is masked from session recordings and the events we record are actions such as "downloaded", not your rules.

    Technical SEO

    Not sure what your site should block?

    We review robots.txt, sitemaps and indexing as part of technical SEO, and fix crawl problems that keep pages out of Google. Ask us for a free website and SEO review.

    Our Location & Area We Serve

    We're based in Ghaziabad and work in person or remotely with businesses across Delhi NCR — including Ghaziabad, Noida, Delhi, and Gurgaon. Since website development and SEO work is fully remote-deliverable, we also take on clients elsewhere in India and internationally.