Robots.txt Generator & Tester
Free SEO Tool

Free Robots.txt Generator & Tester

Generate a clean robots.txt file, test crawler access, validate rules and download your file instantly.

100% Free No Signup No Database No Paid API

Quick Templates

Allow Everything
Block Everything
Basic Website
WordPress
E-commerce
SEO Friendly

Templates are starting points — review paths for your own site before publishing.

AI Crawler Controls

Robots.txt provides crawler instructions but cannot technically prevent non-compliant bots from accessing content.

Test Your Robots.txt

Check whether a URL is allowed or blocked according to a website's robots.txt rules.

This tester provides a practical robots.txt rule check and may differ from individual search engine crawler implementations.

Robots.txt Validation

Generated robots.txt

Live Preview 
User-agent: *
Allow: /

What is a Robots.txt File?

A robots.txt file is a plain-text file placed at the root of a website that gives instructions to search engine crawlers and other automated bots about which parts of the site they may or may not request. It follows the Robots Exclusion Protocol, a long-standing web convention that well-behaved crawlers choose to respect.

Websites use robots.txt to keep crawlers away from areas that don't need to be indexed — admin panels, staging areas, internal search results, or duplicate content — and to conserve crawl budget for the pages that matter most.

Location: robots.txt must live at the root of your domain, for example https://example.com/robots.txt. Placing it in a subfolder will not work.

Basic syntax: a robots.txt file is made of one or more groups, each starting with a User-agent line followed by Allow and/or Disallow rules that apply to that user-agent. A separate Sitemap line can point crawlers to your XML sitemap.

Allow vs Disallow: Disallow tells a crawler not to request a path; Allow carves out an exception within a broader disallowed path. When rules overlap, the most specific (longest) path generally takes precedence.

Sitemap declaration: adding a Sitemap: line is optional but helps crawlers discover your sitemap without needing it submitted elsewhere.

Common mistakes: blocking your entire site with Disallow: / by accident, placing the file in the wrong location, using robots.txt to try to hide sensitive content (it doesn't provide access control), and forgetting that some crawlers simply ignore the file.

Robots.txt Example

User-agent: *
Allow: /

Disallow: /admin/
Disallow: /login/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Frequently Asked Questions

It's a plain-text file at your site's root that tells crawlers which paths they may or may not request.

At the root of your domain, e.g. https://example.com/robots.txt. It won't be recognized in a subfolder.

No. If it's missing, crawlers generally assume they can crawl everything. Adding one gives you explicit control.

No. It only asks crawlers not to request a path — it doesn't restrict human visitors or provide access control.

You can disallow Googlebot from specific paths, but this affects crawling, not necessarily indexing — a disallowed URL can still appear in search results if it's linked elsewhere.

It instructs the named crawler not to request any page on the site — effectively blocking the whole site from that crawler.

It explicitly permits crawling of the entire site, or carves out an exception inside a broader Disallow rule.

It's a common and helpful practice — it gives crawlers an easy way to discover your sitemap.

You can add Disallow rules for named AI crawler user-agents such as GPTBot, Google-Extended, or ClaudeBot. Compliant bots will respect this, but robots.txt cannot technically stop non-compliant ones.

Yes — it's completely free to use, with no signup, no database, and no paid API required.