How do I block one folder but allow the rest of the site?

Use the Block /admin preset, or keep an Allow rule for / and add a Disallow rule for the folder, such as /private/. Paths must start with / or *. Both lines are written under the same User-agent line.

What does an empty Disallow line do?

An empty Disallow allows everything for that user-agent, so the generator keeps it empty instead of turning it into /, which would block the whole site. A note appears under the row. An empty Allow is flagged because it needs a path.

Can I set different rules for one crawler?

Yes. Add a separate user-agent block for that crawler and give it its own Allow and Disallow rules. If you leave a user-agent blank it is flagged and written as *, so the file never contains a broken User-agent line.

Will robots.txt keep a page out of Google?

Not reliably. Disallow asks crawlers not to fetch a URL, but it does not remove the URL from results. To keep a page out of an index, use a noindex robots meta tag, which the meta tag generator can write.

Can I import or test my existing robots.txt here?

No. The generator builds a new file from the form and does not read an existing one or try URLs against it. What you type stays in your browser while you edit.

Robots.txt Generator

Generate a robots.txt file with a form-based builder.

Loading…

About Robots.txt Generator

A robots.txt file sits at the root of a domain and tells crawlers which paths they may request. Build one from a form instead of remembering the syntax: each block starts with a user-agent, holds as many Allow and Disallow rules as you need, and the file updates as you type. Three presets give you a starting point, and warnings catch blank agents, paths that do not start with a slash and bad sitemap addresses before you copy the result.

Features

Allow and Disallow rules in any number of user-agent blocks
Presets: Allow all, Block all and Block /admin
Optional Sitemap line, checked for a public http or https address
Warnings for blank agents, paths that do not start with a slash and paths with spaces
Identical rules inside a block are written once

Limitations

Only User-agent, Allow, Disallow and Sitemap lines are produced; there is no Crawl-delay and no noindex directive.
A robots.txt file is a request that well-behaved crawlers follow, not access control, and it does not remove a page from search results.
The tool does not read an existing robots.txt or test your rules against real URLs.
Downloading robots.txt needs a free account; copying the text does not.

How it works

Start from a preset (Allow all, Block all or Block /admin), or edit the default block, which allows everything for every crawler.
Set the user-agent for the block: * for all crawlers, or the name of a single crawler.
Choose Allow or Disallow for each rule, type its path, and use + Add rule when a block needs more.
Use + Add user-agent block to give another crawler its own set of rules.
Enter a sitemap URL if you want a Sitemap line at the end of the file.
Copy the file, or download it as robots.txt and upload it to the root of your site.

Example input

User-agent: *
Allow: /
Disallow: /admin

Sitemap: https://example.com/sitemap.xml

Frequently asked questions

: Use the Block /admin preset, or keep an Allow rule for / and add a Disallow rule for the folder, such as /private/. Paths must start with / or *. Both lines are written under the same User-agent line.
: An empty Disallow allows everything for that user-agent, so the generator keeps it empty instead of turning it into /, which would block the whole site. A note appears under the row. An empty Allow is flagged because it needs a path.
: Yes. Add a separate user-agent block for that crawler and give it its own Allow and Disallow rules. If you leave a user-agent blank it is flagged and written as *, so the file never contains a broken User-agent line.
: Not reliably. Disallow asks crawlers not to fetch a URL, but it does not remove the URL from results. To keep a page out of an index, use a noindex robots meta tag, which the meta tag generator can write.
: No. The generator builds a new file from the form and does not read an existing one or try URLs against it. What you type stays in your browser while you edit.