Robots.txt Generator & Tester

A robots.txt file is four lines of text that can quietly remove a site from search results, and its rules do not work the way most people assume. Our free robots txt generator builds the file — and, more usefully, tests it.

The part that catches people out is precedence. Rules are not applied in order: the longest matching path wins wherever it sits in the file, and when an Allow and a Disallow match with equal length, the Allow wins. That means a Disallow near the top can be overridden by an Allow near the bottom, and vice versa. The built-in tester takes a URL path and a crawler and tells you the verdict, which group applied, and precisely which rule decided it — the check almost every other generator leaves out.

Presets cover the usual starting points: allow everything, block everything, a sensible default for a typical site, and a group blocking the major AI crawlers. Wildcards and end anchors are supported, so patterns like /*.pdf$ behave the way Google documents.

One thing to be clear about, because it causes real damage: robots.txt controls crawling, not indexing. A blocked URL can still show up in results if other pages link to it, just without a description. To keep something out of the index entirely you must allow crawling and use a noindex tag — blocking it in robots.txt actively prevents Google from seeing that instruction. Pair this with the Meta Tag Generator and the Schema Markup Generator for the rest of your technical setup.