Does your website block AI crawlers?

What that file is doing there

Every website can carry one text file at the same address, called robots.txt. Software that reads websites is expected to check it first and obey it. Search engines have worked this way for decades, and the crawlers behind AI assistants read the same file.

It is written in pairs: who is addressed, then what they may read.

User-agent: GPTBot
Disallow: /

That means "GPTBot, read nothing here". Allow: / means the opposite. A path narrows it: Disallow: /admin/ closes one folder and leaves the rest open. User-agent: * addresses everything not named separately.

The file is a request, not a lock: honoured by the well behaved, ignored by the rest, so nothing private should rely on it. And no file at all is an open door.

The two-minute check

Open your own address with /robots.txt after it. You will see one of four things.

• A blank page or a 404. No file, so nothing is blocked. Nothing to do.
• A short file that says User-agent: * and Allow: /. Everything is welcome. Nothing to do.
• A file with Disallow: lines that name folders (a cart, an admin area, a search page). Normal housekeeping. It does not stop an assistant reading your services page.
• A file that names AI crawlers and refuses them. The one to catch: a run of two-line blocks, each a bot name followed by Disallow: /.

Read the whole file. These blocks are usually at the bottom, often added by a plugin or a hosting default rather than by anybody’s decision. Owners are regularly surprised by their own file.

The names to look for

The crawler names in Droom’s own file, and who runs each:

• GPTBot, OAI-SearchBot, ChatGPT-User from OpenAI
• ClaudeBot, Claude-Web and anthropic-ai from Anthropic
• PerplexityBot and Perplexity-User from Perplexity
• Google-Extended from Google, separate from its ordinary search crawler
• Applebot and Applebot-Extended from Apple
• Bingbot from Microsoft
• CCBot, the crawler behind Common Crawl, a public archive of the web
• meta-externalagent from Meta

The companies change what each name is for, and several run more than one crawler. Treat this as words to search your file for, not a specification. If one sits above a Disallow: /, that is your answer.

What Droom’s file says, and why

Here is ours, at droom.ca/robots.txt. The robots.txt on droom.ca names fourteen AI crawlers and allows every one of them, with the reason written in as a comment so nobody quietly reverses it:

“AI crawlers: deliberately allowed. Blocking these means zero visibility in ChatGPT, Perplexity, Claude and Google AI Overviews. For a local service business that answers questions (‘what does a small business website cost in Vancouver’), being quotable in an AI answer is worth more than the crawl it costs.”

Two things are closed off, for the same reason, and neither is caution about AI. One is the page describing our own crawler. The other is /for/, where a rebuilt concept of somebody else’s homepage lives while we show it to them privately. That one also carries an instruction in the server’s response: the file is the polite half of a refusal, the header the enforced half. Everything else is open to everything.

For a small local business the trade is plain: an assistant can quote the pages that answer a customer’s question, and never mentions a page it cannot read. The same thinking shapes every site built here: structured data and an llms.txt file, so an assistant can read who the business is and what it does.

You found a block. Now what?

It is a small job for whoever runs your site: delete or rewrite a few lines. Some platforms have a settings switch, often worded as blocking AI training or AI bots. Some let you edit the file. Some will not let you touch it at all.

If nobody you can reach can change one text file on your own website, robots.txt is not the real problem. Who owns your website files covers what to ask for. If the builder has gone quiet, the rescue checklist starts further back, with who owns your domain.

One caution: do not open something closed on purpose, or widen a narrow rule. Removing a single AI block is safe. Deleting the whole file because it looked untidy is how a checkout page ends up in search results.

When blocking is the right call

If your writing, photography or research is the product and people pay to read it, letting it be absorbed for free is a decision to make on purpose, not by default.

A plumber, a dentist, a bakery or a law firm is the opposite. Their pages exist to be found and repeated by anyone who might send a customer. Keeping an assistant out protects no revenue; it only gives up mentions you will never see.