Should You Block AI Crawlers?

For most businesses, no. For publishers whose revenue depends on pageviews, it is a genuine question with defensible answers on both sides.

The reason the answer differs so sharply is that blocking does two things at once, and which one dominates depends on how you make money.

What blocking actually does

Adding AI crawlers to your robots.txt stops them fetching your pages. Two consequences follow, and they are inseparable.

Your content stops being used in generated answers — which is what people want when they block.

And your brand stops appearing in those answers, including the ones that would have recommended you. That is the part that gets underweighted.

You cannot have the second without the first. There is no setting for "mention us but do not summarise us".

Not every AI crawler does the same job

"AI crawler" gets treated as one category with one on/off switch, and that framing hides the actual decision. In practice these bots split into a few distinct jobs: some fetch pages to train a model, some fetch pages live to ground a specific answer to a specific user's question, and some are exploratory bots collecting or evaluating public content for reasons that have nothing to do with either.

Those are three different trade-offs, not one. Blocking a training crawler stops your content becoming part of a future model. Blocking a retrieval crawler stops you being cited in an answer someone is asking right now. Treating both the same way, with the same blanket rule, means accepting the downside of the second to get the upside of the first — when a more selective robots.txt can usually get you the training protection without losing the citation.

A quick reference

The publisher case for blocking

If your revenue is advertising against pageviews, an answer that satisfies the reader destroys the transaction. Your article gets read, your server gets no request, your advertiser gets no impression.

For a large publisher, that is not an abstraction. It is measurable revenue loss caused by content you paid to produce being consumed somewhere you cannot monetise.

Blocking is a coherent response to that, and several major publishers have chosen it. Some have gone further and licensed access instead, which is the same logic with an invoice attached.

Why it is usually the wrong call for everyone else

If you sell software, services or products, your website is not the product. It is how people find and evaluate the product.

In that shape, an AI answer that names you is closer to a recommendation than a theft. Someone asked which tool to use and a system said yours. No click was recorded, but the association happened at the exact moment the decision was forming.

Blocking removes you from that moment. Your competitors, who did not block, get named instead — and their absence from your analytics looks identical to your absence from theirs.

There is also an asymmetry worth naming. The traffic you lose to summaries is visible and measurable. The recommendations you forgo by blocking are invisible and unmeasurable. It is easy to over-correct against the loss you can see.

The middle positions

The decision is not binary, and the middle ground is where most sensible answers live.

Block selectively by crawler. Different platforms use different user agents for different purposes — some for training, some for live retrieval when a user asks a question. Blocking training while allowing retrieval keeps you present in answers without contributing to the next model. Whether providers honour that distinction reliably is a fair question, but it is available.

Block selectively by section. Allow crawling of your marketing pages, product pages and glossary, where being quoted helps. Block your premium archive or paywalled reporting, where it does not. This maps the decision to how each part of the site actually earns.

Do nothing, and measure. Underrated. Establish how often you are currently named, then decide with numbers instead of instinct.

Why this is easier to execute on Webflow than it sounds

Webflow gives site owners direct control over robots.txt without needing a developer or server access. The platform auto-generates a robots.txt file for every site, and it can be customised from Site Settings, under the SEO or Indexing section — user-agent by user-agent, path by path.

How to actually edit it

In Webflow, open Site Settings, go to the SEO or Indexing area, and update the robots.txt rules directly. Changes only take effect once the site is republished, so a rule saved but not published is not live yet. After publishing, check the real file at yourdomain.com/robots.txt — that is the only way to confirm what policy is actually active, rather than trusting what the settings panel implies.

Webflow

1. Open Site Settings for your project.
2. Go to SEO or SEO > Indexing.
3. Paste your robots.txt rules into the dedicated field.
4. Save the changes.
5. Republish the site.
6. Check yourdomain.com/robots.txt to confirm the file is live.

Framer

1. Open your Framer project dashboard.
2. Go to Static Files or the file upload area for the project.
3. Upload a file named exactly robots.txt.
4. Republish the site.
5. Check yourdomain.com/robots.txt to confirm the file is live.

WordPress

1. Check whether your site already has a robots.txt file at yourdomain.com/robots.txt.
2. If you use an SEO plugin, edit robots.txt from the plugin settings.
3. If you have server or file access, add or edit the file in the site root.
4. Save the changes.
5. Recheck yourdomain.com/robots.txt in the browser.

Wix

1. Open the site's SEO Dashboard.
2. Find the Robots.txt Editor.
3. Edit the file content.
4. Save the changes.
5. Verify the result at yourdomain.com/robots.txt.

Squarespace

1. Open Settings for the site.
2. Go to the crawling or indexing controls.
3. Use Squarespace's built-in SEO/indexing options rather than direct file editing.
4. Apply the change.
5. Verify the crawling behavior from the site settings, since manual robots.txt editing is limited.

Shopify

1. Open Online Store > Themes.
2. Click Edit code on the live theme.
3. Create or edit robots.txt.liquid.
4. Paste your rules and save.
5. Confirm the live file at yourdomain.com/robots.txt.

Staging is not production

Webflow's staging subdomains often discourage indexing by default, which is usually correct behaviour for a staging environment — but it tells you nothing about what the production domain is doing. Do not assume the two match. Check the live robots.txt on the real domain before making a strategic decision, because a policy that looks intentional on staging may just be Webflow's default, unrelated to what you actually want live.

A practical framework, section by section

Most Webflow sites do not need an all-or-nothing answer. A workable default looks like this:

Allow crawlers on service pages, blog posts, FAQs, comparison pages, and location pages — the content built to answer real questions and support discovery.

Restrict access to drafts, client-only areas, member hubs, gated documents, thank-you pages, and anything staging-like — paths that carry cost or exposure without a corresponding visibility benefit.

Review per crawler rather than assuming every bot deserves the same treatment, using the distinction above between training, retrieval, and exploratory access.

Revisit the policy every few months, since AI search products and referral patterns are still changing quickly enough that a rule set six months ago may no longer match reality.

What this looks like in practice

A local service business publishing city pages, FAQs, and service explainers gains far more from being findable in AI-generated answers than it risks by being crawled — the sensible move is to allow the crawlers that matter on that content and block only the thin or irrelevant paths, like internal thank-you pages.

A content-led SaaS company generally wants product education, use-case pages, and comparison content open to the crawlers that feed AI-driven discovery, while keeping experiment pages, internal campaign URLs, and private onboarding resources closed off.

A publisher or research brand with content that has real standalone value on its own may take the opposite default — staying open for classic Search visibility while restricting non-essential AI training bots or premium content paths specifically, rather than blocking indiscriminately.

Common mistakes

Copying a robots.txt template without reading it. Old templates, inherited agency defaults, or outdated SEO snippets often block more than intended, especially once AI-specific user agents get added later with no clear strategy behind them.

Confusing staging behaviour with production policy. Webflow's staging environment may intentionally discourage crawling — that says nothing about what should happen on the live domain, and the two need to be checked separately.

Treating robots.txt as security. It is an instruction file compliant bots choose to follow, not an access control system. If something genuinely needs to stay private, robots.txt is not the tool — use real authentication or remove it from public reach.

Framing the decision as ideology instead of business strategy. This is a commercial question first. It should be decided by what the business earns from visibility versus what it protects by restricting it, then implemented cleanly — not settled by a general feeling about AI.

Before you decide, check what you are already doing

Plenty of sites already block AI crawlers without anyone having chosen to. It arrives in a template, a plugin default, or a security configuration nobody revisited.

Open your robots.txt and read it. If you find AI crawler entries you did not put there, you have been running the experiment already — and any conclusion you have drawn about your AI visibility was drawn from a rigged test.

The question that settles it

Ask what a mention is worth to you when it produces no click.

If the answer is "nothing, we sell impressions", blocking is defensible and you should model the cost properly.

If the answer is "quite a lot, because that is how buyers hear about us", then blocking trades a visible loss for an invisible one that is probably larger.

Most businesses are in the second group and answer as though they were in the first, because one number appears in a dashboard and the other never will.