robots.txt
A text file at a site's root that tells search engine and AI crawlers which pages or bots are allowed to access it.
Detailed Explanation
robots.txt is a plain-text file placed at a website's root directory that gives crawlers instructions about which parts of the site they may or may not access. It can allow or block specific bots by user-agent name, including AI crawlers like GPTBot or Google-Extended, separately from traditional search bots like Googlebot.
Why It Matters
robots.txt is the first control point for AI visibility — blocking the wrong bot here silently removes a brand from AI training data and live retrieval.
Example
Synonims
Robots Exclusion Protocol
Related FAQs
We're updating this category. Come back soon.
