Skip to main content
robots.txt is a standard file that tells crawlers and bots which sections of your website they’re allowed to crawl and index — including AI crawlers.

​
Why configure robots.txt

  • Control which pages search engines and AI crawlers can index.
  • Protect private or administrative sections from external crawling.
  • Optimize your crawl budget to improve your site’s SEO.
For AI visibility, make sure you’re not accidentally blocking the AI crawlers you want to reach you. A misconfigured robots.txt can keep your brand out of AI answers entirely.

​
Create your robots.txt

1

Generate with AI

Open Actions → robots.txt and click Generate with AI for a suggested file tailored to your site.
2

Edit

Adjust the rules in the editor. The file stays Suggested until confirmed.
3

Confirm

Click Confirm to lock the content and enable downloading.
4

Download & publish

Upload the file to your site’s root so it’s reachable at https://yourdomain.com/robots.txt.
5

Verify publication

Use Verify publication to confirm the live file matches your confirmed content.

Check AI crawler access

The Site Check verifies your robots.txt and crawl rules for AI agents.