> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usecliro.com/llms.txt
> Use this file to discover all available pages before exploring further.

# robots.txt

> Generate and publish a robots.txt that controls which crawlers and AI agents can access your site.

`robots.txt` is a standard file that tells crawlers and bots which sections of your website they're allowed to crawl and index — including AI crawlers.

## Why configure robots.txt

* Control which pages search engines and AI crawlers can index.
* Protect private or administrative sections from external crawling.
* Optimize your crawl budget to improve your site's SEO.

<Note>
  For AI visibility, make sure you're **not accidentally blocking** the AI crawlers you want to reach you. A misconfigured `robots.txt` can keep your brand out of AI answers entirely.
</Note>

## Create your robots.txt

<Steps>
  <Step title="Generate with AI">
    Open **Actions → robots.txt** and click **Generate with AI** for a suggested file tailored to your site.
  </Step>

  <Step title="Edit">
    Adjust the rules in the editor. The file stays **Suggested** until confirmed.
  </Step>

  <Step title="Confirm">
    Click **Confirm** to lock the content and enable downloading.
  </Step>

  <Step title="Download & publish">
    Upload the file to your site's root so it's reachable at `https://yourdomain.com/robots.txt`.
  </Step>

  <Step title="Verify publication">
    Use **Verify publication** to confirm the live file matches your confirmed content.
  </Step>
</Steps>

<Card title="Check AI crawler access" icon="search-check" href="/en/actions/site-audit">
  The Site Check verifies your robots.txt and crawl rules for AI agents.
</Card>