# Robots.txt & AI Crawler Generator

Build a readable robots.txt policy for search and AI crawlers, exclude sensitive paths from compliant crawlers, and add your sitemap.

- Category: Technical SEO
- Result: A robots.txt file ready to review and publish
- Price: Free
- Account required: No

## What it includes

- Search and named AI crawler controls
- Allow and disallow path rules
- Sitemap directive and access summary

## Robots.txt controls crawling, not access

A disallow rule asks compliant crawlers not to fetch a path. It does not make the path private. Protect dashboards, customer data, staging sites, and internal files with authentication and authorization.

## Review broad rules before publishing

A single slash or wildcard can unintentionally block an entire site or important assets. Test the final policy against the public pages, JavaScript, images, and API paths your product page needs to render and be discovered.

## Frequently asked questions

### Where does robots.txt go?

Publish it at the root of the hostname it controls, such as https://example.com/robots.txt. Rules on one subdomain do not automatically control another.

### Does disallow remove a page from search results?

Not necessarily. A blocked URL can still be known from links. Use appropriate noindex controls on crawlable pages or restrict access when removal or privacy is required.

### Do all AI services use the same crawler name?

No. Services publish their own user-agent names and policies, which can change. Verify the current documentation for each service before relying on a rule.

Canonical: https://shipyardhq.dev/tools/robots-txt-ai-crawler-generator