llms.txt, robots.txt and AI crawlers: what each file actually controls

Start with the outcome, then choose the control

Teams often ask whether they should add an llms.txt file when the more important question is what they are trying to control. Search discovery, search snippets, model training, site navigation and entity description are different outcomes with different technical layers.

robots.txt tells compliant crawlers which paths they may request. Page-level robots directives such as noindex and nosnippet influence indexing or presentation for systems that support them. A sitemap supplies canonical discovery hints. Structured data describes facts already visible on the page. llms.txt proposes a concise reading guide for language models.

No single layer grants a recommendation. The practical job is to align access policy, canonical URLs, visible evidence and measurement so a useful page can be found and evaluated.

• Use robots.txt for crawler access policy.
• Use page directives for indexing and snippet controls where supported.
• Use XML sitemaps to surface canonical, indexable URLs.
• Use structured data only when it matches the visible page.
• Use llms.txt as an optional orientation layer, not a permission system.

OpenAI gives search and training separate user agents

OpenAI's publisher guidance says sites that want their content considered for summaries and snippets in ChatGPT Search should allow OAI-SearchBot. It separately identifies GPTBot as the crawler publishers can disallow for pages they want excluded from potential model training.

That distinction lets a company support search discovery without automatically treating every AI crawler as the same business decision. The policy should be documented by user agent, outcome, owner and review date.

A robots rule is only one checkpoint. Firewalls, CDN bot controls, authentication and JavaScript rendering can still prevent retrieval. Test the production response and monitor server logs when they are available.

Google Search access and Google-Extended are not the same switch

Google states that the existing SEO practices for Search remain relevant to AI Overviews and AI Mode. Access for those Search experiences is managed through Googlebot, while snippet controls such as nosnippet, data-nosnippet and max-snippet can limit what is shown.

Google-Extended is a separate product token that lets publishers manage whether crawled content may be used to help improve Gemini Apps and grounding in Gemini APIs. Google says it does not affect inclusion or ranking in Google Search.

The operational lesson is to avoid copying a crawler blocklist without mapping each token to a real goal. Search access, generative training and API grounding can have different policies.

What llms.txt can and cannot do

The llms.txt proposal recommends a markdown file at the site root with a short summary and curated links to important resources. Its intent is to give language models a clean orientation layer when a website is difficult to process or too large to fit into context.

It remains a proposal, not a robots standard or a guaranteed input to major search-ranking systems. The file cannot override robots rules, make a blocked page indexable, validate a claim or compel an assistant to cite the site.

It can still be useful as a low-cost, human-readable map when it is accurate, concise and maintained. THIRD P_RTY now publishes one for that reason, while keeping the sitemap, article schema and crawl policy as separate layers.

Build the minimum viable discovery stack

Start with a clean robots policy, an XML sitemap containing canonical public URLs, descriptive internal links and pages that return healthy status codes. Add accurate Article, Organization or other supported schema only when the visible content supports it.

Then publish an llms.txt guide if it helps summarize the site. Keep it small enough to audit. Link to the service, methodology, contact path, articles and other authoritative pages rather than duplicating the entire website.

For THIRD P_RTY, the stack includes robots.txt, sitemap.xml, llms.txt, canonical metadata, Article JSON-LD, breadcrumb markup and contextual source links. Each layer has a specific job.

Measure access, eligibility and visibility separately

Access means the crawler can retrieve the URL. Eligibility means the page can be indexed or considered under the platform's rules. Visibility means the page or brand actually appears for a relevant query. Those are three different measurements.

A useful monthly review checks robots responses, sitemap health, index status, crawler requests, AI referral traffic, brand mentions and cited pages. Keep the observations separate so a crawl fix is not misreported as a visibility win.

When policy changes, record the date and retest. Crawlers may take time to revisit a site, and answer systems can vary even when the underlying page does not change.

Use a governance record instead of a copied snippet

For every crawler decision, record the user agent, desired outcome, rule, business rationale, approver, implementation owner and next review date. This turns robots.txt from an inherited artifact into an accountable policy.

Review the record after platform migrations, CDN changes, redesigns and legal-policy updates. A correct rule in source control can still be undermined by a cached edge response or a bot-management product.

The downloadable playbook includes the decision register, test protocol, release checklist and evidence log needed to maintain that policy over time.