Last updated:
Quick answer
Two things matter most: your robots.txt must let the AI crawlers you want in, and your key pages should be plain readable text. The llms.txt file is a proposal: an H1 title, a short summary, and lists of links to your best pages.
Free PDF
Get this setup as a PDF
Confirm your email and we'll open a print-ready copy of this setup you can save as a PDF. You also get the free starter pack and one setup a week. Unsubscribe anytime.
Already a subscriber? Unlock it on this device
At a glance
All outcome figures are estimates from cited sources, not independent verification by us.
- What it is allowed to do
- Drafts for you
Writes drafts (emails, posts, plans, replies). You review and send. Nothing goes out without you.
- Time saved (estimate)
- About 45 minutes for a small site.
- Difficulty
- Intermediate
- Tools needed
- ChatGPT, Claude
- Typical cost
- Free tiers can draft the files. Hosting and CMS limits depend on your site. Prices: Not stated in sources.
Reviewed Oct 31, 2026
Basis (estimate): Estimate only: based on the steps above, not timed.
What does this setup do?
Check which AI crawlers your site allows, add an llms.txt file, and make sure your key pages read as plain text, then re-check with the free AI-ready check.
Who is it for?
Owners of a small business website who want AI assistants to be able to read it. You need to be able to edit robots.txt or ask whoever manages the site. Do not block a crawler you want without reading what it does.
How do you set it up?
Tools
- The free AI-ready check at /tools/ai-ready-check/
- Your site's robots.txt, at yoursite.com/robots.txt
- ChatGPT or Claude to draft the llms.txt file
- Whoever can upload files to your site
Steps
- Run your address through the AI-ready check at /tools/ai-ready-check/ and keep the result as a before copy.
- Open yoursite.com/robots.txt. Google says robots.txt manages crawler traffic and is not a way to keep a page out of Google, so read it as traffic rules.
- Decide each crawler on purpose. OpenAI says OAI-SearchBot is for search results in ChatGPT, GPTBot is for training, and ChatGPT-User is for user actions, and the settings are independent. Anthropic lists ClaudeBot for training, Claude-User for user requests and Claude-SearchBot for search, and says blocking the last two may reduce your visibility in answers.
- Ask the assistant to draft an llms.txt file with the prompt below. The proposal says the only required part is an H1 with the site name, followed by an optional summary and lists of links with short notes.
- Upload it at yoursite.com/llms.txt. Check that every link in it opens and goes to a page with readable text.
- The proposal also suggests a clean markdown version of key pages at the same address with .md added. Skip it if your site builder cannot do that.
- Run the AI-ready check again and compare it with the before copy.
Example output
Illustrative. Your results will differ.
- # Rivera Plumbing
- > Family plumber serving Burlington, Vermont. Repairs, installs and emergency calls.
- ## Pages
- [Services](https://example.com/services): what we fix and install
Where to paste this prompt
ChatGPT → Project instructions, or ChatGPT Tasks for scheduled reminders (once a day on Free)
Copy-paste prompt
Adapt bracketed fields to your situation.
Draft an llms.txt file for my business website. Use only the facts I give you.
Business name: [NAME]
One-sentence summary: [WHAT YOU DO, WHERE, FOR WHOM]
My 5 to 8 most useful pages: [PAGE NAME - URL - what it covers in a few words]
Format:
- An H1 with the business name
- A blockquote with the summary
- One H2 section called Pages with a markdown list: [name](url): short note
Rules:
- Do not invent services, prices, awards or claims.
- Only list URLs I gave you.
- No marketing language. Plain sentences.
- Show me the file first. Do not publish anything.Newsletter
Get the free starter pack
Free starter pack with five beginner copy-paste prompts after you confirm, plus one researched setup in your inbox each week. Unsubscribe anytime.
What usually goes wrong?
Where agents tend to stall on this job. More patterns in Where AI gets stuck.
- Blocking all bots in robots.txt by accident. Test after any edit.
- Blocking a search crawler and then wondering why AI answers skip you. Anthropic says disabling its search and user bots may reduce visibility.
- Linking pages that need a login or are mostly images. Use pages with plain text.
Which chat tools work with this prompt?
Paste the prompt into any of these assistants (free or paid tiers).
- ChatGPT
- Claude
Common questions
- What is llms.txt?
- A proposal for a markdown file at the root of a site with a title, a short summary and lists of links to the pages an AI should read. Its page calls it a proposal.
- Should I block GPTBot?
- OpenAI says GPTBot is used for training and OAI-SearchBot is for search results in ChatGPT, and each setting is independent. Decide each one separately.
- Does robots.txt keep a page out of Google?
- Google says no. It says robots.txt manages crawler traffic and is not a mechanism for keeping a page out of Google.
Where should you go next on this site?
What tips make it smoother?
- The llms.txt proposal is a proposal. Whether a given AI product reads it: Not stated in sources.
- Anthropic and OpenAI each list their crawlers on their own help pages. Check them before you edit robots.txt.
- Keep the file short. A few good pages beat a long list.
Where do the claims come from?
Summaries use public sources linked below, not independent verification. How setups are picked.
- Read Jeremy Howard’s write-up (llmstxt.org)
Jeremy Howard: The /llms.txt file v2: proposal, required H1, blockquote summary, H2 file lists, .md versions of pages (read 2026-10-31 through a page fetch tool, not a direct browser open)
- Read OpenAI’s write-up (developers.openai.com)
OpenAI: Overview of OpenAI crawlers: OAI-SearchBot, GPTBot, ChatGPT-User, each setting independent (read 2026-10-31 through a page fetch tool, not a direct browser open)
- Read Anthropic’s write-up (support.claude.com)
Anthropic: Anthropic crawlers: ClaudeBot, Claude-User, Claude-SearchBot and what blocking each does (read 2026-10-31 through a page fetch tool, not a direct browser open)
Related setups
- Add business schema to your site
Write LocalBusiness JSON-LD from your real name, address and hours, add it to your site, and test it with Google's Rich Results Test.
- Optimize your Google Business Profile to show up locally
Work through a one-time optimization pass on your free Google listing so it is complete, accurate, and review-active, with the AI drafting the description and review replies.
- Build a lead-gen site with an AI bot
Use an AI coding or copy assistant to draft a simple landing page and contact form for a local service business, then you approve hosting spend, publish, and wire leads to email or a sheet.