Check which AI crawlers your site allows, add an llms.txt file, and make sure your key pages read as plain text, then re-check with the free AI-ready check.
Who it is for
Owners of a small business website who want AI assistants to be able to read it. You need to be able to edit robots.txt or ask whoever manages the site. Do not block a crawler you want without reading what it does.
Tools
The free AI-ready check at /tools/ai-ready-check/
Your site's robots.txt, at yoursite.com/robots.txt
ChatGPT or Claude to draft the llms.txt file
Whoever can upload files to your site
Steps
Run your address through the AI-ready check at /tools/ai-ready-check/ and keep the result as a before copy.
Open yoursite.com/robots.txt. Google says robots.txt manages crawler traffic and is not a way to keep a page out of Google, so read it as traffic rules.
Decide each crawler on purpose. OpenAI says OAI-SearchBot is for search results in ChatGPT, GPTBot is for training, and ChatGPT-User is for user actions, and the settings are independent. Anthropic lists ClaudeBot for training, Claude-User for user requests and Claude-SearchBot for search, and says blocking the last two may reduce your visibility in answers.
Ask the assistant to draft an llms.txt file with the prompt below. The proposal says the only required part is an H1 with the site name, followed by an optional summary and lists of links with short notes.
Upload it at yoursite.com/llms.txt. Check that every link in it opens and goes to a page with readable text.
The proposal also suggests a clean markdown version of key pages at the same address with .md added. Skip it if your site builder cannot do that.
Run the AI-ready check again and compare it with the before copy.
The prompt
Draft an llms.txt file for my business website. Use only the facts I give you.
Business name: [NAME]
One-sentence summary: [WHAT YOU DO, WHERE, FOR WHOM]
My 5 to 8 most useful pages: [PAGE NAME - URL - what it covers in a few words]
Format:
- An H1 with the business name
- A blockquote with the summary
- One H2 section called Pages with a markdown list: [name](url): short note
Rules:
- Do not invent services, prices, awards or claims.
- Only list URLs I gave you.
- No marketing language. Plain sentences.
- Show me the file first. Do not publish anything.
Tips
The llms.txt proposal is a proposal. Whether a given AI product reads it: Not stated in sources.
Anthropic and OpenAI each list their crawlers on their own help pages. Check them before you edit robots.txt.
Keep the file short. A few good pages beat a long list.