The Protocol

8 actionable rules to make your site correctly understood and used by AI, step by step.

Publish llms.txt and llms-full.txt

Give LLMs a machine-readable index and full text of your site, so AI reads it right the first time.

llms.txt is a convention: a plain-text file in your site root that lists a short description and section links in Markdown style, so large language models can index your site quickly before fetching pages. llms-full.txt provides the full text of your site.

Goal

Let LLMs and AI agents understand your site structure accurately, with zero friction, and locate relevant content fast.

Why it matters

Crawling an entire site is slow and expensive for LLMs; llms.txt gives a low-cost, high-accuracy entry point.
Proactively providing an index means proactively being “seen” — an advocacy approach for the AI era.
It is an open, verifiable convention any site can adopt immediately.

Checklist

Publish /llms.txt in the site root; declare the spec version on the first line, then list the H1 site description and sections.
Provide /llms-full.txt with fuller body content (optional but recommended).
Use absolute URLs and precise, concise descriptions — avoid marketing fluff.
Keep it in sync with the actual site content and update it regularly.

Acceptance

Accessing /llms.txt returns 200 with correct text/plain content.
Content parses into clear sections and links.
Loading it as LLM context yields an accurate site overview.

Correct robots.txt directives

Allow mainstream AI crawlers reasonable access without accidental blocking.

robots.txt is your first “pass” with AI crawlers. A correct robots.txt precisely expresses which content may be crawled — not a blanket ban on everything.

Goal

Let mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) access your public content in a controlled way, while blocking what you do not welcome.

Why it matters

Blocking all crawlers makes your site “invisible” in the AI era, missing retrieval and recommendation.
Incorrect User-agent casing or directives let crawlers misunderstand your intent.
This is a language both sides understand — the clearer, the fairer.

Checklist

List the mainstream AI crawlers you allow, with explicit Allow or Disallow rules.
Set Disallow on specific paths you do not want crawled, rather than blocking the whole site.
Add a comment at the end explaining your policy for humans and AI alike.
Point to sitemap.xml so crawlers discover your map.

Acceptance

Verify each rule matches intent with a robots.txt testing tool.
Allowed AI crawlers can fetch key pages and llms.txt.
Sensitive paths are explicitly blocked.

A complete, locale-aware sitemap.xml

A clear site map so AI and search engines crawl every page.

sitemap.xml gives crawlers a complete list of your site’s pages, and combined with lastmod and multi-language alternates, makes crawling more efficient and complete.

Goal

Ensure every public page is indexed, with clear mappings between language versions.

Why it matters

Without a sitemap, deep pages may go undiscovered for a long time.
Multi-language sites without version mapping may have the wrong language fetched.
lastmod tells crawlers what changed, reducing wasted crawls.

Checklist

Auto-generate sitemap.xml containing all public pages.
Mark lastmod on each URL (content modified date).
Provide xhtml:link rel="alternate" hreflang alternates for multi-language pages.
Declare the sitemap URL in robots.txt.

Acceptance

All listed URLs return 200.
Multi-language mappings are correct.
The sitemap passes validation without errors.

Structured data (JSON-LD / schema.org)

Express products, organizations, FAQs as structured data so AI understands precisely.

Structured data (JSON-LD + schema.org) expresses the human-readable information on a page additionally in a machine-readable way, removing ambiguity.

Goal

Let AI and search engines accurately identify your organization, products, pricing, FAQ, and more.

Why it matters

AI can read “facts” directly from structured data instead of guessing from text.
It boosts accuracy and presence in AI search and conversational recommendations.
It is an open, public standard — low cost, direct benefit.

Checklist

Inject Organization / WebSite JSON-LD on the homepage.
Use Product + Offer (with price, availability, currency) on product pages.
Use FAQPage for FAQ pages or common questions.
Validate JSON-LD syntax regularly with a validator.

Acceptance

Passes Schema.org validation (no syntax errors).
Key fields (name, price, description) are correctly recognized.
Rich results render in supported structured-data testing tools.

Semantic HTML and clear hierarchy

Use correct tags and heading levels so content structure is obvious.

Semantic HTML makes the “skeleton” of content visible: whether something is a heading, a paragraph, or navigation is clear from the structure, not from styling guesses.

Goal

Let AI and any parser reconstruct the information hierarchy of a page from the HTML structure alone.

Why it matters

Correct heading levels (h1>h2>h3) are core signals for AI understanding document structure.
Semantic tags like <main>/<article>/<nav> reduce comprehension noise.
Plain-text extractability means better accessibility and readability regardless of technology.

Checklist

Use exactly one h1 per page, with subheadings progressing by level.
Use semantic regions: <main>, <article>, <nav>, <header>, <footer>.
Avoid embedding key information (price, specs) in images or text-less interactive components.
Avoid heavy client-side JS rendering of core content.

Acceptance

Core content remains readable with JS disabled.
An accessibility/structure checker finds no skipped heading levels.
The page yields complete semantics by plain-text extraction alone.

Correct language declaration and meta tags

Tell AI what language your page uses and provide complete share metadata.

The html lang declaration tells AI the language of your page; meta and Open Graph tags provide standardized description for sharing and retrieval.

Goal

Let AI recognize your page language accurately, and provide a complete title, description, and preview on any platform.

Why it matters

A wrong language declaration makes AI interpret your content with the wrong language model.
Without a meta description, AI can only guess from the body, which is prone to distortion.
Proper social share tags let your content present well on AI platforms and social networks alike.

Checklist

Declare the primary language correctly in <html lang="..."> (e.g. zh-CN, en).
Provide meta description and canonical.
Configure Open Graph (og:title, og:description, og:url, og:image) and Twitter Card.

Acceptance

Page language is recognized correctly and matches the actual content language.
Crawling tools read a complete title, description, and OG tags.
Multi-language pages each declare their correct language.

Content accessibility and plain-text extractability

Keep key information obtainable without images or scripts.

Content accessibility means no key information should live only in images, video, or pure client-side scripts. AI must be able to get the full content through plain-text extraction.

Goal

Ensure core information remains fully readable under no-script, plain-text extraction.

Why it matters

Text inside images needs OCR, which is costly and error-prone.
Heavy reliance on client-side JS leaves many crawlers with an empty shell.
Accessibility is also a responsibility to people with disabilities.
AI and assistive technologies share the same readable path.

Checklist

Present key information (price, specs, contact, terms) as text, not only images.
Provide meaningful alt text for images.
Serve core content server-side (SSR/SSG) rather than requiring JS execution.
Use sufficient contrast and readable font sizes (no effect on AI, beneficial to humans).

Acceptance

Full core content is available with JS disabled.
Plain-text extraction includes all key facts.
Image alt independently describes the image content.

Make it callable by AI agents (advanced)

Provide structured descriptions of your interfaces so AI agents can call them directly.

For higher-level sites, go further: let AI agents not only read you but safely call your public capabilities — such as queries, purchase guidance, or information retrieval. This is the upgrade from “understandable” to “collaborative”.

Goal

Let AI agents discover and correctly use your public interfaces, securely.

Why it matters

AI agents are becoming the entry point for executing tasks, not just information lookup.
Providing structured descriptions (OpenAPI, machine-readable operation notes) lowers misuse risk.
Proactively designing “how to be called” puts you in a favorable position in the agent ecosystem.

Checklist

Provide machine-readable docs (e.g. OpenAPI/Swagger) for public APIs.
List callable operations and their purposes in llms.txt.
Define permission boundaries: only expose what is genuinely public and safe.
Provide examples and parameter notes to reduce the chance of agent misuse.

Acceptance

An agent can initiate a safe call from the public docs alone.
Unauthorized operations are clearly rejected.
Docs match actual API behavior.