Six things that make a website readable by AI assistants

This is the list I run on every site I build or fix, in the order the failures usually turn up. Each item has a test you can run in a minute and a copyable fix. The vocabulary changes every quarter, answer engine optimisation this year, generative engine optimisation last year, and the page changes underneath are the same six.

How does an AI assistant read a page?

It fetches the HTML and reads what is in it, without running JavaScript. Vercel measured this across its network in December 2024: none of the major AI crawlers rendered JavaScript, including OpenAI's and Anthropic's. They fetch the script files and do not execute them. So a page is what its HTML says, and anything that arrives later, through a script, does not exist for the assistant.

Two kinds of visits matter. Crawlers build an index or a training set ahead of time. Fetchers arrive when a person asks a question and the assistant opens a page to answer it. Each vendor documents its own names, and they are the names your robots file and your firewall see.

1. Are the crawlers allowed in?

A site that never wrote a rule about AI crawlers allows them, because a rule nobody wrote allows everything. The block usually comes from somewhere else: a security plugin with a "block bad bots" switch, or the CDN. Cloudflare has blocked AI crawlers by default on new domains since 1 July 2025, so a site put behind it this year is closed to ChatGPT and Claude unless someone opened it. Check the robots file, then the host's firewall settings, then the response itself.

Whether to let the training crawlers in is your decision. Whether to let the search and user bots in is the whole question of this post: if they cannot fetch the page, nothing else on this list matters. Listing them by name is documentation for whoever adds a blanket rule later.

1# The bots that answer questions with links. Named one by one so a
2# later "block all bots" rule from a plugin or a CDN cannot catch them.
3User-agent: Googlebot
4User-agent: Bingbot
5User-agent: OAI-SearchBot
6User-agent: ChatGPT-User
7User-agent: Claude-SearchBot
8User-agent: Claude-User
9User-agent: PerplexityBot
10User-agent: Perplexity-User
11Allow: /
12
13# The training crawlers. Your call. Allowed here; swap for Disallow: /
14# to opt out of training while staying in the answers.
15User-agent: GPTBot
16User-agent: ClaudeBot
17User-agent: Google-Extended
18Allow: /
19
20User-agent: *
21Allow: /
22
23Sitemap: https://www.example.com/sitemap.xml

The response check matters as much as the file. A header of X-Robots-Tag: noindex keeps a page out of every index whatever robots.txt says, and it is easy to leave on after a launch. This site shipped with exactly that header on every response until launch day, on purpose, and the launch playbook has removing it as a named step because it is so easy to forget.

2. Is the content in the HTML?

Open the page's source and search for a sentence from the middle of it. If the sentence is there, the assistant can read it. If the source is a near-empty shell with a script tag, the page is rendered in the browser and the assistant sees the shell. Sites built on Webflow, Framer, Astro or Next.js on Vercel ship the content in the HTML by default. Single-page apps and pages that load their text after the page do not.

1# Count how often a sentence from the page appears in the HTML the
2# server sends. 0 means the assistant cannot see it.
3curl -sL https://www.example.com/pricing | grep -c "49 EUR a month"

The partial failures are the ones that hide. Tabs and accordions whose panels are fetched when opened. A "read more" that loads the rest of the article. Prices pulled from an API after the page shows. Text inside images. Each of those is invisible to a fetcher that does not run scripts, and the fix is the same in every case: put the text in the HTML, and use the script only to show and hide it.

3. Does each heading ask the question and answer it at once?

An assistant looks for the piece of a page that answers the question it was asked, and a heading in the reader's own words is how it finds that piece. The first forty to sixty words under the heading have to be a complete answer on their own, because that is the size of the excerpt an engine lifts. A section that starts with "as mentioned above" cannot be lifted, and a heading like "Our approach" cannot be matched to anything anyone asks.

1## Our approach to pricing
2
3We believe in transparent, fair pricing that grows with you...
4
5
6## How much does the Studio plan cost?
7
8The Studio plan costs 49 EUR a month, billed monthly, with no
9setup fee and no minimum term. It includes three seats and
10unlimited pages. Annual billing is 490 EUR, two months free.

• Headings are questions the reader would type, not labels the company likes.
• The answer comes first, then the reasons. Facts go in tables and lists, where they can be read one at a time.
• Every section stands alone. No "above", no "as we said", no pronoun whose noun is in another section.
• Full product names on first mention in each section, because a section may be the only part that gets read.

4. Are the summary and the key takeaways at the top?

A summary of two to four sentences, written after the piece and able to stand alone, is the block an assistant lifts when it lifts one thing. A key takeaways list, one sentence per point under a real heading, is the second. Both go at the top, directly under the title and byline, because an assistant often reads the top of a page and stops. A takeaways box at the end is read by people who finished and by almost nothing else. Neither block is the meta description, which is written for a search result, and neither is the intro, which is written to be read on.

The markup side is modest, and honest about it. Schema.org has no type for a takeaways list, and inventing one is worse than leaving it out. What exists: the article's abstract carries the summary text, and a speakable specification points at the two blocks by their ids, the one property that means "this block answers on its own". Google documents speakable for news; other engines read it as a hint. The takeaways list is plain HTML under a real H2, which is what gets read.

1{
2  "@type": "BlogPosting",
3  "abstract": "The summary, exactly as it appears on the page.",
4  "speakable": {
5    "@type": "SpeakableSpecification",
6    "cssSelector": ["#in-short", "#takeaways"]
7  }
8}

5. Is the FAQ real?

A FAQ block is worth having only when the questions are ones people actually ask, in their words, with a direct answer of forty to eighty words each. The reason used to be Google's FAQ rich result, the expandable questions under a listing. Google limited that to government and health sites in August 2023 and stopped showing it altogether on 7 May 2026. The markup is still valid, and the questions still get read by the assistants. What changed is that padding no longer pays.

• Sources, in order: questions customers and readers put to you, People Also Ask under your own queries, real threads on Reddit and Hacker News.
• Four to eight per page. Fewer real ones beat more invented ones.
• Direct answer first, then the reason, forty to eighty words. Every answer complete without the rest of the page.
• The FAQPage markup repeats the visible text exactly. Markup for questions that are not on the page is the one thing that can get a site penalised here.

1{
2  "@type": "FAQPage",
3  "mainEntity": [
4    {
5      "@type": "Question",
6      "name": "Does the price change for existing customers?",
7      "acceptedAnswer": {
8        "@type": "Answer",
9        "text": "No. Anyone on the Studio plan before 1 October keeps 39 EUR a month for as long as the subscription runs. The new price applies to plans started on or after that date."
10      }
11    }
12  ]
13}

6. Does the structured data name a real author?

An article with a named person behind it, on a page of their own with a photo, a job title and links to their profiles, is more citable than the same words under "Team". The markup makes the connection explicit: the article's author is a reference to a Person node, and that node lives on the author page with the same id. The author page is one route and an afternoon's work. It is also the part I see skipped most often.

• One graph per page: the article, a breadcrumb, and references by id to the site's Organization, WebSite and Person nodes, which are emitted once from the layout.
• On the article: headline, description, image, datePublished, dateModified, mainEntityOfPage, publisher, wordCount, and author as a reference, never a string.
• On the author page: ProfilePage containing the Person, with image, jobTitle, worksFor, knowsAbout and sameAs for LinkedIn and X.
• Never Review or AggregateRating unless real people left real ratings. There are none on a blog.
• Generate the markup from the same data that renders the page, so it cannot drift from what the page shows.

1{
2  "@context": "https://schema.org",
3  "@graph": [
4    {
5      "@type": "BlogPosting",
6      "@id": "https://www.example.com/blog/pricing-from-october",
7      "mainEntityOfPage": "https://www.example.com/blog/pricing-from-october",
8      "headline": "Pricing from October",
9      "description": "What changes on 1 October and what stays the same.",
10      "datePublished": "2026-09-25",
11      "dateModified": "2026-09-25",
12      "author": { "@id": "https://www.example.com/authors/jane-doe#person" },
13      "publisher": { "@id": "https://www.example.com/#organization" }
14    },
15    {
16      "@type": "Person",
17      "@id": "https://www.example.com/authors/jane-doe#person",
18      "name": "Jane Doe",
19      "url": "https://www.example.com/authors/jane-doe",
20      "jobTitle": "Head of Web",
21      "worksFor": { "@id": "https://www.example.com/#organization" },
22      "sameAs": ["https://www.linkedin.com/in/janedoe"]
23    }
24  ]
25}

The author page on this site is the reference implementation, and the schema audit skill checks a site's markup against this list.

What about llms.txt?

Optional, and cheap when it is generated. No major assistant has said it reads the file, and the adoption studies through this year found no effect on citations. It costs nothing when it derives from the same route list as the sitemap, which is how this site produces its own. What goes in the file, and whether anything reads it, is its own post.

How do you check a page?

Six tests, one per item, each under a minute. On this site they run as a build gate: a page with a missing description, a second H1, or structured data that does not name the page fails the build rather than shipping. The launch check skill does the same for any site from a terminal.