AI crawler
Also called: AI bot, AI web crawler, LLM crawler
Definition
An AI crawler is a program that automatically visits web pages on behalf of an AI company. There are three families: training crawlers, which collect pages for future models, search crawlers, which index the web for answers with sources, and on-demand crawlers, which read a page when a user asks for it.
Most of these crawlers don't run JavaScript. A study by Vercel and MERJ covering more than 500 million GPTBot fetches found no JavaScript execution at all: a site whose content only appears in the browser looks empty to them.
Example
A React site without server-side rendering can show its visitors a beautiful page and send GPTBot nothing but a shell with five words in it.
What it means for your AI visibility
Check three things: that your robots.txt lets search crawlers through, that your server or firewall doesn't turn them away, and that your text is in the HTML without JavaScript.
Free toolAI crawler simulatorSee your page the way GPTBot, ClaudeBot and PerplexityBot see it.Related terms
- GPTBotGPTBot is the OpenAI crawler that collects pages to train its models. OpenAI uses other crawlers for ChatGPT's search.
- robots.txtrobots.txt is the file that tells crawlers, including AI crawlers, which parts of a site they're allowed to visit.
- JavaScript renderingWhen a page's content only appears after JavaScript runs, most AI crawlers don't see it. Server-side rendering fixes the problem.
- Google-ExtendedGoogle-Extended is a robots.txt token that lets you refuse the use of your pages for Gemini models. It has no effect on Google Search.
- llms.txtllms.txt is a file placed at the root of a site that introduces the brand to AI and lists the pages to read first.
Updated on · All glossary terms