Glossary
AI crawler

AI crawler

An AI crawler is a bot that fetches web pages so an AI company can train its models or answer user questions with current information.

Common examples include GPTBot, ClaudeBot, and PerplexityBot. You can allow or block them in robots.txt, and tracking which pages they request is a core part of AEO. Webflow AEO Analytics reports AI crawler traffic and the most-crawled pages.

Relevant courses

No items found.

Other glossary terms

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Hmm…we couldn’t find any results. Try a different search term or reset the filter.
Reset the filter
Load more