Glossary
AI crawler
AI crawler
An AI crawler is a bot that fetches web pages so an AI company can train its models or answer user questions with current information.
Common examples include GPTBot, ClaudeBot, and PerplexityBot. You can allow or block them in robots.txt, and tracking which pages they request is a core part of AEO. Webflow AEO Analytics reports AI crawler traffic and the most-crawled pages.
Other glossary terms
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Hmm…we couldn’t find any results. Try a different search term or reset the filter.
Reset the filter