geo-crawlers
Lets your agent analyze AI crawler access. This helps understand how search engines see your site.
What it does
This tool checks if AI robots can access your website. If they are blocked, your content won't be used by AI tools, even if it's high quality.
When to use it
- ✓If you want your site to appear in AI-generated answers.
- ✓If you're worried old settings might be blocking AI robots.
- ✓To ensure your content is visible to tools like ChatGPT and Perplexity.
Example of what to ask your AI
“Check if AI robots can access my website.”
Want more? Open the sections below.
How to use
The easiest way: copy the message below and paste it into Claude, ChatGPT, Cursor or any AI. It reads the skill and follows the instructions.
Skill link
Ready-to-paste message
How it works
The tool looks at your website's rules for robots (called 'robots.txt'). It checks if important AI robots, like OpenAI's GPTBot, are allowed.
Installation
Easiest way: copy the skill link and paste it into Claude, ChatGPT or Cursor asking it to follow the instructions.
In the terminal (Claude Code, Codex, Cursor), in your project folder:
npx skills add sickn33/agentic-awesome-skills --skill geo-crawlersOr download just the file:
curl -L https://raw.githubusercontent.com/sickn33/agentic-awesome-skills/main/plugins/agentic-awesome-skills-claude/skills/geo-crawlers/SKILL.md -o SKILL.mdTechnical content (original SKILL.md)
Original text by the author, straight from GitHub.
AI Crawler Access Analysis Skill
Purpose
This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO.
Key Insight
As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,000 websites block at least one major AI crawler, and 5-10% block all AI crawlers. Blocking AI crawlers is the single fastest way to become invisible in AI-generated search results.
Complete AI Crawler Reference
Tier 1: Critical for AI Search Visibility (RECOMMEND: ALLOW)
These crawlers power the AI search products where users actively look for answers. Blocking them directly reduces your visibility in AI-generated responses.
GPTBot
- Operator: OpenAI
- User-Agent:
GPTBot - Full User-Agent String:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) - Purpose: Fetches content for ChatGPT's web browsing, plugins, and search features. Content accessed by GPTBot may be used to improve OpenAI models.
- Impact of Blocking: Content will NOT appear in ChatGPT Search results or be accessible when users ask ChatGPT to browse the web. This is the highest-impact AI crawler to allow.
- Recommendation: ALLOW -- ChatGPT has 300M+ weekly active users as of 2025. Blocking GPTBot removes your content from one of the largest AI search surfaces.
OAI-SearchBot
- Operator: OpenAI
- User-Agent:
OAI-SearchBot - Full User-Agent String:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://docs.openai.com/bots/overview) - Purpose: Specifically powers ChatGPT's search feature. Unlike GPTBot, content accessed by OAI-SearchBot is NOT used for model training -- only for live search results.
- Impact of Blocking: Content will not appear in ChatGPT's search results even if GPTBot is allowed.
- Recommendation: ALLOW -- This is a search-only crawler with no training implications. There is no strategic reason to block it.
ChatGPT-User
- Operator: OpenAI
- User-Agent:
ChatGPT-User - Full User-Agent String:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) - Purpose: Used when a ChatGPT user explicitly asks the model to visit a specific URL. Acts like a browser agent on behalf of the user.
- Impact of Blocking: ChatGPT cannot visit your pages when users ask it to read or summarize them. This prevents direct user-initiated traffic.
- Recommendation: ALLOW -- Blocking this bot prevents users who are actively trying to engage with your content from accessing it through ChatGPT.
ClaudeBot
- Operator: Anthropic
- User-Agent:
ClaudeBot - Full User-Agent String:
ClaudeBot/1.0; +https://www.anthropic.com/claude-bot - Purpose: Fetches web content for Claude's features including web search, citations, and analysis tools.
- Impact of Blocking: Content will not be accessible to Claude for web search or when users ask Claude to analyze specific URLs.
- Recommendation: ALLOW -- Claude is a major AI assistant with growing market share. Blocking ClaudeBot reduces your AI search footprint.
PerplexityBot
- Operator: Perplexity AI
- User-Agent:
PerplexityBot - Full User-Agent String:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) - Purpose: Powers Perplexity's AI search engine, which provides sourced answers with direct citations and links back to source pages.
- Impact of Blocking: Content will not appear in Perplexity search results. Perplexity is one of the best referral traffic sources among AI search products because it always displays source links.
- Recommendation: ALLOW -- Perplexity drives actual referral traffic and always attributes sources. High-value AI crawler for publishers and businesses.
Tier 2: Important for Broader AI Ecosystem (RECOMMEND: ALLOW)
…
FAQ
Why do AI robots need to access my website?
AI robots access your site to gather information and learn from it. This allows your content to be used in AI-generated responses, like those from ChatGPT.
What happens if I block AI robots?
If you block AI robots, your content won't be found or used by these tools. This means it won't show up in AI searches or generated answers.
Which AI robots are most important?
The most important robots include GPTBot (OpenAI), OAI-SearchBot (OpenAI), ChatGPT-User (OpenAI), ClaudeBot (Anthropic), and PerplexityBot (Perplexity AI). Allowing them access is crucial for AI visibility.
Details
Popularity
- GitHub stars
- 48k
- Ranking position
- #3104
- License
- MIT
- Updated
- 10/15/2026
Comments
- …
More skills from this repository
00-andruia-consultant
Helps your agent start or improve AI projects. It analyzes code, suggests tools, and creates action plans in Spanish.
sickn33/agentic-awesome-skills
007
Lets your agent perform security audits and threat modeling. It's useful for protecting projects against attacks and vulnerabilities.
sickn33/agentic-awesome-skills
10-andruia-skill-smith
Helps your agent create or improve AI tools. It defines how tools work, writes documentation, and validates the structure.
sickn33/agentic-awesome-skills
20-andruia-niche-intelligence
Lets your agent create a summary about a specific industry. It's useful for understanding the context, rules, usage patterns, and risks of a project, using reliable sources.
sickn33/agentic-awesome-skills
Similar skills
competitive-platform-analysis
Helps understand your competitors. It identifies, categorizes, and filters competing companies before any deeper analysis.
affaan-m/ECC
competitive-report-structure
Creates comprehensive competitor reports. It combines information to show the overall landscape, recommendations, and helps align the team.
affaan-m/ECC
lead-intelligence
Helps your agent find and contact important people. It analyzes leads, suggests approaches, and manages outreach via email and social media.
affaan-m/ECC