Crawl Analyzer
Paths
URLs
Dead-ends
Frequency
Orphans
Budget
Verify
Tools
Orphan Analysis
AutomaticBarrier
Compare crawled URLs against your sitemap
Sitemap URL (optional — fetches live)
— or paste sitemap XML directly —
Bot filter
All bots
WordPress-Pingback (14538)
ClaudeBot (4791)
AhrefsBot (4126)
Uptime-Kuma (2751)
Python-requests (2741)
Go-http-client (2379)
Axios (2316)
Barkrowler (1729)
Amazonbot (1513)
SemrushBot (1290)
MetaBot (996)
TerraCotta (952)
GPTBot (929)
DataForSeoBot (860)
DotBot (801)
Baiduspider (760)
Googlebot (566)
Bingbot (536)
OAI-SearchBot (486)
Applebot (456)
Bytespider (316)
Python-httpx (215)
MJ12bot (141)
Mediapartners-Google (48)
FacebookBot (39)
NewsAI (28)
DuckDuckBot (21)
ChatGPT-User (15)
SpiderLing (11)
Google-CloudVertexBot (6)
Google-Site-Verification (4)
PerplexityBot (3)
GeedoBot (2)
Perplexity-User (1)
GoogleOther (1)
AdsBot-Google (1)
Analyze