Crawl Analyzer
Paths
URLs
Dead-ends
Frequency
Orphans
Budget
Verify
Tools
Orphan Analysis
RisingArmBarriers
Compare crawled URLs against your sitemap
Sitemap URL (optional — fetches live)
— or paste sitemap XML directly —
Bot filter
All bots
ClaudeBot (8853)
AhrefsBot (7219)
Amazonbot (5949)
WordPress-Pingback (3508)
Barkrowler (3273)
Go-http-client (3047)
Uptime-Kuma (2999)
Axios (2319)
GPTBot (2201)
SemrushBot (2026)
DataForSeoBot (1686)
Bingbot (1599)
Python-requests (1280)
DotBot (1090)
Baiduspider (1058)
WebScraperBot (936)
Googlebot (883)
Bytespider (720)
Applebot (718)
MetaBot (653)
TerraCotta (606)
NewsAI (485)
OAI-SearchBot (451)
SpiderLing (423)
MJ12bot (347)
Python-httpx (335)
GoogleOther (260)
GeedoBot (185)
Mediapartners-Google (95)
FacebookBot (80)
CrawlAnalyzer (41)
DuckDuckBot (25)
ChatGPT-User (17)
Google-CloudVertexBot (8)
Google-Site-Verification (8)
Perplexity-User (4)
PerplexityBot (2)
Analyze