A directory of known web crawlers and bots
Need the entire list of bots with User Agents and IP ranges? Buy the complete Botective dataset
No bots found
2ip_bot
2ip.ru
360Spider
360 Search
adbeat_bot
Adbeat
AhrefsBot
Ahrefs
AI21Bot
AI21 Labs
AI2Bot
Allen Institute for AI
aiHitBot
aiHit
Algolia-Crawler
Algolia
Amazonbot
Amazon
AmazonKendra
Amazon
AmazonQ
Amazon
AndersPinkBot
Anders Pink
Andibot
Andi Search
AppleBot
Apple
Arquivo-web-crawler
Arquivo.pt
AwarioBot
Awario
Baiduspider
Baidu
Barkrowler
Babbar
BazQux
BazQux Reader
bedrockbot
Amazon
BetterUptime
BetterStack
Bingbot
Microsoft
bitlybot
Bitly
BitSightBot
BitSight
BLEXBot
Link Assistant
Botify
Botify
Bravebot
Brave
Brightbot
Bright Data
Bytespider
ByteDance
CCBot
Common Crawl
CensysBot
Censys
ChatGPT-User
OpenAI
Checkly
Checkly
CheckMarkNetwork
CheckMarkNetwork
Chrome-Lighthouse
Claude
Anthropic
Cloudflare Healthchecks
Cloudflare
coccocbot
Coc Coc
Cocolyzebot
Cocolyze
CohereBot
Cohere
Cookiebot
Cookiebot
CookieYesbot
CookieYes
Cortex-Xpanse
Palo Alto Networks
Cotoyogi
Research Organization of Information and Systems
Crawlspace
Crawlspace
CriteoBot
Criteo
DatadogSynthetics
Datadog
DataForSeoBot
DataForSEO
Deepcrawl
Lumar
Detectify
Detectify
Diffbot
Diffbot
Discordbot
Discord
DomainStatsBot
DomainStats
DotBot
Moz
DuckAssistBot
DuckDuckGo
DuckDuckBot
DuckDuckGo
EchoboxBot
Echobox
EtaoSpider
Alibaba
ev-crawler
Headline
FacebookBot
Meta
Factset_spyderbot
FactSet
Feedly
Feedly
FirecrawlAgent
Firecrawl
GeedoProductSearch
Geedo
Gemini-Deep-Research
GenomeCrawler
Nokia
Googlebot
Googlespecial
GoogleUserTriggered
GPTBot
OpenAI
HaoSouSpider
Haosou (360)
heritrix
Internet Archive
HetrixTools
HetrixTools
hypestat
HypeStat
iaskspider
iAsk
IbouBot
Ibou
ICC-Crawler
NICT
ImagesiftBot
Hive
img2dataset
LAION
InternetMeasurement
drifnet
IonCrawl
IONOS
ISSCyberRiskCrawler
ISS
KagiBot
Kagi
KangarooBot
Kangaroo LLM
Klaviyo
Klaviyo
KStandBot
URL Classification
linkdexbot
Linkdex
LinkedInBot
linkfluence
Meltwater
magpie-crawler
Brandwatch
Mail.RU_Bot
Mail.Ru Group
Mediatoolkitbot
Determ
MistralAI-User
Mistral AI
MJ12bot
Majestic
Modat
Modat
MojeekBot
Mojeek
NaverBot
Naver
NetcraftBot
Netcraft
NewRelicbot
New Relic
Nicecrawler
NiceCrawler
OAI-SearchBot
OpenAI
omgili
Webz.io
OnCrawl
OnCrawl
Onetrust CMP Scanner
OneTrust
OpenRobotsTXT
OpenRobotsTXT
Operator
OpenAI
Pandalytics
Pandalytics
PanguBot
Huawei
Panscient
Panscient
Perplexity-User
Perplexity
PerplexityBot
Perplexity
PetalBot
Huawei
PhindBot
Phind
Pingdom
SolarWinds
PinterestBot
Prerender
Prerender.io
Proximic
Comscore
QualifiedBot
Qualified
QuillBot
QuillBot
Qwantify
Qwant
RamblerBot
Rambler
rogerbot
Moz
SBIntuitionsBot
SB Intuitions
Scrapy
Zyte
Screaming Frog SEO Spider
Screaming Frog
SeekportBot
Seekport
SemaltBot
Semalt
SemanticScholarBot
Semantic Scholar
SemrushBot
Semrush
SentryUptimeBot
Sentry
SenutoBot
Senuto
SeobilityBot
Seobility
SEOkicks-Robot
SEOkicks
SEOlizer
SEOlizer
serpstatbot
Serpstat
SeznamBot
Seznam
Site24x7
Zoho
SkypeUriPreview
Microsoft
Slackbot
Slack
SlurpBot
Yahoo
SnapBot
Snap Inc.
Sogou
Sogou
SSL Labs
Qualys
StartpageBot
Startpage
StatusCake
StatusCake
Stripe
Stripe
Sucuri
Sucuri
Swiftbot
Swifttype
SwisscowsBot
Swisscows
TelegramBot
Telegram
Teoma
Ask.com
Timpibot
Timpi
TinEye
TinEye
TTD-Content
The Trade Desk
Twilio
Twilio
Twitterbot
X (Twitter)
UptimeRobot
UptimeRobot
VirusTotalBot
Google (VirusTotal)
WARDBot
ward.ai
WebPageTest
WebPageTest
WellKnownBot
Well-Known
Meta
wpbot
Unknown
yacybot
YaCy
YahookMailProxy
Yahoo
YahooLinkPreview
Yahoo
YandexBot
Yandex
YisouSpider
SM.cn
YouBot
You.com
ZoomBot
SEOZoom
ZumBot
Zum