# User-agent strings Inktomi Slurp WebCrawler GoogleBot Googlebot ZyBorg Openbot Scooter ia_archiver UrlDispatcher Ask Jeeves larbin MarkWatch www.walhello.com Netprospector Crawler Ultraseek AlkalineBOT libwww-perl bumblebee@relevare.com Inet32 Ctrl 3D_SEARCH LinkLint Teleport vspider BaiDuSpider TITAN/ GetRight/ WebTrends Link Analyzer MyApp/ SystemSearch-robot contype lwp-trivial/ LexiBot linkbot cosmos/ webcollage byteserver/ NetAnts/ RaBot NEC-MeshExplorer Crawl_Application Aport nabot_ Verity-URL-Gateway WebSpider viasarchivinginformation.html searchengine autoemailspider Xenu WebStripper Robot Web Downloader WebCopier BizWorks Retriever Pompos/ Gulliver/ ASPSeek/ FirstGov.gov Search psbot/ Sqworm/ MOSES WebWasher CacheBlaster/ FlashGet wwwxref/ VoilaBot/ Gaisbot/ htdig/ grub-client Grubclient- SQ Webscanner SlySearch/ Mercator- gsa-crawler DOF-Verify/ WebCapture EasyDL/ Rotondo/ InternetLinkAgent/ Gigabot/ DigOut4U Cyberdog/ ASPseek/ http://www.almaden.ibm.com/cs/crawler eCatch/ indylabs_marius b2w/ AnswerChase PROve Mass Downloader/ Teradex Mapper Lycos_Spider RPT-HTTPClient/ WebBOT Szukacz/ GaisBot/ targetblaster.com/ Infoseek Sidewinder/ InfoSeek Sidewinder/ Infoseek SideWinder/ DiaGem/ Steeler/ Fluffy the spider searchhippo.com webbandit/ webfetch/ ah-ha.com crawler Lytranslate/ Vagabondo/ moget/ WebZIP/ Robozilla/ Oracle Ultra Search w3m/ EmailLeach OrangeBot iCab/ metacarta TurnitinBot/ DISCoFinder InfoLink/ crawler_for_infomine minibot NetResearchServer/ EmailWolf eidetica.com/spider polybot HomePageSearch spider WebFetch AnswerBot UdmSearch NG/ 3DE_SEARCH2 Mozilla/3.0 (compatible) Mozilla/3.01 (compatible;) DoCoMo/ URLBlaze Ad Muncher Net Vampire/ EyeNetIE www.galaxy.com/info/crawler.html Net Vampire/ DoCoMo/ nabot Pita Knowledge Engine Webclipping.com SSM Agent INGRID/ Mister PiX IAArchiver- DISCo Pump NPBot- Python-urllib/ Zeus Lachesis lachesis HLoader Caddbot/ Zao/ WWW-Mechanize/ Nessus VoilaBot Wget/ WebReaper LARBIN-EXPERIMENTAL efp@gmx.net Goldfire Server Mozilla/2.0 (compatible; NEWT ActiveX; Win32) T-H-U-N-D-E-R-S-T-O-N-E thatrobotsite.com NPBot WebSauger Mozilla/3.0 (compatible; Indy Library) WebGather MFHttpScan NutchOrg/ SiteFinder/ search.usgs.gov TestBot Space Bison/ Sleipnir QuepasaCreep MSNBOT/ pavuk/ VSE/ SearchSpider.com/ SpiderKU/ Dattatec.com-Sitios-Top Microsoft Internet Explorer/4.40.426 WebVac YahooSeeker/ infomine.ucr.edu SiteSweeper PageFetcher/ msnbot/ GetHTMLContents Cowbot- freshlinks.exe Cowdog Bot Art-Online.com AnswerBus MultiText/ Dattatec.com NaverBot- Panopy Bot ez-robot Scich/ Dumbot Web Magnet LinkWalker TranSGeniKBot LinkScan/ Links SQL Google/ antibot- Wotbox/ SpiderEngine X-clawler X-crawler TygoBot TulipChain NetResearchServer LinkSweeper/ Z-Add Link Checker web-agent/ NaverBot_dloader/ mini-robot/ www.searchnear.com Botswana nuSearch Spider slurp CrawlConvera geometabot/ kuloko-bot/ Webdup/ JoeDog/ Dillo/ Ocelli/ GPU p2p crawler TheUsefulbot_ Unitek UniEngine/ COAST scan engine/ BlackMask.Net Search Engine Offline Explorer/ EAH/ Spider.TerraNautic.net Scrubby/ Check&Get IECheck GoForIt.com sohu-search NuSearch Spider Search-Channel NutchCVS/ LinkAlarm/ CSE HTML Validator oBot WISEbot/ boitho.com-dc/ EMPAS_ROBOT Jetbot/ iVia Site Checker findlinks/ USS-Cosmix/ mod_accessibility ht://check/ LookBot CheckLinks/ Html Link Validator osis-project.jp Seekbot/ Iltrovatore-Setaccio/ Website Downloader BDFetch CreativeCommons/ Nutch SOFT411 Directory Yandex/ ClariaBot/ MSNPTC/ heritrix/ HiSoftware AccVerify Diamond/ OmniFind Download Ninja Artera contype FAST-WebCrawler/ Yahoo-VerticalCrawler-FormerWebCrawler/ falcon/ Shareaza ZipppBot/ WorQmada/ Web Link Validator CFNetwork/ CyberSpyder FAST Data Search Document Retriever/ ImageBot ABACHOBot Microsoft Office Protocol Discovery Microsoft Data Access Internet Publishing Provider Cache Manager EmailSiphon searchmarks f-bot test pilot Antro.Net www.sygol.com HSFT - Link Scanner w3search Buscaplus Robi/ adminshop.com statcrawler Clushbot/ Blinkx/DFS-Fetch Header_Test_Client fast-search-engine WebXM WebPix DiamondBot Express WebPictures www.superbargainspace.com archive.org_bot/ omnifind statbot Combine/ SpiderMan searchmarking/ SiteXpert ContentSmartz gazz/ FindAnISP.com_ISP_Finder CCGCrawl/ TravelBot/ NSDL_Search_Bot pipeLiner/ Webinator-WBI/ KummHttp/ crawler.kpricorn.org versus.integis.ch search.updated.com websearchbench Cafi/ LinkChecker/ Microsoft_Site_Analyst/ webcrawl.net HTTrack Buibui Stumbler athenusbot HiSoftware AccMonitor Server MetaGloss SURF TREX Validator/ HooWWWer/ www.thebananatree.org/ Spinne/ Favorites Sweeper MJ12bot/ maxomobot/ KnowItAll VisWeb WebIndex/ BruinBot ShowLinks/ Acme.Spider Nogate CD-Preload websphinx.test Missigua Locator ApacheBench RSSbot/ Yahoo-Newscrawler/ P3P Client googlebot WebAuto/ SiteSnagger Plucker/ NetShift= websitemirror Eventware/ Akamai-SiteSnapshot/ zagrebin theusefulbot inetbot/ iSiloX/ hcat/ gnome-vfs/ didaxusbot WebTrafficExpress/ Talkro Web-Shot/ TPSystem SuperBot/ SpaceBison/ SiteSucker/ SEB Spider Poodle predictor PhpDig/ PageRank Monitor NASA Search MonkeyCrawl/ Mag-Net Lament/ LMQueueBot/ K.S.Bot Hooblybot-Image/ HTMLParser/ GoodJelly/ DiGi-RSSBot CydralSpider/ Amfibibot/ AONDE-Spider/ moduna.com/ BecomeBot/ YottaCars_Bot/ DOY/ Forest Conservation Spider Patwebbot tankvit@e-mail.ru Eco-Portal Spider sherlock/ Knowledge.com/ smartwit.com W3C-WebCon/ Digger/ USyd-NLP-Spider ichiro/ Link Checker/ accoona EduGovSearch/ WinHTTP Example/ FDM 1.x unchaos_crawler_ TygoProwler grub crawler IRLbot/ GigabotSiteSearch/ FreshDownload/ WinHttp.WinHttpRequest. ClimateArk Spider BoaConstrictor/ CyberNavi_WebGet/ updated/ wish-la Custo InelaBot/ oegp v. Water Conserve Spider labourunions411/ SpeedySpider FindWeb OmiExplorer_Bot/ btbot/ Govbot/ tScholarsBot OmniExplorer_Bot/ www.unchaos.com NWSpider www.nameprotect.com cfetch/ AtlocalBot/ abot/ SCrawlTest/ Forests.org Spider aipbot/ MojeekBot/ DataFountains/DMOZ Downloader Speedy Spider McBot/ BigCliqueBot/ eStyleSearch ProjectWF-java-test-crawler Rational SiteCheck wgao@genieknows.com Download Master Kitenga-crawler-bot BigCliqueBOT/ Ocean Conserve Spider snap.com beta crawler IlTrovatore-Setaccio/ Blaiz-Bee/ Metager2 GOFORITBOT genevabot versus crawler Filangy/ Twiceler Eco-Portal http://www.environmentalsustainability.info/ Yahoo-MMAudVid/ nicebot Squid-Prefetch parasite cookieNET PageBitesHyperBot/ Dir_Snatch.exe CipinetBot WebZIP Arachmo ccubee/ IEAutoDiscovery Bilbo/ WebZIP CE-Preload endeca HSFT - LVU Scanner rssImagesBot/ Twisted PageGetter Space Fung/ BrowserEmulator/ Microsoft Scheduled Cache Content Download Service locust PoCoHTTP Web Site Downloader Seeker.lookseek.com Norbert the Spider searchbot grapeFX/ CSHttpClient/ AIBOT/ Miva iVia/ WebIndexer/ Microsoft URL Control AF Knowledge Now Verity Spider MemacBot ExactSearch UofTDB_experiment www.Syntryx.com Charlotte/ NewMedhunt/ Metaspinner/ LeechGet yacy.net MSRBOT/ SygolBot Zyte/ axfeedsbot/ testbot OnetSzukaj/ LinksManager.com_bot Helix/ MyEngines-US-Bot maxamine.com--robot SBIder/ Newsgroupreporter IpselonBot/ byindia/ Xerka WebBot Xerka MetaBot Der große BilderSauger Tecomi Bot fgcrawler TutorGigBot/ yoono/ LocalcomBot/ dtSearchSpider WebMiner/ wbdbot via translate.google.com WebDownloader for X wume_crawler/ Eco Earth Spider Checkbot/ Thumbnail.CZ robot National Park Service Dan Buan (301) 213-4549 KATATUDO-Spider xirq/ Tarantula/ its-learning crawler Cuasarbot # Host names rubys-e-1-msl.cutthroatcom.net 220.114.3.9 cvn012123.bai.ne.jp host81-130-144-116.in-addr.btopenworld.com node-c-4e0c.a2000.nl 62.194.78.12 216.32.82.18 65.119.150.201 65.119.150.202 65.119.150.203 65.119.150.204 65.119.150.205 65.119.150.206 65.119.150.207 65.119.150.208 65.119.150.209 65.119.150.210 65.119.150.211 65.119.150.212 65.119.150.213 67.11.222.90 209.217.67.146