# User-agent strings Inktomi Slurp WebCrawler GoogleBot Googlebot ZyBorg Openbot Scooter ia_archiver UrlDispatcher Ask Jeeves larbin MarkWatch www.walhello.com Netprospector Crawler Ultraseek AlkalineBOT libwww-perl bumblebee@relevare.com Inet32 Ctrl 3D_SEARCH LinkLint Teleport vspider BaiDuSpider TITAN/ GetRight/ WebTrends Link Analyzer MyApp/ SystemSearch-robot contype lwp-trivial/ lwp-request/ LexiBot linkbot cosmos/ webcollage byteserver/ NetAnts/ RaBot NEC-MeshExplorer Crawl_Application Aport nabot_ Verity-URL-Gateway WebSpider viasarchivinginformation.html searchengine autoemailspider Xenu WebStripper Robot Web Downloader WebCopier BizWorks Retriever Pompos/ Gulliver/ ASPSeek/ FirstGov.gov Search psbot/ Sqworm/ MOSES WebWasher CacheBlaster/ FlashGet wwwxref/ VoilaBot/ Gaisbot/ htdig/ grub-client Grubclient- SQ Webscanner SlySearch/ Mercator- gsa-crawler DOF-Verify/ WebCapture EasyDL/ Rotondo/ InternetLinkAgent/ Gigabot/ DigOut4U Cyberdog/ ASPseek/ http://www.almaden.ibm.com/cs/crawler eCatch/ indylabs_marius b2w/ AnswerChase PROve Mass Downloader/ Teradex Mapper Lycos_Spider RPT-HTTPClient/ WebBOT Szukacz/ GaisBot/ targetblaster.com/ Infoseek Sidewinder/ InfoSeek Sidewinder/ Infoseek SideWinder/ DiaGem/ Steeler/ Fluffy the spider searchhippo.com webbandit/ webfetch/ ah-ha.com crawler Lytranslate/ Vagabondo/ moget/ WebZIP/ Robozilla/ Oracle Ultra Search w3m/ EmailLeach OrangeBot iCab/ metacarta TurnitinBot/ DISCoFinder InfoLink/ crawler_for_infomine minibot NetResearchServer/ EmailWolf eidetica.com/spider polybot HomePageSearch spider WebFetch AnswerBot UdmSearch NG/ 3DE_SEARCH2 Mozilla/3.0 (compatible) Mozilla/3.01 (compatible;) DoCoMo/ URLBlaze Ad Muncher Net Vampire/ EyeNetIE www.galaxy.com/info/crawler.html Net Vampire/ DoCoMo/ nabot Pita Knowledge Engine Webclipping.com SSM Agent INGRID/ Mister PiX IAArchiver- DISCo Pump NPBot- Python-urllib/ Zeus Lachesis lachesis HLoader Caddbot/ Zao/ WWW-Mechanize/ Nessus VoilaBot Wget/ WebReaper LARBIN-EXPERIMENTAL efp@gmx.net Goldfire Server Mozilla/2.0 (compatible; NEWT ActiveX; Win32) T-H-U-N-D-E-R-S-T-O-N-E thatrobotsite.com NPBot WebSauger Mozilla/3.0 (compatible; Indy Library) WebGather MFHttpScan NutchOrg/ SiteFinder/ search.usgs.gov TestBot Space Bison/ Sleipnir QuepasaCreep MSNBOT/ pavuk/ VSE/ SearchSpider.com/ SpiderKU/ Dattatec.com-Sitios-Top Microsoft Internet Explorer/4.40.426 WebVac YahooSeeker/ infomine.ucr.edu SiteSweeper PageFetcher/ msnbot/ GetHTMLContents Cowbot- freshlinks.exe Cowdog Bot Art-Online.com AnswerBus MultiText/ Dattatec.com NaverBot- Panopy Bot ez-robot Scich/ Dumbot Web Magnet LinkWalker TranSGeniKBot LinkScan/ Links SQL Google/ antibot- Wotbox/ SpiderEngine X-clawler X-crawler TygoBot TulipChain NetResearchServer LinkSweeper/ Z-Add Link Checker web-agent/ NaverBot_dloader/ mini-robot/ www.searchnear.com Botswana nuSearch Spider slurp CrawlConvera geometabot/ kuloko-bot/ Webdup/ JoeDog/ Dillo/ Ocelli/ GPU p2p crawler TheUsefulbot_ Unitek UniEngine/ COAST scan engine/ BlackMask.Net Search Engine Offline Explorer/ EAH/ Spider.TerraNautic.net Scrubby/ Check&Get IECheck GoForIt.com sohu-search NuSearch Spider Search-Channel NutchCVS/ LinkAlarm/ CSE HTML Validator oBot WISEbot/ boitho.com-dc/ EMPAS_ROBOT Jetbot/ iVia Site Checker findlinks/ USS-Cosmix/ mod_accessibility ht://check/ LookBot CheckLinks/ Html Link Validator osis-project.jp Seekbot/ Iltrovatore-Setaccio/ Website Downloader BDFetch CreativeCommons/ Nutch SOFT411 Directory Yandex/ ClariaBot/ MSNPTC/ heritrix/ HiSoftware AccVerify Diamond/ OmniFind Download Ninja Artera contype FAST-WebCrawler/ Yahoo-VerticalCrawler-FormerWebCrawler/ falcon/ Shareaza ZipppBot/ WorQmada/ Web Link Validator CFNetwork/ CyberSpyder FAST Data Search Document Retriever/ ImageBot ABACHOBot Microsoft Office Protocol Discovery Microsoft Data Access Internet Publishing Provider Cache Manager EmailSiphon searchmarks f-bot test pilot Antro.Net www.sygol.com HSFT - Link Scanner w3search Buscaplus Robi/ adminshop.com statcrawler Clushbot/ Blinkx/DFS-Fetch Header_Test_Client fast-search-engine WebXM WebPix DiamondBot Express WebPictures www.superbargainspace.com archive.org_bot/ omnifind statbot Combine/ SpiderMan searchmarking/ SiteXpert ContentSmartz gazz/ FindAnISP.com_ISP_Finder CCGCrawl/ TravelBot/ NSDL_Search_Bot pipeLiner/ Webinator-WBI/ KummHttp/ crawler.kpricorn.org versus.integis.ch search.updated.com websearchbench Cafi/ LinkChecker/ Microsoft_Site_Analyst/ webcrawl.net HTTrack Buibui Stumbler athenusbot HiSoftware AccMonitor Server MetaGloss SURF TREX Validator/ HooWWWer/ www.thebananatree.org/ Spinne/ Favorites Sweeper MJ12bot/ maxomobot/ KnowItAll VisWeb WebIndex/ BruinBot ShowLinks/ Acme.Spider Nogate CD-Preload websphinx.test Missigua Locator ApacheBench RSSbot/ Yahoo-Newscrawler/ P3P Client googlebot WebAuto/ SiteSnagger Plucker/ NetShift= websitemirror Eventware/ Akamai-SiteSnapshot/ zagrebin theusefulbot inetbot/ iSiloX/ hcat/ gnome-vfs/ didaxusbot WebTrafficExpress/ Talkro Web-Shot/ TPSystem SuperBot/ SpaceBison/ SiteSucker/ SEB Spider Poodle predictor PhpDig/ PageRank Monitor NASA Search MonkeyCrawl/ Mag-Net Lament/ LMQueueBot/ K.S.Bot Hooblybot-Image/ HTMLParser/ GoodJelly/ DiGi-RSSBot CydralSpider/ Amfibibot/ AONDE-Spider/ moduna.com/ BecomeBot/ YottaCars_Bot/ DOY/ Forest Conservation Spider Patwebbot tankvit@e-mail.ru Eco-Portal Spider EcoEarth Portal sherlock/ Knowledge.com/ smartwit.com W3C-WebCon/ Digger/ USyd-NLP-Spider ichiro/ Link Checker/ accoona EduGovSearch/ WinHTTP Example/ FDM 1.x unchaos_crawler_ TygoProwler grub crawler IRLbot/ GigabotSiteSearch/ FreshDownload/ WinHttp.WinHttpRequest. ClimateArk Spider BoaConstrictor/ CyberNavi_WebGet/ updated/ wish-la Custo InelaBot/ oegp v. Water Conserve Spider labourunions411/ SpeedySpider FindWeb OmiExplorer_Bot/ btbot/ Govbot/ tScholarsBot OmniExplorer_Bot/ www.unchaos.com NWSpider www.nameprotect.com cfetch/ AtlocalBot/ abot/ SCrawlTest/ Forests.org Spider aipbot/ MojeekBot/ DataFountains/DMOZ Downloader Speedy Spider McBot/ BigCliqueBot/ eStyleSearch ProjectWF-java-test-crawler Rational SiteCheck wgao@genieknows.com Download Master Kitenga-crawler-bot BigCliqueBOT/ Ocean Conserve Spider snap.com beta crawler IlTrovatore-Setaccio/ Blaiz-Bee/ Metager2 GOFORITBOT genevabot versus crawler Filangy/ Twiceler Eco-Portal http://www.environmentalsustainability.info/ Yahoo-MMAudVid/ nicebot Squid-Prefetch parasite cookieNET PageBitesHyperBot/ Dir_Snatch.exe CipinetBot WebZIP Arachmo ccubee/ IEAutoDiscovery Bilbo/ WebZIP CE-Preload endeca HSFT - LVU Scanner rssImagesBot/ Twisted PageGetter Space Fung/ BrowserEmulator/ Microsoft Scheduled Cache Content Download Service locust PoCoHTTP Web Site Downloader Seeker.lookseek.com Norbert the Spider searchbot grapeFX/ CSHttpClient/ AIBOT/ Miva iVia/ WebIndexer/ Microsoft URL Control AF Knowledge Now Verity Spider MemacBot ExactSearch UofTDB_experiment www.Syntryx.com Charlotte/ NewMedhunt/ Metaspinner/ LeechGet yacy.net MSRBOT/ SygolBot Zyte/ axfeedsbot/ testbot OnetSzukaj/ LinksManager.com_bot Helix/ MyEngines-US-Bot maxamine.com--robot SBIder/ Newsgroupreporter IpselonBot/ byindia/ Xerka WebBot Xerka MetaBot Der große BilderSauger Tecomi Bot fgcrawler TutorGigBot/ yoono/ LocalcomBot/ dtSearchSpider WebMiner/ wbdbot via translate.google.com WebDownloader for X wume_crawler/ Eco Earth Spider Checkbot/ Thumbnail.CZ robot National Park Service Dan Buan KATATUDO-Spider xirq/ Tarantula/ its-learning crawler Cuasarbot Ipselonbot/ Deepindex DataSpearSpiderBot/ libiViaCore/ generate_infomine_category_classifiers RufusBot sohu agent Everest-Vulcan Inc./ Yahoo-Blogs/ focused_crawler SocSciBot SiteArchive WebMiner wacbot ObjectsSearch/ Pockey-GetHTML/ OutfoxBot/ SearchBlox PHPCrawl STEGMANN-Bot Plumtree 6.0; AChulkov.NET page walker SearchIt-Bot/ crawler@ LetsCrawl.com/ COAST WebMaster Pro/ voyager/ AVSearch- InspireBot Eco Earth Portal genieBot Theophrastus/ MyFamilyBot/ fr_crawler Myra Wavefire/ Forest Conservation Portal, 1Noonbot ActiveTouristBot DataSpider/ MSRBOT Kyluka krawl Water Conserve Portal Cerberian Drtrs silk/ BOI_crawl_00 Vortex/ ZoomSpider KSbot/ BuyHawaiiBot Arikus_Spider ImagesHereImagesThereImagesEverywhere/ GruBot OpenIntelligenceData/ Evaal/ cvaulev c r a w l 3 r Forschungsportal/ LBot Girafabot combine/ COMBINE/ www.dlese.org virus-detector web crawler Link Checker Scumbot/ SearchSpider.com GulperBot Wysigot AccMonitor Compliance Server DTAAgent Y!J-BSC/ NetSongBot/ full_breadth_crawler DataFountains/ AISIID/ Vermut WebCorp/ Poirot searchmee_v Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler PerMan Surfer rsssupport@repia.com Climate Change Portal http://www.climateark.org/ zimeno/ webcrawler RAMPyBot YahooSeeker-Testing/ vermut +http://vermut.aol.com pucl/ personal ultimate crawler Snoopy __TBJ_WEB_CRAWLER__ TERAGRAM_CRAWLER www.octora.com exactseek.com VIP/ Exabot-Images/ BuildCMS crawler mercuryboard_user_agent_sql_injection.nasl Linkman wwwster/ AdamM Bot, webbot bzBot/ worldshop/ search.msn.com/msnbot.htm NewsGatherer/ EcoEarth.Info Environment Portal bot/ FurlBot/ ScollSpider; Favcollector/ WIRE/ NLese Feedfetcher-Google; FeedBurner/ Jakarta Commons-HttpClient/ IlTrovatore/ INFOMINE/ Mammoth/ Searchmee! Spider Crawl/ BecomeJPBot/ Sokitomi crawl; http://www.sokitomi.com/crawl.html Megite Geomaxenginebot Skywalker VIPr/ WEBCRAWLER@VUNET.ORG KeepNI web site monitor MFcrawler 1on1searchBot/ Yeti SuperPagesBot/ Search Publisher seeqpod-vertical-crawler NewsTroveBot MQbot metaquerier.cs.uiuc.edu Big Brother Froola Bot webGobbler/ zedzo.validate/ zedzo.digest/ FedContractorBot/ EARTHCOM.info/ !Susie Harvest/ BaiduSpider virus_detector Oracle Secure Enterprise Search FlashCapture Alpha Search Agent yoogliFetchAgent yarienavoir.net/ topicblogs/ robotek page_verifier online link validator augurfind Selflinkchecker scSpider/ KakcleBot JetBrains Omea Pro HyperEstraier/ Gigabot 12soso/ pixfinder/ JoyScapeBot/ BySpider DXSeeker/ psycheclone bottybot Exabot-Test/ Exabot-XXX/ Nusearch Spider MaSagool/ WeRelateBot/ Fetch API Request TargetYourNews.com bot robots/ MAINSEEK_BOT Lydia Entity Spider Link Validator BIGLOTRON Bookmark Buddy bookmark checker UniversalSearch/ imds_monitor/ BilgiBetaBot/ VMBot/ crawl@digigetx.com perform_crawl HSlide/ Vital Search'n Urchin AESpider/ NaverBot/ METASpider SumitBot SquidClamAV_Redirector AstroFind/ QFKBot Website Quester DigExt; DTS Agent Weddings.info Bot/ ozelot/ Webinator-search2.fasthealth.com/ PsBot qualidade/ WEPA/ Blog Conversation Project; factbot ODP entries t_st; BilgiBot/ YahooFeedSeeker/ statedept-crawler YahooFeedSeeker Testing/ Exploder/ miniRank/ Spider wastrix/ TridentSpider/ arianna.libero.it WebImages Touche StackRambler/ PicoSearch/ NetMechanic vlad/ JobSpider_BA/ InsumaScout/ Fopper Chameleon/ Climate Ark http://www.climateark.org/ info seeker/ vlsearch (http://vlib.org/admin/robot) MnoGoSearch/ Pagebull http://www.pagebull.com/ http://pressemitteilung.ws/ ConnectSearch SurveyBot/ Factbot iVia Page Fetcher LinkCheck Scanner/ IU_CSCI_B659_class_crawler/ SurfControl kinjabot (http://www.kinja.com) Rondello/ FDSE robot SrevBot holmes/ IlseBot/ gsa-accuracyEval Qweery_robot.txt_CheckBot/ NLESE USEPA BLT/ Earth Science Educator robot crawl@globrix.com Nerima-crawl- TerrawizBot/ DataparkSearch/ Gordon-College-Google-Mini TravelLazerBot/ PrivacyFinder/ cs-crawler +http://citeseer.ist.psu.edu del.icio.us-thumbnails/ LT Scotland Checklink/ nextthing.org/ nys-crawler Y!J-PSC/ Blogslive JemmaTheTourist Web-Sniffer/ (www.cotse.net; Anon Proxy) ShopWiki/ pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp) crawler43.ejupiter.com QweeryBot/ Net::Trackback/ BYINDIA/ FU-NBI/FU-NBI- IIITBot (pvvpr@iiit.net) Y!J-SRD/ Hatena Antenna/ RedCarpet/ GurujiBot/ RSS_READER (mctwist@mail.dr-k.info) WebaltBot/ GigaBot/ AASP/ Whirlpool Web Engine OmniWeb http://www.mozilla.org/docs/en/bot.html; master@mozilla.com AboutUsBot/ TRAAZI/ masidani_bot_ LamerExterminator/ Lsearch/sondeur Yoono; http://www.yoono.com/ core-project/ zibber-v (www.zibb.com/crawler/) Pansophica/ iSearch/ ImageVisu/ Yoriwa/ kulturarw3 +http://www.kb.se/kw3/ bladder fusion 1-More Scanner OutfoxMelonBot/ GeoVisu/ BoardReader-Image-Fetcher cataguru/ FunnelBack; http://cyan.funnelback.com/ LM Harvester Metaeuro Web Search Treezy/ Mozilla/2.0 (compatible; MSIE 4.0; Windows 98) DataCha0s/ wish-project (http://wish.slis.tsukuba.ac.jp/) AlexfDownload My_Little_SearchEngine_Project/ Vacobot; (+http://vaco.ws/bot.html) http://www.t6labs.com// SumeetBot my-heritrix-crawler( yellowJacket/ ChemieDE-NodeBot/ Koninklijke Bibliotheek web archive (heritrix +http://www.kb.nl) BlogMyWay.Net/BlogMyWay-0.8.1 (admin@blogmyway.org) MYCOMPANYBOT RoboPal (http://www.goldcave.com/) Heritrix/ LapozzBot/ domaincrawler/ ScientificCommons.org/ OpenISearch/ HarvestMan Yahoo-Test/ Ocean Conserve http://www.oceanconserve.org/ gsa (Enterprise; GIX- Pogodak.co.yu/ Trovator heritrix bot ExaBotTest/ Little Grabber at Skanktale.com USAF AFKN K2SPIDER kbeta1 +http://www.kotoha.co.jp Exabot Test/ SiteOrbiter Interseek/ SnapPreviewBot Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@kyluka.com here will be link to crawler site testing of bot; PiyushBot ROBOT Tailrank; http://tailrank.com/robot PythonWikipediaBot/ AntiSantyWorm StarDownloader/ fembot (myd@cs.stanford.edu) VisBot/ penthesila/ ucb-nutch/ FeedChecker/ Btsearch/ libcrawl/ grbot Semager/ imo-google-robot-intelink AlexaWebSearchPlatform; +http://websearch.alexa.com http://www.changedetection.com/bot.html TeezirBot/ Canon-WebRecordPro/ pulseBot (pulse Web Miner) FDM 2.x Faviconizer crawler/ Hyperix/ wenbin/search LeapTag/ WebSpear/ Dit/ wwwrobot iim_405/ VadixBot SkreemRBot +http://skreemr.com Bigado.com/ Search-Engine-Studio OsO; http://oso.octopodus.com/abot.html +http://www.convera.com) TapuzBot/ GalaxyBot/ WikiaBot nestReader/ LolongBot/ mozilla (nlmoccssearchadmin@mail.nlm.nih.gov) Dwaar crawler (dwaarbot@dwaar.com) semanticdiscovery/ maxamine.com-robot LeapTag ( IIITBOT/ Sphider2 MSMOBOT http://www.artiesoft.com/lexxbot.php Francis/ RcStartBot Intelix/ Snappy/ imagefortress +http://www.worldbank.org) woriobot (+http://worio.com) opidig_1.0 (dfuhry@cs.kent.edu) BuscadorClarin/ voyager-partner-deep/ Blackbird/ CazoodleBot/ Obvius external linkcheck/ Sphere Scout&v GoogleReport Search Engine - http://www.googlereport.org Attributor.comBot Dwaarbot (dwaarbot@dwaar.com) NatchCVS/ (Natch; http://lucene.apache.org/natch/bot.html; natch-agent@lucene.apache.org) Obvius external linkcheck/ TsWebBot/ Sphere Scout PIENO robot robot; http://www.xrss.eu/robot; Webscan +http://otc.dyndns.org/webscan/ Bot; http://www.activetourist.com Canon-WebRecord/ HD nutch agent/ Distilled-Reputation-Monitor/ slow-crawler +http://casr.ou.edu WebBot/ Camcrawler (+http://www.camdiscover.com/crawler.html) mowserbot; http://www.mowser.com/bot Anonymous/3G bot optidiscover/ Giant/ (Openmaru bot; robot@openmaru.com) ASAHA Search Engine Turkey BlogPulseLive (support@blogpulse.com) Charlotte DiBot Elblindo the Blind Bot MT-Soft (http://www.mt-soft.com.ar) Mediapartners-Google Mozilla/5.0 (FunnelBack) Jim +http://www.hanzoarchives.com) SearchnowBot_v1; +http://www.searchnow.com) SummizeBot +http://www.summize.com) Wazzup1.0. pogodak.ba/ PWeBot/ PageDown/ froGgle/ kinjabot medrabbit/ optidiscover/ semisearch/ stero (http://www.stero.pl; News_Search_App/ Giant/ (Openmaru bot; robot@openmaru.com) Compatible;Viking/ BlogRefsBot/ SummizeFeedReader +http://www.summize.com Wazzup1.0.4800; http://32.fb.354a.static.theplanet.com/Wazzup) dejan/ hul-wax +http://hul.harvard.edu/ois/projects/webarchive/) sslbot +http://www.networking4all.com) rtgibot; http://rtgi.fr/) owsBot/ (owsBot; www.oneworldstreet.com; owsBot) Sapienti/Indexer KeywenBot/ Chilkat/ woriobot (+http://www.worio.com/) REBI-Shoveler/ LiteFinder/ gsa (Enterprise; wastrix/ CRAWLER-ALTSE.VUNET.ORG-Lynx you-dir/ voyager-hc/ DjangoTraineeBot/ BSearchR&D/ webLyzard/ testBOT/ suchclip (Kalooga; http://www.kalooga.com; info@kalooga.com) fetch_ici/ Wikio (http://www.wiko.fr) UCLA%20Google%20Serch%20Appliance%20%232%20%28contact%3A%20 UCLA%20Google%20Serch%20Appliance%20%231%20%28contact%3A%20 sslbot +http://www.networking4all.com) hul-wax +http://hul.harvard.edu/ois/projects/webarchive/) http://32.fb.354a.static.theplanet.com/Wazzup) Najdi.si/ EARTHCOM/ Giant/ (Openmaru bot; robot@openmaru.com) superbot.com; +http://www.super.info) (Exabot-Thumbnails) Zotag Search egothor/ SafariBookmarkChecker/ sportcrew-Bot (Grub.org crawler; http://www.grub.org/; bot@grub.org) search x-bot The Dyslexalizer @ http://spunc.dsturgeon.net taxinomiabot DAUMOA-video; +http://ws.daum.net/aboutkr.html Website Explorer/ iHWebChecker WEP Search Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal) Mozilla/3.0 [en] (AWV2.72f) Proximic crawler; +http://www.proximic.com/en/about-us/contact-us.html) OpiDig DAUMOA-video; +http://ws.daum.net/aboutkr.html) quest.durato/ (Suchmaschine der durato Ltd.; http://quest.durato.de; quest@durato.de) kalooga/ (Kalooga; http://www.kalooga.com; info@kalooga.com) Zotag Search Atomic_Email_Hunter/ besserscheitern-crawl WatzBot Verifactrola/ Toplistbot TopServer PHP Webwasher/ TSM Translation-Search-Machine (www.ttn.ch) REBI-shoveler/ (REBI's great worker; http://rebi.co.kr; noreply@rebi.co.kr) DAUMOA-web; +http://ws.daum.net/aboutkr.html) Yahoo-Kids/ mailto:vertical-crawl-support@yahoo-inc.com) link_checker/ Bot Apoena http://www.katatudo.com.br/ajuda/ AmPmPPC.com (http://www.ampmppc.com/) Topicalizer/www.topicalizer.com) FormulaFinderBot/ AmPmPPC.com (+http://www.ampmppc.com/) Runnk RSS aggregator : http://www.runnk.com/ G10-Bot/ autowebdir 1.1 (www.autowebdir.com) D1GArabicEngine/ crawlmaster@d1g.com) travel-search GoSeebot; +http://www.gosee.com/bot.html) Earth Platform Indexer wikiwix-bot- EnaBot/ ninetowns woriobot +http://worio.com) Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ....../1.0 ) Getleft GetLeft ShablastBot Quintura-Crw/ TinEye/ (http://tineye.com/crawler.html) (XML Sitemaps Generator iearthworm/ iearthworm@yahoo.com.cn C:\Documents and Settings\Joe\Desktop\HARVEST EMAILS\SEGMENTS\ GeonaBot/ BloobyBot iearthworm/ YesupBot/ ; +http://www.yesup.net/bot.html) CCBot/ (+http://www.commoncrawl.org/bot.html) HMSEbot crawler-upgrade-config JoBo/ Daumoa/ proximic; +http://www.proximic.com) ScoutJet; +http://www.scoutjet.com/) InfoUSABot/ Attributor/Dejan- (Test crawler; http://www.attributor.com; info at attributor com) exooba/exooba crawler (exooba; exooba) zermelo; +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com] BitvoUserAgent (+http://www.bitvo.com) Bitvo/ zermelo; +http://www.powerset.com) culsearch/culs/ (crawl@citeulike.org) Hostcrawler Sunrise XP/ AnotherBot wisponbot(http://www.wispon.com,mailto:wispon@theory.snu.ac.kr) woriobot support [at] worio [dot] com +http://worio.com) GrubNG swish-e http://swish-e.org/ search.KumKie.com SeeqBot +http://www.seeqpod.com YebolBot zermelo/ +http://www.powerset.com/about/zermelo) hijbul-heritrix-crawler (+http://mobide.korea.ac.kr/) Knight/ (Zook Knight; http://knight.zook.in/; knight@zook.in) Sphider BpBot/ (- -; http://blitzpost.com; search@blitzpost.com) FeedHub MetaDataFetcher/ attributor/ +http://www.attributor.com) Kyluka crawl; http://www.kyluka.com/static/crawl.html; crawl@kyluka.com) TeamSoft WinInet Component DotSpotsBot/ (crawler; support at dotspots.com) msnbot-products FeedFetcher(www.radian6.com/crawler) ScooperBot www.customscoop.com Yahoo Pipes 3GSE bot (Internet Research Institute UK, http://iri-uk.com) Kyluka crawl; crawl@kyluka.com; http://www.kyluka.com/static/crawl.html) 192.comAgent +http://www.evri.com/evrinid) Google Keyword Tool; +https://adwords.google.com/select/KeywordToolExternal) CamelStampede/ eSyndiCat Bot crawl.UserAgent ShadowWebAnalyzer (http://www.safety-lab.com/) Vishal For CLIA/clia-alpha-testing (Crawling for CLIA project ; www.cfilt.iitb.ac.in; vishalv@cse.iitb.ac.in) CollapsarWEB qihoobot@qihoo.net) Yanga WorldSearch Bot Runnk online rss reader : http://www.runnk.com/ : RSS favorites : RSS ranking : RSS aggregator hybridwse@runnk.com BlitzBOT@tricus.net ; ODI3 Navigator) Axonize-bot DotBot/ Yanga WorldSearch Bot Vishal For CLIA/ (Crawling for CLIA project ; www.cfilt.iitb.ac.in; vishalv@cse.iitb.ac.in) OOZBOT/ (--; http://www.setooz.com/oozbot.html; agentname at setooz dot_com) Runnk online rss reader wauuu engine/Wauuu (wauuu engine; http://www.wauuu.com; wauuu@wauuu.com) NESSUS::SOAP healia/healia (the personalized health search engine.; http://www.healia.com) Climate Ark - http://www.climateark.org/) ^Byte (http://CaretByte.com) A1 Sitemap Generator/ (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot JadynAve - http://www.jadynave.com/robot JadynAveBot; +http://www.jadynave.com/robot Drupal (+http://drupal.org/) (vBSEO; http://www.vbseo.com) crawly@commandcom.com FatBot http://www.thefind.com/crawler) CoolCheck iCopyright Conductor Firebat (http://lms.virtual-presence.org) Google Bot 2 Beta ornl_crawler_1 Mozilla crawl/ (compatible; frt/ BlogScope/ +http://www.blogscope.net/; U of Toronto) snookit/Snookit (domains@snookit.com) NetID.com Bot IOI/ (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org) betaBot xqrobot RIIGHTBOT/RIIGHT- (riight.com; http://www.riight.com/riightbot; riightbot@riight.com) Y!J-BRI/ crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/ Grub/ (IOI crawler; http://index.isc.org/; crawl@index.isc.org) Labhoo+(+http://www.labhoo.com/) # Host names offense.sses.net .crawl.yahoo.net .netvigator.com crawler.bloglines.com .verity.com user.connect.gpo.gov mail.encharter.org sandbox-d.gsfc.nasa.gov sandbox-qa1.gsfc.nasa.gov gcmdstage.gsfc.nasa.gov www.mvspy.com doi-esn-gw.customer.alter.net .keymachine.de p6043-ipbfp305yamaguchi.yamaguchi.ocn.ne.jp 122.54.176.147.pldt.net 64-28-180-114-rev.cernel.net 200.87.117.125 123.226.146.43 210.180.187.248 12.192.176.244 219.134.139.39 124.244.194.27 124244194027.ctinets.com 85.19.205.0 162.140.67.10 .ask.com .maxamine.net .abhsia.telus.net simple5.dragonara.net c30099.upc-c.chello.nl ABTS-North-Dynamic-031.46.161.122.airtelbroadband.in 116.30.36.170