Kaizeku Crawler Maps

    • Dec
    • 18

    Kaizeku Crawler Maps

    Posted by Noah Ark 8 months, 2 weeks ago.

    Crawler List for November 2007

    Crawlers User agent
    Xirq xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com)
    WebSearchBench WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Universität Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de/
    Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
    NimbleCrawler Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
    Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de)
    Gigabot Gigabot/2.0/gigablast.com/spider.html
    Jambot Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)
    Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don\’t___spam_me_@netluchs.de)
    NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com)
    Bigsearch Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)
    UKWizz UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)
    Ilial/Nutch ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com)
    Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
    Holmes holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
    Flatlandbot flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
    IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html)
    Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
    Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/)
    GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.com)
    Pagebull Pagebull http://www.pagebull.com/
    RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com)
    Dodgebot dodgebot/experimental
    Bot bot/1.0 (bot; http://; bot@bot.bot)
    Bigsearch Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)
    FindLinks findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/)
    ConveraCrawler ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl)
    Blaiz-Bee Blaiz-Bee/2.00.5622 ( http://www.blaiz.net)
    KIT_Fireball KIT_Fireball/2.0
    ICC-Crawler ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)
    Pubblisito info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca
    SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)
    WebAlta Crawler WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
    Pumpkin blogsearchbot-pumpkin-3
    Mail.Ru Mail.Ru/1.0
    Mammoth Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1
    Attentio Attentio/Nutch-0.9-dev (Attentio\’s beta blog crawler; www.attentio.com; info@attentio.com)
    GurujiBot GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
    Gigabot Gigabot/3.0 (http://www.gigablast.com/spider.html)
    Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; jobsde@jobscout24.de) ( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesungen/search/robot.html )
    ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;)
    VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.edu
    IWAgent IWAgent/ 1.0 - www.brandprotect.com
    Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html)
    Spock Crawler Spock Crawler (http://www.spock.com/crawler)
    Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
    Nebulla Nebullabot/2.2 (http://bot.nebulla.de)
    EasyDL EasyDL/3.04 http://keywen.com/Encyclopedia/Bot
    LapozzBot LapozzBot/1.4 (+http://robot.lapozz.hu)
    WWW.fi crawler www.fi crawler, contact crawler@www.fi
    Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt
    NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com
    YodaoBot Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
    DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html)
    DAUM Web Robot Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html)
    Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
    ICC-Crawler ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)
    Semager Semager/1.1 (http://www.semager.de/blog/semager-bots/)
    Multicrawler multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
    NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html)
    Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
    CazoodleBot CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)
    RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html)
    Worio bot Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com)
    Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/)
    Combine Combine/3 http://combine.it.lth.se/
    Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com)
    FuseBulb FuseBulb.Com
    Earthcom Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
    Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info)
    LapozzBot LapozzBot/1.5 (+http://robot.lapozz.hu)
    FAST-WebCrawler FAST Enterprise Crawler/6.4.18 (crawler@fast.no)
    BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html)
    Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider
    Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com)
    Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com)
    Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html)
    LibertyW LibertyW (+http://www.libertyw.eu)
    BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers)
    Holmes holmes/3.11 (http://morfeo.centrum.cz/bot)
    DataparkSearch DataparkSearch/4.47 (+http://dataparksearch.org/bot)
    ImageWalker ImageWalker/2.0 (www.bdbrandprotect.com)
    SeznamBot SeznamBot/2.0-test (+http://fulltext.sblog.cz/)
    Entireweb Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
    BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
    BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
    WebRankSpider WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
    Gungho-crawler Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index)
    PWeBot Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php)
    PWeBot PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php)
    Exabot Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
    Bloglines-Images Bloglines-Images/0.1 (http://www.bloglines.com)
    Doubanbot Doubanbot/1.0 (bot@douban.com http://www.douban.com)
    Disco-crawl disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)
    Disco-crawl disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)
    BotSeer Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu)
    ForAll.pl-Crawler ForAll.pl-Crawler/1.0
    Podtech Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net)
    MSRBot MSRBOT (http://research.microsoft.com/research/sv/msrbot/
    Nsyght nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com)
    Backlink-Check Backlink-Check.de (+http://www.backlink-check.de/bot.html)
    ASAHA ASAHA Search Engine Turkey V.001 (http://www.asaha.com/)
    Sphsearch FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg)
    Google-Adsense Mediapartners-Google
    SAIT sait/Nutch-0.9 (SAIT Research; http://www.samsung.com)
    Teemer Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com)
    Euro-spider Euro-Spider Shopping 1.0
    Lovel Lovel as 1.0 ( +http://www.everatom.com)
    Hermits Search Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com)
    ScoutAnt ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/
    Voyager voyager-hc/1.0
    De.com Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com)
    Yahoo Japan robot DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)
    LijitSpider LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com)
    • December 18, 2007 at 9:12 pm
    • July 5, 2008 at 4:00 pm
    • 0.3
    • url

    • No Responses to “Kaizeku Crawler Maps”

    Notice
    1. Comments are closed.
    2. Trackback & Pingback is Closed.
MySQL query error