Comments on: User Agent Notes http://42.kaizeku.com/web-browsers/user-agent-notes-gary-keith/ web development, software, windows tips and trick Wed, 07 Jan 2009 17:17:52 +0000 http://wordpress.org/?v=2.6 By: Scolls http://42.kaizeku.com/web-browsers/user-agent-notes-gary-keith/#comment-7910 Scolls Sat, 24 May 2008 20:28:45 +0000 http://blog.kakkoi.net/tips/user-agent-notes-gary-keith/#comment-7910 ScollSpider 2.0 ran for very short periods in testing during development, running only the front page of sites that were already in our index, and thus allowed by the original ScollSpider which had indeed previously loaded their robots.txt files. No internal pages were ever crawled by ScollSpider 2.0. The short crawls were for performance testing. Thanks ScollSpider 2.0 ran for very short periods in testing during development, running only the front page of sites that were already in our index, and thus allowed by the original ScollSpider which had indeed previously loaded their robots.txt files. No internal pages were ever crawled by ScollSpider 2.0. The short crawls were for performance testing.

Thanks

]]>