I’m quite surprise to see my server logs todays, Some dude decide to scrap my blog content (including my wp translations cache 100mb+ )

The Offending uri:
http://www.shouker.com/user1/baiheinet/2008/1/16/80897.html

I’d blocked the site but it wont stop the search engine crawler from indexing the content .

This is nasty Blackhat SEO methods to get the target website penalize for duplicate content on Major Search Engine. There is few solution that i found at various resources ↓.

• Report to Google, proxyreports@gmail.com provide the url & the google search query.
• Block the Proxy Referrer IP
• Add special no index meta for unknown search engine spiders.

  <META NAME="ROBOTS" CONTENT="NOARCHIVE, NOINDEX, NOFOLLOW">

How to track Google Proxy Hacked Duplicate Contents

1. Monitor your content with Google Alerts try used a unique Search terms for your website. i.e: blog.kakkoi, myname, myunique keywords, url http://blog.kakkoi.net, base64 safe uri encode.
   If you have a Google Webmaster Account go to Statistics » What Googlebot sees used the keywords as your Google Alerts search terms.
2. Search for copies of your page on the Web copyscape

Whitelisting Search Engine Crawler

IMO blocking the IP range of Proxy Server is not very practical. Having a Whitelist of Search Engine Crawler IP (class c) might do the trick. I’m working on a script for whitelisting search engine crawler for my wordpress. Hopefully i can finished it later this week.

Google Algo bugs

Dan Thies at seofaststart.com posts a details analysis regarding this issue, check out his post → Google Proxy Hacking: How A Third Party Can Remove Your Site From Google SERPs.

Recent Update

• Caught the proxy user just after I published this articles. Its human 117.8.222.77 / c-net 117.8.0.0/13 from Tianjin, China.
  
• The IP was graylisted on RBL & cml.anti-spam.org.cn so we send a letter to abuse@cnc-noc.net

When Google Wireless Transcoder (GWT, Googlebot-mobile) translate your website it strip all “scripts” and render it in mobile format (XHTML mobile 1.0)Google version of “Mobile format”. To test this services go to http://google.com/gwt/n. GWT services is actually made for mobile-user but you can still surf with normal browser.

So what the heck wrong with it

The answer is Yes & No. This type of services is bad for webmaster that depend on ads income. Otherwise Normal Surfer would love this services as they wont need to view any ads and surf safely without “javascript embed” (from the originating website).

How to Block Googlebot Mobile Crawler

These are some server environment variables for Google Wireless Transcoder

USER_AGENT

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;)

HTTP_VIA

1.1 proxy.google.com:80 (squid)

HTTP_X_FORWARDED_FOR

xxx.xx.xxx.xxx, unknown

REMOTE_ADDR

209.85.138.136

REMOTE_PORT

56931

block via .htaccess

with mod_setenvif

<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent "^Google\ Wireless\ Transcoder*" gwt_agent=1
SetEnvIfNoCase User-Agent "^Googlebot-Mobile*" gwt_agent=1
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=gwt_agent
</FilesMatch>
</IfModule>

or with mod_rewrite

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Google\ Wireless\ Transcoder [OR]
RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Mobile
RewriteRule ^.* - [F,L]
</IfModule>

Robots Exclusion Standards

User-agent: Googlebot-Mobile
Disallow: /

Google Webmaster Analyze robots.txt

After you add the above robot.txt code login to your Google Webmaster Central.

1. Select Tools > Analyze robots.txt
2. Select Google Mobile : Crawls page for our mobile index on “user-agents dropdown list”.

Embed HTML Meta Link header

<link rel="alternate" media="handheld" href="http://changethis-url-for-mobile-user" />

Google Support

If you want to prevent Google Mobile services from transcoding your page its recommended to request for removal via Google Mobile Support form.

Soap

If google-mobile can restrict this services for mobile only view or maybe implement something like “duggmirror” for normal browsing, it would be welcome.