I’m quite surprise to see my server logs todays, Some dude decide to scrap my blog content (including my wp translations cache 100mb+ )

The Offending uri:
http://www.shouker.com/user1/baiheinet/2008/1/16/80897.html

I’d blocked the site but it wont stop the search engine crawler from indexing the content .

This is nasty Blackhat SEO methods to get the target website penalize for duplicate content on Major Search Engine. There is few solution that i found at various resources ↓.

• Report to Google, proxyreports@gmail.com provide the url & the google search query.
• Block the Proxy Referrer IP
• Add special no index meta for unknown search engine spiders.

  <META NAME="ROBOTS" CONTENT="NOARCHIVE, NOINDEX, NOFOLLOW">

How to track Google Proxy Hacked Duplicate Contents

1. Monitor your content with Google Alerts try used a unique Search terms for your website. i.e: blog.kakkoi, myname, myunique keywords, url http://blog.kakkoi.net, base64 safe uri encode.
   If you have a Google Webmaster Account go to Statistics » What Googlebot sees used the keywords as your Google Alerts search terms.
2. Search for copies of your page on the Web copyscape

Whitelisting Search Engine Crawler

IMO blocking the IP range of Proxy Server is not very practical. Having a Whitelist of Search Engine Crawler IP (class c) might do the trick. I’m working on a script for whitelisting search engine crawler for my wordpress. Hopefully i can finished it later this week.

Google Algo bugs

Dan Thies at seofaststart.com posts a details analysis regarding this issue, check out his post → Google Proxy Hacking: How A Third Party Can Remove Your Site From Google SERPs.

Recent Update

• Caught the proxy user just after I published this articles. Its human 117.8.222.77 / c-net 117.8.0.0/13 from Tianjin, China.
  
• The IP was graylisted on RBL & cml.anti-spam.org.cn so we send a letter to abuse@cnc-noc.net

When Google Wireless Transcoder (GWT, Googlebot-mobile) translate your website it strip all “scripts” and render it in mobile format (XHTML mobile 1.0)Google version of “Mobile format”. To test this services go to http://google.com/gwt/n. GWT services is actually made for mobile-user but you can still surf with normal browser.

So what the heck wrong with it

The answer is Yes & No. This type of services is bad for webmaster that depend on ads income. Otherwise Normal Surfer would love this services as they wont need to view any ads and surf safely without “javascript embed” (from the originating website).

How to Block Googlebot Mobile Crawler

These are some server environment variables for Google Wireless Transcoder

USER_AGENT

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;)

HTTP_VIA

1.1 proxy.google.com:80 (squid)

HTTP_X_FORWARDED_FOR

xxx.xx.xxx.xxx, unknown

REMOTE_ADDR

209.85.138.136

REMOTE_PORT

56931

block via .htaccess

with mod_setenvif

<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent "^Google\ Wireless\ Transcoder*" gwt_agent=1
SetEnvIfNoCase User-Agent "^Googlebot-Mobile*" gwt_agent=1
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=gwt_agent
</FilesMatch>
</IfModule>

or with mod_rewrite

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Google\ Wireless\ Transcoder [OR]
RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Mobile
RewriteRule ^.* - [F,L]
</IfModule>

Robots Exclusion Standards

User-agent: Googlebot-Mobile
Disallow: /

Google Webmaster Analyze robots.txt

After you add the above robot.txt code login to your Google Webmaster Central.

1. Select Tools > Analyze robots.txt
2. Select Google Mobile : Crawls page for our mobile index on “user-agents dropdown list”.

Embed HTML Meta Link header

<link rel="alternate" media="handheld" href="http://changethis-url-for-mobile-user" />

Google Support

If you want to prevent Google Mobile services from transcoding your page its recommended to request for removal via Google Mobile Support form.

Soap

If google-mobile can restrict this services for mobile only view or maybe implement something like “duggmirror” for normal browsing, it would be welcome.

I used gravatars2 plugins to support my new sexy theme. There is some minor issue (throw fatal Error in PHP5.1) with this WordPress plugin. I did asked them to updated it but till today’s this bug still exists with Gravatars2 plugins.

This “fatal error” or conflict happen if you had PHP 5 ( 5.0 > 5.1 above) with HTTPRequest Modules Installed.

Plugin could not be activated because it triggered a fatal error.
Fatal error: Cannot redeclare class httprequest in /../wp-content/plugins/gravatars2.php on line 284

HTTPRequest Classname Conflict

It’s not that hard to fix this “Naming Conflicts”. All you need is “Search and Replace” HTTPRequest class name to different name (ie: _HTTPRequest, HTTP__Request) so it wont conflict with PHP HTTPRequest Standard Class. If you don’t know how to do this. Check the below lists. It wont take long.

1. Open wp-content/plugins/gravatars2.php or http://www.my-domain-name.com/wp-admin/plugin-editor.php?file=gravatars2.php
2. Find on line 284

   class HTTPRequest

   Replace with

   class _HTTPRequest

3. Next find on line 323

   function HTTPRequest($url, $timeout)

   Replace with

   function _HTTPRequest($url, $timeout)

4. Final step find on line 408

   $hr = new HTTPRequest($url, $timeout);

   Replace with

   $hr = new _HTTPRequest($url, $timeout);

5. Save or upload back to wp-content/plugins/

Thats all

Gravatars2

For the record - “Gravatar2 developer doesn’t give support without donation”.

Excerpt from Kip Bond at zenpax.com

I am no longer giving support for this plugin without a donation — it’s becoming repetitive and not very rewarding. You can email me (kip @ this website’s hostname (zenpax.com)) with your question, and I can tell you what minimum donation amount is sufficient per the difficulty of the question. Note that this donation in no way obligates me to any contractual duties. It’s mostly a way to make sure that people have exhausted their own efforts at resolving their own problems before asking for my support. ~kip Bond

I hope these would explain some curiosity.

tips to php developer: used class_exists before declaring any user define class.

Related Links

• Gravatars2 Discussion & Support page

You can used this technique to register on sites and forum. To name a few → digg, myspace, newsvine, 9rules, blogger & msn etc.

Gravatar sign-up process took less than 5 minute to complete and they don’t burden you with filling form chores (i.e: address, newsletter subscriptions, marketing questionnaire). So we pick Gravatar Web services for this guide.

Requirements

You must have the following email.

1. Gmail or Google Aps Gmail

Gmail Plus-Addressing Features

In brief, Gmail services has undocumented plus-addressing features (+) since early 2004. The plus-addressing features is mostly used for writing self notes, email filtering and mapping (google maps).

“Gmail never announce this features (beta forever) as there is no official documentation at Gmail Help Center.” — Anon

RFC 2282 - Internet Message Format

Technically, the plus sign operator is a standard URI protocol for handling Email address including telephone and fax (+tel, +fax ). This standard is maintained by Internet Engineering Task Force (IETF Network Group) → RFC 2282 “Internet Official Protocol Standards - Internet Message Format”.

1. Gravatar Signup Process

The signup pages can be found at the following address ↓

• http://site.gravatar.com/signup

Normal Signup

2. Gravatar Sign-up with Gmail Plus-addressing Features

Gmail Forwarding

Example of valid Gmail plus-sign-address code.

3. How Does It Work?

• Below is raw header received from Gravatar for billgates+mypwned.avatar@gmail.com.

  Delivered-To: billgates@gmail.com
  Received: by 10.114.235.16 with SMTP id i16cs666570wah;
   Tue, 11 Dec 2007 02:13:40 -0800 (PST)
  Received: by 10.100.247.14 with SMTP id u14mr17188447anh.1196668020044;
   Tue, 11 Dec 2007 02:13:40 -0800 (PST)
  Received-SPF: fail (google.com: domain of support@gravatar.com does not designate 72.232.151.155 as permitted sender) client-ip=72.232.151.155;
  Received: by 10.36.223.23 with POP3 id v23mf1698886nzg;
   Tue, 11 Dec 2007 02:13:40 -0800 (PST)
  X-Gmail-Fetch-Info: billgates@gmail.com 1 smtp.gmail.com 995 billgates
  Delivered-To: billgates+mypwned.avatar@gmail.com
  Received: by 10.141.85.4 with SMTP id n4cs336075rvl;
   Tue, 11 Dec 2007 02:13:35 -0800 (PST)
  Received: by 10.151.7.4 with SMTP id k4mr2609357ybi.1197368014852;
   Tue, 11 Dec 2007 02:13:34 -0800 (PST)
  Return-Path: <support@gravatar.com>
  Received: from app1.dfw.gravatar.com (155.151.232.72.static.reverse.ltdomains.com [72.232.151.155])
   by mx.google.com with ESMTP id a13si6664003rnc.2007.12.11.02.13.34;
   Tue, 11 Dec 2007 02:13:34 -0800 (PST)
  Received-SPF: neutral (google.com: 72.232.151.155 is neither permitted nor denied by best guess record for domain of support@gravatar.com) client-ip=72.232.151.155;
  Authentication-Results: mx.google.com; spf=neutral (google.com: 72.232.151.155 is neither permitted nor denied by best guess record for domain of support@gravatar.com) smtp.mail=support@gravatar.com
  Received: from gravatar.com (localhost.localdomain [127.0.0.1])
  	by app1.dfw.gravatar.com (Postfix) with ESMTP id 50EB08F36CA5
  	for <billgates+mypwned.avatar@gmail.com>; Tue, 11 Dec 2007 10:13:34 +0000 (UTC)
  Date: Tue, 11 Dec 2007 10:13:34 +0000
  From: support@gravatar.com
  To: billgates+mypwned.avatar@gmail.com
  Message-Id: <475e62ce4f383_94b15911e666750191132@app1.dfw.gravatar.com.tmail>
  Subject: Welcome to Gravatar
  Mime-Version: 1.0
  Content-Type: text/plain; charset=utf-8
  
  Welcome to gravatar!
  
  To activate your account, simply click on the link below or paste into the url field on your favorite browser:
  
  http://site.gravatar.com/activate/666aff604f5eba8025c1
  
  When you visit the above page, you'll be able to set your password and create your first gravatar, all for free!
  
  If you have any questions about the system, feel free to contact us anytime at support@gravatar.com.
  
  Tom Werner
  Gravatar Founder
  

• When gmail received the email with the plus-addressing code it will split it up as follow
  • billgates+mypwned.avatar@gmail.com
  • login-name = billgates
  • filter-name = mypwned.avatar

  And Gmail will forward the email to the login-name@gmail.com → billgates@gmail.com.

Where to start

Make sure you can received the plus-sign-address email. Try compose new email with the plus sign to yourself.

Example:
Well, assume that your email account is : leeeroyjenkins@gmail.com
So compose new email like so:

From: leeeroyjenkins@gmail.com
Send-To: leeeroyjenkins+selftest@gmail.com
Subject: Hello world
Message-Body: Over, Over, Do u read me.

That should be all. check the related links for further read on Gmail tips and documentations.

Extra notes

There is few special characters you can play with.

1. Period . → billgates+all.your.base.belong.to.us@gmail.com
2. Hypen - → billgates+chuck-norris-fans-club@gmail.com
3. Asterisk *→ billgates+my*space@gmail.com

Related Links

• Gmail Official Discussion Group
• Jim’s tip on How to maintain “notes” in Gmail
• Official Gmail Advanced Search Parameters

There is many reason why you need to block certain website from being access in your network. below is a “the few reason why”.

1. It’s a warez and porn sites.
2. I don’t want my employee to view my Competitor Websites.
3. I’m using illegal software and It seem necessary to disable the automated online registry checkup. ;p
4. I’m against this [countryname] I want to block all this particular domain from being access.
5. I hated this [socialnetworksite]

Safe Blocking

Here’s two methods you can safely used to block or redirect unwanted website from being access without using third party software.

1. Block Website using Windows Host file

Open Window explorer, browse to C:\WINDOWS\system32\drivers\etc click on the file name “host” (the file has no extension) make a backup copy first. Then right click view file properties and disabled the read only attributes and open it with a text editor (i.e: notepad).

Windows host settings instructions note

This file contains the mappings of IP addresses to host names. Each entry should be kept on an individual line. The IP address should be placed in the first column followed by the corresponding host name. The IP address and the host name should be separated by at least one space.

route-to target-hostname
example
127.0.1.1 www.thewebsite.com

note: 127.0.1.1 is you localhost address this is where you want the target-hostname/website to redirect. thewebsite.com is the targeted website URL.

alternatively you can also redirect it to google
64.233.167.99 www.thewebsite.com

Save the file and restore back the read only mode, then type in the block address url in your browser see if works.

OpenDNS filtering

The second methods is universal, its work on any operating systems. OpenDNSfiltering. This articles wont teach you how to setup opendns, you can read it at https://www.opendns.com/start. After you had setup OpenDNS account. Read their KB39 articles

its pretty much straight forward from there on. I’m sure you wont have problem configuring opendns filter . everything is just 2 click way.

Example Blocked Lists

127.0.0.1	babe.the-killer.bz
127.0.0.1	www.babe.the-killer.bz
127.0.0.1	babe.k-lined.com
127.0.0.1	www.babe.k-lined.com
127.0.0.1	did.i-used.cc
127.0.0.1	www.did.i-used.cc
127.0.0.1	coolwwwsearch.com
127.0.0.1	www.coolwwwsearch.com
127.0.0.1	coolwebsearch.com
127.0.0.1	www.coolwebsearch.com
127.0.0.1	hi.studioaperto.net
127.0.0.1	www.hi.studioaperto.net
127.0.0.1	webbrowser.tv
127.0.0.1	www.webbrowser.tv

Notes: Notice the double entries for each domain example.com and www.example.com , You will need both long and short URL for effective blocking. Dont depend on canonical address

DNS (Domain Name System), is the service which translates between Internet names and Internet addresses.

The Domain Name System is one of the foundations of the internet. It is the system that allows the translation of human-readable domain names into machines-readable IP addresses and the reverse translation of IP addresses into domain names.

When you surf on net the dns records will be cache inside your pc. This methods will make your browsing much faster because you wont need to resolve the dns each time you surf a site or network. But sometimes caching can be an issue when certain network change their ip address. You can resolve this by clearing or flushing the dns. Here a few simple command on how to flush your dns cache in PC, Mac & *nix operating systems.

Windows PC

Start > Run > ipconfig /flushdns

Mac

Open Console lookupd -flushcache

Linux

restart nscd daemon.
Open Console /etc/rc.d/init.d/nscd restart

Stop dnscache via Windows services

For windows, If you are having frequent issue with dns caching you can disabled the dns-client caching services with

• Start > Run net stop dnscache
• or manually stopping the services via services.msc (microsoft services console).
  sc servername stop dnscache

I’d many programs installed in my PC (windows XP) within time, the start menu is getting clutter.

so here how you sort the menu in Alphabetical order for windows XP. Its actually pretty easy.

1. Start > All Programs > Select “Any programs”.
2. Right Click and Select “Sort By Name”.

For different version of windows you can digg it at Microsoft KB177482