Comments on: 10 common mistakes using robots.txt on your website http://www.thomsonchemmanoor.com/10-common-mistakes-using-robotstxt-on-your-website.html SEO consultant with 9 years experience in Search Engine Optimization Wed, 14 Dec 2011 21:19:36 +0000 hourly 1 http://wordpress.org/?v=3.3 By: Mal http://www.thomsonchemmanoor.com/10-common-mistakes-using-robotstxt-on-your-website.html/comment-page-1#comment-64966 Mal Tue, 18 May 2010 04:04:23 +0000 http://www.thomsonchemmanoor.com/?p=209#comment-64966 RE: #9 -- google's own robots.txt has many 'Allow' statements: http://www.google.com/robots.txt So, is this still right? RE: #9 — google’s own robots.txt has many ‘Allow’ statements: http://www.google.com/robots.txt

So, is this still right?

]]> By: legal pleading templates http://www.thomsonchemmanoor.com/10-common-mistakes-using-robotstxt-on-your-website.html/comment-page-1#comment-64962 legal pleading templates Wed, 05 May 2010 09:48:55 +0000 http://www.thomsonchemmanoor.com/?p=209#comment-64962 Well Written article robots.txt file. This post is most recommendable to read every Web Master Well Written article robots.txt file. This post is most recommendable to read every Web Master

]]> By: Rakesh http://www.thomsonchemmanoor.com/10-common-mistakes-using-robotstxt-on-your-website.html/comment-page-1#comment-64941 Rakesh Mon, 01 Mar 2010 08:11:22 +0000 http://www.thomsonchemmanoor.com/?p=209#comment-64941 Ideal robots.txt file is as follows User-agent: * Crawl-delay: 2 Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /category Disallow: /tag Disallow: /author Disallow: /trackback Disallow: /*trackback Disallow: /*trackback* Disallow: /*/trackback Disallow: /*?* Disallow: /*.html/$ Disallow: /*feed* # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* Ideal robots.txt file is as follows

User-agent: *
Crawl-delay: 2
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*?*
Disallow: /*.html/$
Disallow: /*feed*

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

]]> By: Website Designing Hyderabad http://www.thomsonchemmanoor.com/10-common-mistakes-using-robotstxt-on-your-website.html/comment-page-1#comment-64895 Website Designing Hyderabad Tue, 01 Dec 2009 08:08:19 +0000 http://www.thomsonchemmanoor.com/?p=209#comment-64895 Thanks for valuable information We will follow your rules to create robots txt file for our websites. Thanks for valuable information We will follow your rules to create robots txt file for our websites.

]]> By: Web development lucknow http://www.thomsonchemmanoor.com/10-common-mistakes-using-robotstxt-on-your-website.html/comment-page-1#comment-64847 Web development lucknow Wed, 16 Sep 2009 10:49:22 +0000 http://www.thomsonchemmanoor.com/?p=209#comment-64847 Thanks for providing this info. Thanks for providing this info.

]]>