Comments on: How Important Is Robots.txt For Google? http://www.johncow.com/how-important-is-robotstxt-for-google/ Making Money Online Blogging by Milking the Internet! Wed, 17 Feb 2010 18:07:21 +0000 http://wordpress.org/?v=abc hourly 1 By: Do you have a ROBOT.txt file? // Blogging Help // BlogCatalog http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-13207 Do you have a ROBOT.txt file? // Blogging Help // BlogCatalog Sat, 19 Jan 2008 10:15:16 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-13207 <!--%kramer-ref-pre%-->[...] minutes ago Kiwipulse You can look an article that johncow did. www.johncow.com/how-important-is-robotstxt-for-google/ Well I allow google bot for image, which I receive lots of traffic from google image. Also I allow [...]<!--%kramer-ref-post%--> [...] minutes ago Kiwipulse You can look an article that johncow did. http://www.johncow.com/how-important-is-robotstxt-for-google/ Well I allow google bot for image, which I receive lots of traffic from google image. Also I allow [...]

]]> By: allen johnson http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-12580 allen johnson Mon, 07 Jan 2008 16:39:56 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-12580 Yah I would n,t blame you problogger is probably the best choice the reason why he doesn't hold anything from google is that he has so many incoming links that probably leads to his older post Yah I would n,t blame you problogger is probably the best choice the reason why he doesn’t hold anything from google is that he has so many incoming links that probably leads to his older post

]]> By: Allyn Paul http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11828 Allyn Paul Tue, 25 Dec 2007 14:05:07 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11828 Cow--thanks for taking some time to reply to my question...I appreciate the solid information! AL Cow–thanks for taking some time to reply to my question…I appreciate the solid information!

AL

]]> By: ajacx http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11821 ajacx Tue, 25 Dec 2007 12:41:01 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11821 can i put this robot.txt in my blog spot? can i put this robot.txt in my blog spot?

]]> By: Holiday Ramblings | Nate Whitehill dot Com http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11810 Holiday Ramblings | Nate Whitehill dot Com Tue, 25 Dec 2007 02:19:42 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11810 [...] Cow writes a detailed analysis of the different methods that some of the top bloggers are using their robots file. Definitely and [...] [...] Cow writes a detailed analysis of the different methods that some of the top bloggers are using their robots file. Definitely and [...]

]]> By: John Cow http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11805 John Cow Mon, 24 Dec 2007 20:11:05 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11805 The sitemap will make it easier for Google to index your site. The robots.txt allows you to tell the search engine crawlers what they can and can't put on file. Because of duplicate content that could be picked up by a crawler (a post in your archives has a different URL but still hods the exact same content as it's main URL) Google for example might think you're spamming. (although we're not convinced by that. Surely Google is smarter than that.) The sitemap will make it easier for Google to index your site. The robots.txt allows you to tell the search engine crawlers what they can and can’t put on file.

Because of duplicate content that could be picked up by a crawler (a post in your archives has a different URL but still hods the exact same content as it’s main URL) Google for example might think you’re spamming. (although we’re not convinced by that. Surely Google is smarter than that.)

]]> By: Allyn Paul http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11803 Allyn Paul Mon, 24 Dec 2007 19:47:22 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11803 This is way over my head! I have a plugin that creates a Google sitemap...does that suffice in terms of this posting and the robot txt? This is way over my head! I have a plugin that creates a Google sitemap…does that suffice in terms of this posting and the robot txt?

]]> By: Nicholas James http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11783 Nicholas James Mon, 24 Dec 2007 00:00:42 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11783 Excellent post. I completly forgot about robots.txt and this reminded me for my blog :wink: Excellent post.

I completly forgot about robots.txt and this reminded me for my blog :wink:

]]> By: Think Like An SOB http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11765 Think Like An SOB Sun, 23 Dec 2007 18:42:24 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11765 lol. I have been using Shoe's robots.txt file for my blog since its inception. Figured, he probably knows what he's doing when it comes to SEO. lol. I have been using Shoe’s robots.txt file for my blog since its inception. Figured, he probably knows what he’s doing when it comes to SEO.

]]> By: vhxn.com http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11761 vhxn.com Sun, 23 Dec 2007 18:12:55 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11761 Thanks for the interesting Post :arrow: Thanks for the interesting Post :arrow:

]]> By: 5 Human Wonders of the World | The Blogrepreneur http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11753 5 Human Wonders of the World | The Blogrepreneur Sun, 23 Dec 2007 08:16:08 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11753 [...] Cow after a long time has published a very good post related to SEO. His blog is in my Top 10 Useless Blog list. Maybe I have to start re-considering it. John, keep [...] [...] Cow after a long time has published a very good post related to SEO. His blog is in my Top 10 Useless Blog list. Maybe I have to start re-considering it. John, keep [...]

]]> By: StumbleUpon - Your page is now on StumbleUpon! http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11748 StumbleUpon - Your page is now on StumbleUpon! Sat, 22 Dec 2007 18:57:18 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11748 <!--%kramer-ref-pre%-->[...] Your page is on StumbleUpon [...]<!--%kramer-ref-post%--> [...] Your page is on StumbleUpon [...]

]]> By: John Cow dot Com » Blog Archive » Limited Opening Hours These Holidays http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11745 John Cow dot Com » Blog Archive » Limited Opening Hours These Holidays Sat, 22 Dec 2007 18:03:44 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11745 [...] we’ve decided to disable the comments section for a while, or at least till some people can make up their minds. We know, this post is probably better than most rubbish that we’ve written in a long time, [...] [...] we’ve decided to disable the comments section for a while, or at least till some people can make up their minds. We know, this post is probably better than most rubbish that we’ve written in a long time, [...]

]]> By: Althaf http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11744 Althaf Sat, 22 Dec 2007 17:37:09 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11744 Good to see a useful post after a long time. Good to see a useful post after a long time.

]]> By: John Cow http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11743 John Cow Sat, 22 Dec 2007 17:34:06 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11743 Mojo, Mike? Mojo, Mike?

]]> By: Mike Huang http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11742 Mike Huang Sat, 22 Dec 2007 17:20:14 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11742 Milk Man, even though this is an interesting post...we all know you can do better. Where's the mojo? :) -Mike Milk Man, even though this is an interesting post…we all know you can do better. Where’s the mojo? :)

-Mike

]]> By: Simon http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11739 Simon Sat, 22 Dec 2007 15:41:42 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11739 I take the Shoemoney / Chow approach, as I only allow the indexing of posts themselves, and a couple of pages. Shoemoney was given a few tips by Aaron Wall on how to get out of supplementals (when they were public), which mostly revolved around the robots.txt - it was to stop duplicate content being indexed, and according to Shoemoney: "I am happy to report not only am I out of supplemental hell but my Google traffic has increased 1400% in only 1 month after implementing his list of stuff." I would expect that ProBlogger has been better organised from the start, and has never needed robots.txt to help with duplicate content. All his archives (categorys, dates etc) are just excerpts, which has always been thought to help. I take the Shoemoney / Chow approach, as I only allow the indexing of posts themselves, and a couple of pages.

Shoemoney was given a few tips by Aaron Wall on how to get out of supplementals (when they were public), which mostly revolved around the robots.txt – it was to stop duplicate content being indexed, and according to Shoemoney:

“I am happy to report not only am I out of supplemental hell but my Google traffic has increased 1400% in only 1 month after implementing his list of stuff.”

I would expect that ProBlogger has been better organised from the start, and has never needed robots.txt to help with duplicate content. All his archives (categorys, dates etc) are just excerpts, which has always been thought to help.

]]> By: Clog Money http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11737 Clog Money Sat, 22 Dec 2007 14:37:11 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11737 No it needs to be publicly accessible, you would be surprised how many robots actually look for this file, not just google but tons of other scrapers. Who knows you may even get the odd back link here and there from it. No it needs to be publicly accessible, you would be surprised how many robots actually look for this file, not just google but tons of other scrapers. Who knows you may even get the odd back link here and there from it.

]]> By: Chris Jacobson http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11736 Chris Jacobson Sat, 22 Dec 2007 14:12:16 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11736 Is there any way to hide robots.txt from prying eyes and make it only viewable to search robots? Is there any way to hide robots.txt from prying eyes and make it only viewable to search robots?

]]> By: Ruchir http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11735 Ruchir Sat, 22 Dec 2007 14:10:44 +0000 http://www.johncow.com/how-important-is-robotstxt-for-google/#comment-11735 I don't think it makes much of a difference. Shoe and John just disallow all those directories for security purposes... I don’t think it makes much of a difference. Shoe and John just disallow all those directories for security purposes…

]]>