If you are new here then you will want to CLICK HERE to enter to WIN an LCD TV, Ipod Touch, Flip Mino and More!

Just how important is your blog’s robots.txt file if you want the world to find you and your blog in the search engines?

The robots.txt file is a file on your site that is meant to give instructions as to where search engine spiders may and may not go. This is not a wall but a permission system, which means that you can not force “bad” bots to listen to it. Bad bots are the bots that go all over your site but do not offer you any value at all.

The powerful reason for the robots.txt file is that it is listened to by the majority of all search engines and it helps to ensure that your site gets spidered and indexed properly. That means the pages you want to be found, can be found and the pages you want hidden will remain hidden.

We do not want to go into a long lesson on this, as there are loads of resources available on the topic that can be explained much better then we can explain them. What we will share with you however is that you want to use one and you want to upload it to the root directory on your server, located in the same place as your index page.

You can see the robot.txt we use at http://www.johncow.com/robots.txt

Out of curiosity weve been snooping around a little to see how others do it. Surprisingly enough, we found that there seem to be two different approaches to the system that are total opposites. We’re comparing 4 well known blogs here.

Shoemoney.com / PR6 / Alexa 2,988 - We know Jeremy is pretty tech savvy and he probably is the one with the most knowhow about how this would work. Then again, he might not give a poop about it and just let it be.

Here’s his robots.txt:

User-agent: Googlebot

Disallow: /wp-content/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /sitemap.xml
Disallow: /index.php
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
Disallow: /page/
Disallow: /tag/
Disallow: /category/

User-agent: Googlebot-Image
Disallow: /wp-includes/

User-agent: Mediapartners-Google*
Disallow:

User-agent: ia_archiver
Disallow: /

User-agent: duggmirror
Disallow: /

User-Agent: Googlebot
Disallow: /link.php
Disallow: /gallery2
Disallow: /gallery2/
Disallow: /category/
Disallow: /page/
Disallow: /pages/
Disallow: /feed/
Disallow: /feed

JohnChow.com / PR4 / Alexa 3,071 - Mr Chow has been around the block and we’re assuming he’s quite tech savvy too. Why else would he run a site called TheTechZone for over 8 years? His robots.txt is quite similar to Jeremy’s:

sitemap: http://www.johnchow.com/sitemap.xml

User-agent: *
Disallow: /cgi-bin/
Disallow: /go/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /author/
Disallow: /page/
Disallow: /category/
Disallow: /wp-images/
Disallow: /images/
Disallow: /backup/
Disallow: /banners/
Disallow: /archives/
Disallow: /trackback/
Disallow: /feed/

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Mediapartners-Google
Allow: /

User-agent: duggmirror
Disallow: /

Problogger.net / PR6 / Alexa 2,600 - The Problogger seems to take a totally different approach to things. Being part of B5 Media, an organization that makes money by running blogs, we’re pretty sure that the technical knowhow of SEO is widely available in a team of professionals. A copy of Darren’s robots.txt:

User-agent: *
Disallow:

That’s right. The Problogger doesn’t hold back any secrets for the search engines of this world. Anyone is allowed to crawl through all of Darren’s content.

MattCutts.com / PR7 / Alexa 5,059 - Matt has been working for Google nearly eight years now and is currently head of Google’s webspam team. Surely its safe to assume that Matt knows what he’s doing. Like Problogger, Matt withholds almost nothing for the crawlers, just a files/ folder:

User-agent: *
Disallow: /files/

Eventhough files/ won’t be indexed, Matt has put an index.html saying ‘Sorry’ in place to keep nosy cows like us out of there. Afterall, a robots.txt file is available for anyone to see. put one and one together and you can try to have a peak at the contents of a directory that’s specified in there.

As you can see, there seem to be two trains of thought on the subject.

26 Moos » ~ ~ Random Post

Tons of bloggers used this form of building pagerank when they started out. Heck, we still do it because we know it’ll be worth it in the end. We’re not going to discuss how pagerank can affect your advertising sales. We won’t go into details of how pagerank gets passed on from a high pagerank page that holds a link to your page. We’re not telling you how loads of blogs are using the Nofollow tags for comments, and we’re not telling you how much time it takes to find a high pagerank post where you can leave a (useful) comment, that doesn’t get terminated by the nofollow tag. We all know this is a pain in the behind process and that’s one of the reasons many bloggers stopped doing this.

But then there was Comment Kahuna!

Some clever buggers developed a FREE (we know how much you love free stuff!) application that does exactly what you think it does! Find high pagerank blog posts that allow you to comment freely, without having to fear the nofollow tag

All you do is fill out your profile including your name, email address and your blog URL (This is so that you can later on have the application automatically fill out the comment filds on the blog posts you wish to comment.) Then you save that profile, enter a keyword to narrow down the field to niche related posts, select the minimum pagerank a blog needs to have and hit search! seconds later Comment Kahuna will tell you where to go. It comes with a build-in browser and you can then sort out exactly which of the high PR posts you can abuse where you’d like to get your pr juice from leave behind your meaningful comment.

We’ve discovered this gem a bit late - the guys who’ve created Comment Kahuna are actually closing their doors in less than 20 hours! If you want to take advantage of this devilishly handy piece of freeware, you better get that behind of yours over to their site and grab it before its gone. While you’re there, check out their video to see how easy it really is.

60 Moos » ~ ~ Random Post

So we’ve had a good night sleep and the first thing we did when we woke up was obviously jump out of bed to check yesterday’s statistics and see the damage Digg.com had done to our little blog. Just before we fell asleep, we had counted 1300 diggs, at this moment the number of diggs is a massive 2256! That’s a new record for us seeing the last time we managed to get ‘just’ 1547 diggs for some poor bastard falling off his bike.

So why do we use other pages to get dugg and not just post Digg-bait on our own blog?

Well there’s two main reasons why we prefer to use this method:

1) A lot of the times, popular articles on Digg are silly pictures or video clips. Now even though it doesn’t look like it, we really do try to maintain a certain level of quality on this blog. Wordless image posts is not our idea of a quality post, and thus we choose not to post these here. We have bought JohnCow.* other than dotcom domains for this purpose. Besides having an outlet for Diggbait, it also significantly reduces the risk of getting our dotcom banned from Digg. We all know what happened to John Chow right?

2) Once the picture post hits the frontpage and doesn’t get buried, changes are you’ve just created a nice PR5 domain. Not just because of the linkjuice Digg passes on, but there are tons of sites out there that publish the popular stories from the Digg front page. Popurls.com is one of many for example. Also a lot of .edu sites have mirrors up in case the Dugg site crashes. (We did once yesterday, but a few Apache tweaks sorted that out.) All you do is wait for the next Google update to come around and you’re more than likely going to receive a nice PR. Wrongbob.com is one of our sites that managed to get a PR5 this way. (And now it’s just dead in the water because we don’t know what to do with it )

As far as statistics go, even though it was the weekend, Digg has sent over 50,000 visitors to the submitted article and because we linked everything on that post straight to this blog, we’ve captured over 10,000 unique visits from there. And that’s just day one. If this is going to be anything like the previous Digg, loads of traffic will keep flowing in for days to come. Here’s a screen shot of our .US site that hosts the Digg article, it’s just been online since last week.

Obviously Digg traffic has a high bounce rate and it’s not considered good quality, but we’ve yet to hear the first webmaster or blogger complain about being on the front page! It might not convert in good money, but if you use the method we use,make sure that your landing page has your blog’s brand on it. When people come across your blog months later, they’ll vaguely remember your logo. color scheme and think to themselves;”Hey! I’ve been here before, must be worth checking out”.

And forget about slapping ads on, the few bucks you’ll make of that aren’t worth it in the long run. The price you’ll pay is getting hundreds of diggs less because you will get buried as a spammer.

Right, the digg is over, nothing here to see folks - lets moove along please!

(PS Just wanted to thank Sergio from SOSFACTORY.com for yet another fabulous custom cartoon header and mascot! )

27 Moos » ~ ~ Random Post

Sorry for the late night post again fans, but we’ve been busy trying to keep our server up and running for the last 5 hours because something of ours got dugg hard and we’ve been running around the server farm like mad cows. Currently we’re at 1250 diggs and counting! Thankfully we just moved to the 2nd page after 4 full hours of digg frontpage exposure.

Besides Digg, we were also handling some Del.icio.us traffic and we can only guess how many hits we got in total. We don’t mind, really, but it’s getting late and we want to go to bed! We’ll keep you updated tomorrow.

For those of you that are curious what got dugg, here’s what made a lot of people chuckle.

*Edit* The creator just mailed us, requesting for a linkback to his site. Here you go Steve!

19 Moos » ~ ~ Random Post

We just wanted to give a quick recap on the Entrecard program by Graham Langdon. They ordered a review with us three weeks ago and we signed up with them on that day. Since then we’ve received 729 unique hits from this free ad-exchange program. Not too shabby for any blog if you ask us! If you haven’t signed up yet with Entrecard.com, now’s the time to do it. We’re in a generous mood and will give a random commentator, who displays the entrecard widget on their blog, all 750 of our hard earned free credits. That’s enough to get free ads on JohnChow, Problogger and us - and you’ll still have enough left to advertise on a bunch of cheaper blogs too! (cheap as in cheap entrecard credit-wise, nor cheap as in cheapcheap, if you get what we mean!)

So all you people have to do is leave a comment on this post and we’ll pick a random person (via random.com) to credit the points to his or her account within the next 24 hours. We will obviously be checking if you’ve got the widget running on your blog, so if you’re not signed up yet with entrecard.com, do it now and get a head-start!

70 Moos » ~ ~ Random Post