SEO

I prob'ly don't need to tell you this, but don't ever ask search engines to delist your site. Ever. They actually do it.

A bit of background...

Last month I relaunched justin hileman dot info with a fresh new theme, and a couple of cool features. While I was developing the replacement site, I had a staging version which I didn't want Google to index. I used a handy little robots.txt file to keep the search engine spiders at bay:

User-agent: *
Disallow: /

This file did the trick.

The staging site wasn't indexed, exactly as advertised. Unfortunately when the time came to deploy, my über-restrictive robots.txt file overwrote the existing file and slipped into the live site.

Google, Yahoo, and friends dutifully ignored every page on my site. The majority of the damage was instantaneous. Most high traffic and high PR pages were unindexed within a few hours. My crawl rate dropped, the major search engines removed more pages every time they crawled my site.

My SERP traffic, understandably, tanked.

Yesterday my search engine referrals hit zero.

The sharp drop in SERP traffic on February 12th coincides with the first Google crawl with the new robots.txt. The second drop, around the 23rd was the result of Yahoo's reindex. In just a few days my site was completely unlisted from the major search engines.

As of the time of this post, a search for "justin hileman dot info", which should result in about 1500 pages on this domain, returns nothing.

How could this have been avoided?

Google Webmaster tools provides a great overview of your site. It dutifully lists any problems encountered while spidering your domain, and what might have caused them. In my case, there's a huge red flag:

A few restricted URLs is normal, but 817 is certainly a bit excessive. Had I paid attention to the tools Google provides, I would have noticed an abrupt change in crawl rate, and the spike in restricted URLs. But at the time, I only saw the decline in traffic, and didn't think to consult the webmasters tools.

The moral of the story:

First, search engines actually respect your robots.txt file. Second, it's a really bad idea to tell them to go away, because they will. And take your traffic with them.

Google, Yahoo, I've learned my lesson... Please relist me.

Some of you might remember that I tried to start a grassroots movement a year or so ago. After a brief moment of success, my campaign backfired spectacularly. Within a week or so I became the target of exactly the effect I was trying to cause to cause for my friend Jacob.

I found something interesting in my server logs tonight. It turns out that my long-forgotten blog post is now the top Microsoft Live Search result for starting a grassroots movement. Who would have thought?

I've been using Google's "Edit Search Results" since I blogged about it yesterday, and I have a couple further observations.

Read about Google's social search experiment after the jump.

I Googled something a minute ago and noticed that my SERP looked a little different:

In particular, check out those three grey icons. They appear next to every result on the page, but there isn't any noticeable indication why they're there. So I clicked one :)

The link turned green, which was fun. But the most exciting thing was that it moved to the top of the page—with a nice, slick animation. Apparently Google is letting me customize the results of my search.

Check out Google's editable search result experiment after the jump.

A while ago gThing noticed that utah's latest pyramid scheme MLM opportunity, goyin, had an unintentionally comical—and not very search engine friendly—website.

Well, it looks like they've outdone themselves. Their website now sports no real text at all—not even in meta tags. And they top it all off with a really annoying flash based splash page.

Somehow, in just over a thousand characters, they managed to earn zero SEO points, work in an Internet Explorer CSS hack, fail XHTML validation, make their site completely inaccessible to anyone with a screen reader, and annoy the crap out of me. Great job, guys.

the complete landing page source code after the jump.

apparently somebody from Kingston Upon Thames was searching for "cach the poo" and ended up at my blog. huh?

the things you learn from server logs.

Note: Myspace recently changed the way they make msplinks. This tool prob'ly doesn't work anymore...

a while back i mentioned that MySpace and msplinks do dirty things to links.

this bugs me a bit, so i've been trying to figure out ways to mess with 'em. i've got nothing good so far, but in the process i noticed that MySpace doesn't generate a new link for each address (like TinyURL, for example). instead they encode the address in the msplinks url. so an address can be encoded into a msplink even if MySpace has never seen that address before.

while i was playing with msplinks, i created a converter. put any url into it, and it'll spit out a msplinks address. i haven't figured out anything particularly devious to use it for yet, but if you come up with something let me know.

have fun kids :-)

msplinks address converter after the jump.

a couple of days ago my coworker jacob was looking through his server logs and noticed that several people found his site by searching for "jacob is so stupid". when we found out about it, he was near the bottom of the second page on google...

we (a.k.a. myself and other coworkers, not jacob) decided that we could do a bit of grassroots seo work for his site, and help his page rankings in the process. so if you have a minute, google "jacob is so stupid" (no quotes), and click on his blog. (his domain name is jacob.peargrove.com, the page is called "Jacob's Blog" or "Jacob?s Blog » 2007 » January").

as of the time of this post, his blog was showing up around the top of page two.

MySpace has been redirecting all external links in comments through msplinks.com for a couple of months. Msplinks turns any posted link into a redirect, supposedly in an attempt to cut down on spammer redirects and phishing.

From Tom:

Hey everybody, we just launched another program to stop dirty spammers from hasseling[sic] you. When you input a link in myspace it may be converted to a redirect link.... They still point to their original url, but let us easily turn off links to spam, phishing, or virus sites. booyah!

Until a week or so ago, only links in comments were redirected through msplinks. This fit well with the explanation given in Tom's message. Now external links in profiles are redirected through msplinks as well. The effects of this change are drastic and somewhat sinister.

Google acquired DoubleClick this month. which should be a good thing. but it looks like they acquired a conflict of interest too... they got quite the package deal