google

Seriously. It even does realtime updates.

Strange Google keywords that landed at my blog

• [searching for something dirty]
• message link generator

Unrelated Yahoo search results that ended up here

• 404 justin
• 404 music / justin
• acta "only one ipod"
• base structure
• best punchlines humor
• blog [searching for something dirty]
• cach a poo
• cach the poo
• clickable link generator
• [searching for something dirty]
• [searching for something dirty]
• googol justin
• how do you make fans
• how much is the music industry worth
• my space cool
• [searching for something dirty]
• [searching for something dirty]

A whole grip of Microsoft Live/MSN search terms which, in some crazy stretch of the imagination, might possibly relate to my blog

• "types of stupidity"
• +brain fard
• +dot your happy friend dot info
• +dot your space friend dot info
• +what is acquaintances
• add,message,comment links
• again or contact info justin
• alphebetize iphone menu
• cach a poo
• cach a poo 2
• cach the poo
• catch a poo
• catchthe poo
• chuck hileman blogger
• clickable link generator
• [searching for something dirty]
• [searching for something dirty]
• define musing
• difference between a single dot ("./myimage.jpg") and a double dot ("../myimage.jpg")
• dot your freind dot info
• dot your happy friend dot info
• dot your space friend dot info
• frankie leman
• google justin
• googol hot open girls
• hi my conputer info
• hileman beer
• [searching for something dirty]

And it appears that Bing might be carrying on the tradition

• google justin
• jastin for laught
• [searching for something dirty]
• "girl jeans"
• advanced healing bandaid
• arnold bennett
• beautified quotes
• bringo stop talking to machines
• cach a poo
• cartoon register groceries
• doophp
• dot unicode
• download high def trailers
• drupal webfm securing
• em,px,
• firefox really slowing down my internet
• google.justin
• gustaso.com
• how to get firefox not to use memory when us load pages
• how to save e-mail info on xp full install
• howto disable url search ie
• i think i have disabled number lock
• ie6 backspace key
• inspiration when sleepy

Context: Each list above consists of the unrelated search terms found in the top 50 keyword referrers from each search engine. For example, Google was wrong in about 4% of its top fifty search terms. Microsoft Live search failed in an astounding 76% of its tries.

For the curious: All the [searching for something dirty] placeholders were variations on the name of a certain website which will remain nameless.

I prob'ly don't need to tell you this, but don't ever ask search engines to delist your site. Ever. They actually do it.

A bit of background...

Last month I relaunched justin hileman dot info with a fresh new theme, and a couple of cool features. While I was developing the replacement site, I had a staging version which I didn't want Google to index. I used a handy little robots.txt file to keep the search engine spiders at bay:

User-agent: *
Disallow: /

This file did the trick.

The staging site wasn't indexed, exactly as advertised. Unfortunately when the time came to deploy, my über-restrictive robots.txt file overwrote the existing file and slipped into the live site.

Google, Yahoo, and friends dutifully ignored every page on my site. The majority of the damage was instantaneous. Most high traffic and high PR pages were unindexed within a few hours. My crawl rate dropped, the major search engines removed more pages every time they crawled my site.

My SERP traffic, understandably, tanked.

Yesterday my search engine referrals hit zero.

The sharp drop in SERP traffic on February 12th coincides with the first Google crawl with the new robots.txt. The second drop, around the 23rd was the result of Yahoo's reindex. In just a few days my site was completely unlisted from the major search engines.

As of the time of this post, a search for "justin hileman dot info", which should result in about 1500 pages on this domain, returns nothing.

How could this have been avoided?

Google Webmaster tools provides a great overview of your site. It dutifully lists any problems encountered while spidering your domain, and what might have caused them. In my case, there's a huge red flag:

A few restricted URLs is normal, but 817 is certainly a bit excessive. Had I paid attention to the tools Google provides, I would have noticed an abrupt change in crawl rate, and the spike in restricted URLs. But at the time, I only saw the decline in traffic, and didn't think to consult the webmasters tools.

The moral of the story:

First, search engines actually respect your robots.txt file. Second, it's a really bad idea to tell them to go away, because they will. And take your traffic with them.

Google, Yahoo, I've learned my lesson... Please relist me.

Google's messing with search engine results again, but this time it looks like they're trying to take on Wikipedia, not Digg. At least they're calling their editable search engine result experiment SearchWiki this time around. It still looks a lot like it did last time I reported on it (here and here). Check out a bunch of screenshots from this iteration here:

Things are a bit smoother this time. Moving results is a nice, polished animation. Overall, SearchWiki pretty rad. I can't wait for more community features to show up. Mebbe they'll even use these Google public profiles we all have but don't know about?

If you wanna give it a shot but don't see the fun little grey buttons next to your search results, try adding &swm=2 to the end of a Google search URL. (swm=2 was the same parameter they used in the last iteration. YMMV). If that doesn't work, try 0 or 1 or 2. Can't hurt, right? :)

Is it better to compete in an industry dominated by a single corporation, or a market full of inconsequential competitors? What if that single competitor is so well established that they're a now a verb? Maybe have an entire market named after themselves?

IBM, a name once synonymous with personal computers no longer has a PC division. In 2005 they sold it to a Chinese corporation. When I xerox a document, I usually use a copier made by Ricoh or Brother. I do use Band-Aid brand adhesive bandages, but only because they continue to innovate—Side note: if you haven't tried Band-Aid Advanced Healing, they will change your life. So does any search stand a chance against Google at this point? After all, you never "Live Search" or "Yahoo!" anything…

I guess my point is that at some point someone had the audacity to challenge the companies who defined the market. It takes guts to pick a fight with the biggest kid on the playground. But if you never fight the fight, you're guaranteed to lose.

Good luck, Cuil, you're going to need it.

I've been using Google's "Edit Search Results" since I blogged about it yesterday, and I have a couple further observations.

Read about Google's social search experiment after the jump.

I Googled something a minute ago and noticed that my SERP looked a little different:

In particular, check out those three grey icons. They appear next to every result on the page, but there isn't any noticeable indication why they're there. So I clicked one :)

The link turned green, which was fun. But the most exciting thing was that it moved to the top of the page—with a nice, slick animation. Apparently Google is letting me customize the results of my search.

Check out Google's editable search result experiment after the jump.

one of my favorite features of the better gmail firefox plugin is the "smart read button". when you select a message, it says either "mark read" if it's an unread message, or "mark unread" if you've already read it. or at least that's what it used to do. apparently the updates to gmail broke that feature. i'm a bit bummed, someone will come up with a fix soon...

another notable change is the urls: gmail is combining url fragments with javascript to fake out your bookmarks and browser navigation. for example, you can now go directly to your spam folder by typing http://mail.google.com/mail/#spam

they also added a filter assistant, kinda like the one better gmail had. when you're reading a message, click the dropdown where you used to add labels. there's an option for mute (which used to be available only via hotkey), to create an event, and to "filter messages like these". the filter assistant is pretty slick... almost like they had some inspiration :)

it seems like gmail's been stagnant for a while, but it looks like they're following up the new Gmail IMAP support with a few more features. hopefully google has more good stuff in store for us. then maybe we won't need Better Gmail anymore.

all of my reports have flatlined :-(

i want a refund.

a couple of days ago my coworker jacob was looking through his server logs and noticed that several people found his site by searching for "jacob is so stupid". when we found out about it, he was near the bottom of the second page on google...

we (a.k.a. myself and other coworkers, not jacob) decided that we could do a bit of grassroots seo work for his site, and help his page rankings in the process. so if you have a minute, google "jacob is so stupid" (no quotes), and click on his blog. (his domain name is jacob.peargrove.com, the page is called "Jacob's Blog" or "Jacob?s Blog » 2007 » January").

as of the time of this post, his blog was showing up around the top of page two.

MySpace has been redirecting all external links in comments through msplinks.com for a couple of months. Msplinks turns any posted link into a redirect, supposedly in an attempt to cut down on spammer redirects and phishing.

From Tom:

Hey everybody, we just launched another program to stop dirty spammers from hasseling[sic] you. When you input a link in myspace it may be converted to a redirect link.... They still point to their original url, but let us easily turn off links to spam, phishing, or virus sites. booyah!

Until a week or so ago, only links in comments were redirected through msplinks. This fit well with the explanation given in Tom's message. Now external links in profiles are redirected through msplinks as well. The effects of this change are drastic and somewhat sinister.

Google acquired DoubleClick this month. which should be a good thing. but it looks like they acquired a conflict of interest too... they got quite the package deal