search engines

Apparently nobody cares what you had for lunch. Nobody wants to hear about your bodily functions (or so I hear). So what else is Twitter good for?

Keeping your finger on the pulse of awesome, of course!

And the best tool for the job is Twitter Search. Check out some of my favorite Twitter searches below. Make up some of your own and share. Enjoy!

Note: Because some people on Twitter use offensive language, these searches occasionally return content that gets apps blocked from the iTunes App Store.

I prob'ly don't need to tell you this, but don't ever ask search engines to delist your site. Ever. They actually do it.

A bit of background...

Last month I relaunched justin hileman dot info with a fresh new theme, and a couple of cool features. While I was developing the replacement site, I had a staging version which I didn't want Google to index. I used a handy little robots.txt file to keep the search engine spiders at bay:

User-agent: *
Disallow: /

This file did the trick.

The staging site wasn't indexed, exactly as advertised. Unfortunately when the time came to deploy, my über-restrictive robots.txt file overwrote the existing file and slipped into the live site.

Google, Yahoo, and friends dutifully ignored every page on my site. The majority of the damage was instantaneous. Most high traffic and high PR pages were unindexed within a few hours. My crawl rate dropped, the major search engines removed more pages every time they crawled my site.

My SERP traffic, understandably, tanked.

Yesterday my search engine referrals hit zero.

The sharp drop in SERP traffic on February 12th coincides with the first Google crawl with the new robots.txt. The second drop, around the 23rd was the result of Yahoo's reindex. In just a few days my site was completely unlisted from the major search engines.

As of the time of this post, a search for "justin hileman dot info", which should result in about 1500 pages on this domain, returns nothing.

How could this have been avoided?

Google Webmaster tools provides a great overview of your site. It dutifully lists any problems encountered while spidering your domain, and what might have caused them. In my case, there's a huge red flag:

A few restricted URLs is normal, but 817 is certainly a bit excessive. Had I paid attention to the tools Google provides, I would have noticed an abrupt change in crawl rate, and the spike in restricted URLs. But at the time, I only saw the decline in traffic, and didn't think to consult the webmasters tools.

The moral of the story:

First, search engines actually respect your robots.txt file. Second, it's a really bad idea to tell them to go away, because they will. And take your traffic with them.

Google, Yahoo, I've learned my lesson... Please relist me.

Some of you might remember that I tried to start a grassroots movement a year or so ago. After a brief moment of success, my campaign backfired spectacularly. Within a week or so I became the target of exactly the effect I was trying to cause to cause for my friend Jacob.

I found something interesting in my server logs tonight. It turns out that my long-forgotten blog post is now the top Microsoft Live Search result for starting a grassroots movement. Who would have thought?

Is it better to compete in an industry dominated by a single corporation, or a market full of inconsequential competitors? What if that single competitor is so well established that they're a now a verb? Maybe have an entire market named after themselves?

IBM, a name once synonymous with personal computers no longer has a PC division. In 2005 they sold it to a Chinese corporation. When I xerox a document, I usually use a copier made by Ricoh or Brother. I do use Band-Aid brand adhesive bandages, but only because they continue to innovate—Side note: if you haven't tried Band-Aid Advanced Healing, they will change your life. So does any search stand a chance against Google at this point? After all, you never "Live Search" or "Yahoo!" anything…

I guess my point is that at some point someone had the audacity to challenge the companies who defined the market. It takes guts to pick a fight with the biggest kid on the playground. But if you never fight the fight, you're guaranteed to lose.

Good luck, Cuil, you're going to need it.