Quite unambiguously, I rate the Google Adwords Search Query report a number out of 10. That’s right, a stunning #/10.

The Google Adwords search query report is designed to help you see the entire phrase your visitors searched for, when that search matches a ‘broad match’ or ‘phrase match’ - in other words, when what the user searched is not exactly what you were bidding on.

Awesome I exclaim with as much glee as PPC can make me muster - which is exactly not much… but really, this should be the talk of the town - this is the first made-for-the-public way to view all of these long-tail keywords. This represents the first simple end-user glimpse at the real truth of how people are reaching your site via those mystery-meat Broad and Phrase match terms.

I don’t fear telling you, I’m in love with the long-tail. It’s a torrid affair that dates back years, to the halcyon days of my youth when search engines were mere tools, not omnipresent overseers of the Internet. And in my heart of hearts, I truly believe that this romance would become infectious to the general web-site-running populace at large, if made just a little easier to understand, just a little easier to see.

This simplicity is what the promise of a no-frills Google report would seem to provide. Stuck doing the occasional PPC work on semi-virgin accounts (from an optimization standpoint), I would love any help in the gathering of insightful offerings for friends and clients.

But Google misses the mark by more than a little here. Remarkably they have chosen to exclude phrases with a low impression rate (what I affectionately refer to as… the long tail) in favour of saving money on processing power and server space, and perhaps bandwidth (I’m not going to source that, as it’s made up, (edit: oh hey maybe it’s not!) but I can’t think of any other reason - the data is obviously there).

Okay so perhaps my experience here is atypical. It is a rather niche site where, as you can see, the impressions over the course of a year are often in the hundreds if not less. For a larger advertiser there may be some more useful information to gather on the mid-tail - but I still expect the long of the long tail to cumulate into a good collective portion of conversions, and so a tool which would appear to be designed for peering AT THE LONG TAIL would be a little more useful if didn’t filter out low impression keywords. Just my take.

Okay rant over, here are some ways you can pull the same data while Google upgrades to 486’s:

> Look at your server logs. Better yet, tag your links in adwords (with Google Analytics tags, for example) and write a script that parses your server logs and couples the tagging with the referrer phrase - this offers a little more information at least. If you’re unfamiliar with server logs, you might first want to check with your hosting provider to see if they give you access to your ‘raw server logs’ - then have a read through this U. of Michigan primer on the topic.

> Hack Google Analytics: Our friendly neighbourhood ROI Revolution handed along this fancy-dancy little chunk of code a few months back. If you use Google Analytics, this is the easiest, cheapest and most useful solution (not to mention reveling in the irony of solving the problem with google’s own tools via some custom code is supremely satisfying).

Anybody got a third?

Let me tell you a little story.I was going to pick up a friend at the Airport in Montreal and he called me to say his flight was going to be delayed. “No problem I said, I’ll check your updated arrival in real time and come and get you when you land!”

So off I went to Google, to look for the official Montreal airport website which has real-time updates on flight departures and arrivals. I tried 10-15 different searches, here are some examples:

Montreal Airport
Pierre Elliot Trudeau Airport Montreal
Montreal Airport Flight Arrivals
Etc.

The first 100 results for all my searches did not bring the official site www.admtl.com ! However, I was able to find hundreds of Hotels, Limousine services, Google Map results of businesses around the area, Youtube videos about the airport, airline ticket purchase services, and hey… who would have thought: Wikipedia’s entry about the Airport.

Now you can sit there and tell me that it is the airport’s fault for not having “Montreal Airport” on the title tag, and not optimizing, blah, blah, blah. But the reality is that Google’s job is to crawl, index and present relevant results. They have the responsibility of ordering all of the chaotic content that appears on the internet each day to help users sift through it in an organized manner. That is their service. If they fail to do this, they becomes useless, people will no longer waste their time using Google.

Yahoo! gave me the same horrible results. What to do!!??? Surprisingly Live.com showed the site in the 1st page. Not the 1st result as that is always reserved for Wikipedia, but at least it was listed on the first page.

Why did Live.com get an appropriate result if the word Airport does not appear as text anywhere on the page? I looked at the results snippet for Live.com:

Sounds a lot like a Dmoz entry huh? Well that’s because it is. When there is no Meta Description tag Live.com does what Google used to do: Grab a DMOZ description.

You can hate DMOZ all you want, but it sure helped Live.com rank the appropriate relevant site.  Strange how such an old school, primitive and manual method helped Live.com get an appropriate result even though the site contains no mention of “airport” whatsoever.

The reality is this: White hat/Government and Official sites are not always optimized. In fact many companies and organizations don’t even know SEO exists. However, BlackHat Spammers do, and this puts them in a considerable advantage against “white hat, relevant content” websites:

1. Black Hats know what to do to rank. They have tested so many sites, gotten so many sites de-indexed and penalized that they are able to tell just how far they can push Google.
2. Black Hats develop automated methods of creating thousands of websites in a short period of time. These websites are created by hundreds or thousands of different Blackhats, and will all have millions of varying footprints if any at all.

So how do you algorithmically remove these millions of results appearing everyday? How do you do it manually? Even combining the two as Google does, has limitations.

Imagine a librarian trying to organize millions of books by category, author, title, etc. And each day hundreds of trucks come in and dump more books on top of the few which were already organized. I don’t envy Google nor Yahoo! For what they have to accomplish to stay relevant as an industry, but no matter how many Phd’s you hire, massive volume always prevails.

Black Hat Spammers will not be stopped from producing new content anytime soon, and Google does not seem to have figured out a permanent way to remove these sites from their index. It seems that the Google index no longer belongs to Google.

The ‘progress’ Google is making in its algo’s don’t seem to be reflected in the actual SERPs – a shift in paradigms seems to be an inevitable necessity, but my question to you is, does it have to be a shift to a new paradigm, or would a shift to an old one be more effective?