URLRewriter and Googlebot Issues
This morning I made a frightening discovery — Googlebot has been unable to successfully read any of the pages on my new ASP.NET web sites!! UGH! I made this discovery by accident, but I’m sure glad I did. I was using the free Google Keyword Tool and decided to enter in the URL of a page on one of my sites. Now, the thing you need to know is that my sites are written with ASP.NET and I’m using an open source URL rewriter call URLRewriter.Net so I can have meaningful URLs without actually having long URLs.
So, I enter in one of my meaningful URLs into the Google keyword tool for it to analyze. Instead of it displaying keywords about ipods like I expected it listed a bunch of "error, asp.net error, vpscript error, …" keywords. That was not a good sign. I did a few more tests and had the same results. I did a few Google searches for URLRewriting and Googlebot and discovered that what I was seeing was not unique to me. Sure enough, there is a bug in ASP.NET and I was being bitten hard by it. The good news is that this explained why none of my pages were showing up in the Google index!
I did some more searches and research to find a solution. Luckily I was able to find a posting by someone that came up with a very quick and easy way to solve the problem. All I had to do was add an App_Browsers folder to my ASP.NET project, add a file call browserfile.browser with the follow contents:
<browsers>
<browser refID="Mozilla">
<capabilities>
<capability name="cookies" value="true" />
</capabilities>
</browser>
</browsers>
I did this, published my site, and re-ran my tests with the Google keyword tool. The result was a thing of beauty! Needless to say I immediately made the same changes to my other ASP.NET sites and got those re-published as well. Hopefully this means I will start to get some pages showing up in the Google index soon.
This was very frustrating for me to discover. I had felt pretty good about my previous decision to use ASP.NET instead of PHP; but after running into this issue I’m not so sure it was a good decision. The URL rewriting in Apache works without issue. If I encounter another issue like this with ASP.NET I think I’ll have no choice but to rewrite my sites using PHP….COME ON MICROSOFT!!!
P.S. - A more detailed explanation of the issue along with a more complex solution can be found here and on the URLRewriter forum.
