Hunting the Wiley 404

I hate finding 404's in my web logs. There was a time when my logs were full of them. This was from carelessness and forgetfulness on my part as well as from simple typing mistakes by both me and others. I've cleaned up most of my own mistakes and done links or redirects for the mistakes of others. This has cut my error log down to about a third of 1% - that is, there's one entry in my error log for every 300 entries in the access log. Most of those are script-kiddie attacks: people looking for exploitable php or cgi scripts.

The tool that has been most helpful to me in cleaning up the real errors has been the Web Crawl report at Google's Webmaster Tools. They have offered the "not found" report for some time, but recently they added info about the source page - the page that led them to the 404 on your site. This makes all the difference in the world: if the offending page is on your own site, you can immediately fix it. If it's someone else's page, you can decide whether to put in a link or a redirect to handle it.

While the relative importance of inbound links for SEO has decreased, you certainly don't want to throw away whatever value a link could give you. As Matt Cutts said at Free links to your site:

Why would you care about this? The simple reason is that if someone is linking to a non-existent page on your site, it can be a bad experience for users (not to mention that you might not be getting credit for that link with search engines unless you're doing extra work). Some of the easiest links you'll ever get are when people tried to link to you and just messed up.

Absolutely true.