[Organizers] Fwd: Query: Tools to search for stale links (was Re: is NEFFA LinkFest still active)

I have a question for any of you who are involved with website administration (e.g., for your local dance organization): Do anyone know of good tools that will automatically go through a site searching for stale links? I've been prompted to make this query in part by the recent messages about the NEFFA LinkFest, which is apparently no longer maintained because it got to be too much work relative to the perceived benefit (particularly relative to perceived benefit for the volunteer maintainer(s)). However, I've noticed that many local dance groups' websites include lists of external links, and that while these lists are typically much smaller than the LinkFest (containing perhaps a dozen to a hundred links vs. almost 3000 in the LinkFest), they also often are not assiduously maintained and so often include at least a few links that no longer point to the expected content. I recognize that the automatic detection of stale links may not be a trivial problem, since there can be a variety of different symptoms. I'e just gone looking through lists of links on several different sites. Among the things I've found are: * Links that fail with Error 404 (Not Found) * Links that fail with Error 403 (Forbidden: You don't have permission to access ...) * Links that fail because the browser cant find the server. * Links that get to the site that apparently used to contain the desired page but now show a message like "We're sorry, but we were unable to locate the page you requested." (Perhaps attempts to follow these links actually produce a 404 code even if the text displayed to the user doesn't include the string "404".) * Links that go to sites offering to sell you the (expired) domain name that included the target page. * Links to pages that appear to have been taken over by new owners and no longer display the original content. Among other things, I've found pages full of text in Chinese or Cyrillic characters; pages that used to have dance information and seem to have been taken over by real estate agents or financial service organizations; and pages that say "WARNING!! THIS SITE CONTAINS ADULT MATERIALS ..." (I did not click on the "Continue" button after the warning to see whether it got me to traditional-dance-related cotent). * Links to pages that admit to being no longer maintained. * Links to pages that in turn offer a (possibly good) link to a new page with the desired content. * Links that go to what seem to be the original target pages, but that also seem not to have been updated in several years. (For example a site might refer to "upcoming" event that are now several years in the past. One might then wonder whether there's a new, actively maintained, site somewhere with similar, but current, information.) If someone creates a link the points to a recent blog entry, someone who follows the link several years later might get to now-most-recent page of the blog and only be able to reach the originally referenced content by clicking something like "older posts" a large--and unknown--number of times. Similarly, links to online newspaper or magazine content might now point to pages that are evidently full of newer content. The old content might or might not be available somewhere, but there may be no visible advice about how to find it. When people migrate their websites, it seems to be common that, if links into the old site get redirected at all, they end up redirecting to some place like the home page of of the new site instead of to the exact page (if one exists) corresponding to the old target of the link. Despite these comments about how links can go stale in various ways that aren't immediately obvious, it seems to me that stale link detection would be a useful feature for a wide variety of website administrators--and not just for dance organizers. So perhaps someone somewhere has put some effort into doing a good job of it. I'd be interested if anyone knows of examples. Thanks. --Jim _______________________________________________ Organizers mailing list Organizers(a)lists.sharedweight.net http://lists.sharedweight.net/listinfo.cgi/organizers-sharedweight.net