Spider Spider in the Web
We've recently written a spider for a client that walks their whole site and lists all connections between pages. This produces a brute listing of 29,000+ connections. We're doing this in preparation for a Drupal 7 (that's right d7!) transition. All links in the site content need to go to the correct new locations in the new site. For this to happen we need to correlate the new locations with the old locations so we can correctly rewire the new links.
The spider helps us prepare for the site pour of the old site into the new Drupal site. We have done similar work with the 25,000 programs in the CCTV archive in their transition to Drupal some years ago.
Drop a line with any spider inquiries. We write spiders in Perl based on some great code from Higher Order Perl.