WebSpider

aka Web Crawler

a subtype of Web Robot which focuses on finding new URI-s to grab by parsing the HREF-s on each page it consumers.

Mainly associated with Search Engine code.

book of spidering hacks ISBN:0-596-00577-6 http://www.oreilly.com/catalog/spiderhks/chapter/ (rather Perl-focused, it sounds)