spider.py 0.5 : Python Package Index

spider.py 0.5 : Python Package Index

spider.py 0.5

Multithreaded crawling, reporting, and mirroring for Web and FTP

This module provides multithreaded crawling, reporting, and mirroring for Web
and FTP in one convenient library. Crawling depth, maximum number of URLs to
crawl, and maximum number of threads are user-configurable. Reports can be
generated on external URLS, internal redirects to outside URLs, unparsable HTML,
non-HTTP/FTP URLs, and broken links.