PennCrawler is a distributed web crawler and search engine designed to be very flexible. It is scalable to the size of the Internet. And it offers a servlet based interface to query the search engine.
These details are provided for information only. No information here is legal advice and should not be used as such.