Select a tag to browse associated projects and drill deeper into the tag cloud.
ht://Dig
Analyzed over 2 years ago
The ht://Dig system is a complete WWW indexing and searching system for a domain or intranet. This system is not meant to replace the need for internet-wide search systems like Lycos, Infoseek, Google, and AltaVista. Instead, it is meant to cover the search needs for a single company, campus, or
Licenses:
gpl
YaCy
Analyzed about 2 months ago
YaCy is a P2P search engine for the WWW including a crawler and HTTP proxy
Grub
Analyzed over 1 year ago
Grub Next Generation is distributed web crawling system (clients/servers) which helps to build and maintain free (as in freedom) index of the Web. At this moment we have very simple search engine (as a proof of concept) too.
Licenses:
BSD-3-Clause, gpl3_or_l...
Apache ManifoldCF
Claimed by Apache Software Foundation Analyzed 6 days ago
ManifoldCF is an effort to provide an open source framework for connecting source content repositories like Microsoft Sharepoint and EMC Documentum, to target repositories or indexes, such as Apache Solr. ManifoldCF also defines a security model for target repositories that permits them to enforce source-repository security policies.
Licenses:
apache_2
crawler4j
C
Analyzed 2 days ago
Crawler4j is an open source Java Crawler which provides a simple interface for crawling the web. Using it, you can setup a multi-threaded web crawler in 5 minutes!
Sample UsageFirst, you need to create a crawler class that extends WebCrawler. This class decides which URLs should be crawled and
Licenses:
apache_2
OpenSearchServer
Analyzed over 1 year ago
OpenSearchServer is a powerful, enterprise-class, search engine program. With the web user interface, the crawlers (web, file, database, ...) and its REST API you will be able to integrate quickly and easily advanced full-text search capabilities in your application. OpenSearchServer runs on Windows
Serialist
Analyzed over 2 years ago
Serialist crawls serial stories on the web (such as webcomics), and provides a web interface for users to navigate these serials, mark where they left off, and find out when new pages exist.
Mostly written in language not available
Licenses:
AGPL3
Ex-Crawler
Analyzed over 2 years ago
Ex-Crawler Project is divided into three subprojects:
The main part is the Ex-Crawler daemon server, a highly configurable, flexible (Web-) crawler written in Java. It comes with it's own socket server, where you can manage the server, own usermanagement, distributed grid / volunteer computing
Mostly written in language not available
Licenses:
gpl3_or_l...
LinkChecker
Analyzed 6 months ago
Check websites and HTML documents for broken links.
* recursive and multithreaded checking
* output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats
* HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support
* restriction of link
Murloc
Analyzed over 2 years ago
Murloc is a website parser framework written in PHP. It features a plugin system that allows quick and easy development of new parsers. It provides several functions that are more or less commonly needed by parsers.
Murloc comes with a bunch of handy plugins.
As it is written in PHP, it should
Mostly written in language not available
Licenses:
bsd_2clau...