Select a tag to browse associated projects and drill deeper into the tag cloud.
ht://Dig
Analyzed about 3 hours ago
The ht://Dig system is a complete WWW indexing and searching system for a domain or intranet. This system is not meant to replace the need for internet-wide search systems like Lycos, Infoseek, Google, and AltaVista. Instead, it is meant to cover the search needs for a single company, campus, or
Licenses:
GPL-2.0+
YaCy
Analyzed about 5 hours ago
YaCy is a P2P search engine for the WWW including a crawler and HTTP proxy
Licenses:
GPL-2.0+
Grub
Analyzed 10 days ago
Grub Next Generation is distributed web crawling system (clients/servers) which helps to build and maintain free (as in freedom) index of the Web. At this moment we have very simple search engine (as a proof of concept) too.
Licenses:
BSD-3-Clause, GPL-3.0+
Apache ManifoldCF
Claimed by Apache Software Foundation Analyzed 4 months ago
ManifoldCF is an effort to provide an open source framework for connecting source content repositories like Microsoft Sharepoint and EMC Documentum, to target repositories or indexes, such as Apache Solr. ManifoldCF also defines a security model for target repositories that permits them to enforce source-repository security policies.
Mostly written in language not available
Licenses:
Apache-2.0
OpenSearchServer
Analyzed 15 days ago
OpenSearchServer is a powerful, enterprise-class, search engine program. With the web user interface, the crawlers (web, file, database, ...) and its REST API you will be able to integrate quickly and easily advanced full-text search capabilities in your application. OpenSearchServer runs on Windows
Licenses:
GPL-3.0
crawler4j
C
Analyzed about 10 hours ago
Crawler4j is an open source Java Crawler which provides a simple interface for crawling the web. Using it, you can setup a multi-threaded web crawler in 5 minutes!
Sample UsageFirst, you need to create a crawler class that extends WebCrawler. This class decides which URLs should be crawled and
Licenses:
Apache-2.0
Serialist
Analyzed over 6 years ago
Serialist crawls serial stories on the web (such as webcomics), and provides a web interface for users to navigate these serials, mark where they left off, and find out when new pages exist.
Licenses:
AGPL-3.0
Murloc
Analyzed 4 months ago
Murloc is a website parser framework written in PHP. It features a plugin system that allows quick and easy development of new parsers. It provides several functions that are more or less commonly needed by parsers.
Murloc comes with a bunch of handy plugins.
As it is written in PHP, it should
Mostly written in language not available
Licenses:
BSD-2-Clause
Ex-Crawler
Analyzed 4 months ago
Ex-Crawler Project is divided into three subprojects:
The main part is the Ex-Crawler daemon server, a highly configurable, flexible (Web-) crawler written in Java. It comes with it's own socket server, where you can manage the server, own usermanagement, distributed grid / volunteer computing
Mostly written in language not available
Licenses:
GPL-3.0+
Eclipse SMILA
Claimed by Eclipse Foundation Analyzed over 3 years ago
The amount and diversity of information is growing exponentially, mainly in the area of unstructured data, like emails, text files, blogs, images etc. Poor data accessibility, user rights integration and the lack of semantic meta data are constraining factors for building next generation enterprise
Licenses:
EPL-1.0