Apache Solr

Adoption of Apache Solr is accelerating. Being accessible though HTTP makes it possible for Solr (a Java webapp) to be used with any language. All you need is support for making HTTP calls and parsing one of the many available formats such as XML or ... [More] JSON.DrupalDrupal is one of the most popular CMS available as open source. It is written in PHP and boasts of a huge user and developer base. Recently, the Drupal community has integrated Apache Solr into Drupal for vertical search. The integration is available as a Drupal module at http://drupal.org/project/apachesolr. There are some excellent tutorials available on how to get started with using this as well as a hosted solution by Acquia.RubyRuby integration has been present in Solr since a long time. There is a module called solr-ruby as well as acts_as_solr. Solr even has a ruby response writer which outputs search results serialized in ruby. Blacklight is an open source project I know that uses Solr and is built in Ruby. Today, I came to know about SunSpot - A Solr powered search engine for Ruby. More details at this article in LinuxMag.PythonSolr has a python response writer as well as many clients. See http://wiki.apache.org/solr/SolPython for details. Reddit is one site that uses Solr with a python front-end application. There is also HayStack for Django which can use Solr among other engines such as Xapian and Whoosh.Solr 1.4 is nearing release with a number of features and performance improvements. On the other hand, Lucene is getting ready for near real-time search as well. Things are getting interesting in the Solr world! [Less]

One of the many performance improvements in the upcoming Solr 1.4 release involves improved filtering performance. Solr 1.4 filters are both faster (anywhere from 30% to 80% faster to calculate intersections, depending on configuration), take less ... [More] memory (40% smaller), and are more efficiently applied to the query during a search. In previous Solr releases, filters […] [Less]

One of the many performance improvements in the upcoming Solr 1.4 release involves improved filtering performance. Solr 1.4 filters are both faster (anywhere from 30% to 80% faster to calculate intersections, depending on configuration), take less ... [More] memory (40% smaller), and are more efficiently applied to the query during a search. In previous Solr releases, filters [...] [Less]

One of the many performance improvements in the upcoming Solr 1.4 release involves improved filtering performance. Solr 1.4 filters are both faster (anywhere from 30% to 80% faster to calculate intersections, depending on configuration), take less ... [More] memory (40% smaller), and are more efficiently applied to the query during a search. In previous Solr releases, filters […] [Less]

There is a large amount of work being done in Lucene 2.9, in which a large portion is related to adding support for near real-time search.To put it very simply, search engines transfer a lot of work from query-time to index-time. The reason this is ... [More] done, is to speed up queries at the cost of adding documents slower. Until now, Lucene based systems have had problems with dealing with scenarios in which the searchers need to see the changes instantly (think Twitter Search). There exist a variety of tricks and techniques to acheive this even now. However, near real-time search support in Lucene itself is a boon to all those people who have been building and managing such systems because the grunt work will be done by Lucene itself.This is still under development and will probably take a few more months to mature. Solr will benefit from it as well but before that can happen, a lot of work will be needed under the hood particularly in the way Solr handles its caching.Michael McCandless has summarized the current state of Lucene trunk in this email on java-dev mailing list. In fact, there is so much activity that, at times, it becomes very difficult to follow all the excellent discussions that go on. There are some very talented people on that forum and it is a lot of learning for a guy like me, who started with Solr and is still trying to find his way in the Lucene code base.Lucene 2.9 will bring huge improvements and I'm looking forward to working with other Solr developers to integrate them with Solr. [Less]

There is a large amount of work being done in Lucene 2.9, in which a large portion is related to adding support for near real-time search.To put it very simply, search engines transfer a lot of work from query-time to index-time. The reason this is ... [More] done, is to speed up queries at the cost of adding documents slower. Until now, Lucene based systems have had problems with dealing with scenarios in which the searchers need to see the changes instantly (think Twitter Search). There exist a variety of tricks and techniques to acheive this even now. However, near real-time search support in Lucene itself is a boon to all those people who have been building and managing such systems because the grunt work will be done by Lucene itself.This is still under development and will probably take a few more months to mature. Solr will benefit from it as well but before that can happen, a lot of work will be needed under the hood particularly in the way Solr handles its caching.Michael McCandless has summarized the current state of Lucene trunk in this email on java-dev mailing list. In fact, there is so much activity that, at times, it becomes very difficult to follow all the excellent discussions that go on. There are some very talented people on that forum and it is a lot of learning for a guy like me, who started with Solr and is still trying to find his way in the Lucene code base.Lucene 2.9 will bring huge improvements and I’m looking forward to working with other Solr developers to integrate them with Solr. [Less]

Google has announced support for building Java applications on the App Engine platform. This is great news for new App Engine developers and especially for those Java developers who had to learn Python to use App Engine.I created a project for App ... [More] Engine using Maven for builds. These were the steps I needed to follow:1. Publish the App Engine libraries to the local Maven repository. Goto the app-engine-java-sdk directory (where app-engine sdk is installed) and execute the following commands:mvn install:install-file -Dfile=lib/appengine-tools-api.jar -DgroupId=com.google -DartifactId=appengine-tools -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=truemvn install:install-file -Dfile=lib/shared/appengine-local-runtime-shared.jar -DgroupId=com.google -DartifactId=appengine-local-runtime-shared -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=truemvn install:install-file -Dfile=lib/user/appengine-api-1.0-sdk-1.2.0.jar -DgroupId=com.google -DartifactId=appengine-sdk-1.2.0-api -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=truemvn install:install-file -Dfile=lib/user/orm/datanucleus-appengine-1.0.0.final.jar -DgroupId=org.datanucleus -DartifactId=datanucleus-appengine -Dversion=1.0.0.final -Dpackaging=jar -DgeneratePom=true2. Create a maven pom file. This is the one that I used:<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0http://maven.apache.org/maven-4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.shalin</groupId> <artifactId>test</artifactId> <packaging>war</packaging> <version>1.0</version> <name>Test</name> <url>http://shalinsays.blogspot.com</url> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>com.google</groupId> <artifactId>appengine-tools</artifactId> <version>1.2.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.google</groupId> <artifactId>appengine-local-runtime-shared</artifactId> <version>1.2.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.google</groupId> <artifactId>appengine-sdk-1.2.0-api</artifactId> <version>1.2.0</version> <scope>compile</scope> </dependency> <dependency> <artifactId>standard</artifactId> <groupId>taglibs</groupId> <version>1.1.2</version> <type>jar</type> <scope>runtime</scope> </dependency> <dependency> <artifactId>jstl</artifactId> <groupId>javax.servlet</groupId> <version>1.1.2</version> <type>jar</type> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.geronimo.specs</groupId> <artifactId>geronimo-el_1.0_spec</artifactId> <version>1.0.1</version> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.geronimo.specs</groupId> <artifactId>geronimo-jsp_2.1_spec</artifactId> <version>1.0.1</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.geronimo.specs</groupId> <artifactId>geronimo-servlet_2.5_spec</artifactId> <version>1.2</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.geronimo.specs</groupId> <artifactId>geronimo-jpa_3.0_spec</artifactId> <version>1.1.1</version> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.geronimo.specs</groupId> <artifactId>geronimo-jta_1.1_spec</artifactId> <version>1.1.1</version> <scope>compile</scope> </dependency> <dependency> <groupId>org.datanucleus</groupId> <artifactId>datanucleus-appengine</artifactId> <version>1.0.0.final</version> <scope>compile</scope> </dependency> </dependencies> <build> <finalName>test</finalName> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.5</source> <target>1.5</target> </configuration> </plugin> </plugins> </build></project>3. Create the standard maven directory structure and add the pom.xml in the same directory as the src directory.You're done!I tested this with a simple servlet based application and it worked fine. I did not test the JPA/JDO integration so it might be a little rough around the edges. But it should work for the most part. Note, App Engine supports Java 6. If you want to use Java 6, you can change the "source" and "target" in the build section to 1.6 instead of 1.5 [Less]

Google has announced support for building Java applications on the App Engine platform. This is great news for new App Engine developers and especially for those Java developers who had to learn Python to use App Engine.I created a project for App ... [More] Engine using Maven for builds. These were the steps I needed to follow:1. Publish the App Engine libraries to the local Maven repository. Goto the app-engine-java-sdk directory (where app-engine sdk is installed) and execute the following commands:mvn install:install-file -Dfile=lib/appengine-tools-api.jar -DgroupId=com.google -DartifactId=appengine-tools -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=truemvn install:install-file -Dfile=lib/shared/appengine-local-runtime-shared.jar -DgroupId=com.google -DartifactId=appengine-local-runtime-shared -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=truemvn install:install-file -Dfile=lib/user/appengine-api-1.0-sdk-1.2.0.jar -DgroupId=com.google -DartifactId=appengine-sdk-1.2.0-api -Dversion=1.2.0 -Dpackaging=jar -DgeneratePom=truemvn install:install-file -Dfile=lib/user/orm/datanucleus-appengine-1.0.0.final.jar -DgroupId=org.datanucleus -DartifactId=datanucleus-appengine -Dversion=1.0.0.final -Dpackaging=jar -DgeneratePom=true 2. Create a maven pom file. This is the one that I used:xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0http://maven.apache.org/maven-4_0_0.xsd"> 4.0.0 com.shalin test war 1.0 Test http://shalinsays.blogspot.com junit junit 3.8.1 test com.google appengine-tools 1.2.0 provided com.google appengine-local-runtime-shared 1.2.0 provided com.google appengine-sdk-1.2.0-api 1.2.0 compile standard taglibs 1.1.2 jar runtime jstl javax.servlet 1.1.2 jar compile org.apache.geronimo.specs geronimo-el_1.0_spec 1.0.1 compile org.apache.geronimo.specs geronimo-jsp_2.1_spec 1.0.1 provided org.apache.geronimo.specs geronimo-servlet_2.5_spec 1.2 provided org.apache.geronimo.specs geronimo-jpa_3.0_spec 1.1.1 compile org.apache.geronimo.specs geronimo-jta_1.1_spec 1.1.1 compile org.datanucleus datanucleus-appengine 1.0.0.final compile test maven-compiler-plugin 1.5 1.5 3. Create the standard maven directory structure and add the pom.xml in the same directory as the src directory.You’re done!I tested this with a simple servlet based application and it worked fine. I did not test the JPA/JDO integration so it might be a little rough around the edges. But it should work for the most part. Note, App Engine supports Java 6. If you want to use Java 6, you can change the “source” and “target” in the build section to 1.6 instead of 1.5 [Less]