Apache Solr

What do BitTorrent, Orkut, TopCoder, del.icio.us, Wikipedia and hundreds of products/services have in common? Seasoned webbies may recognize the pattern and say that all of them are examples of Web 2.0 (which...ahem...yes they are). But apart from ... [More] this fact, the one underlying theme behind the diverse fabrics of all these products/services is that they are highly community driven.The new poster-boys of today's internet have utilized the power of the community channeling the huge cluster of the internet users themselves to improve the services offered. The whole idea is pretty simple and has been in use since a long time in the hospitality industry. It's called Self-Service. Their are no waiters and the customer services himself. He goes to the ordering counter, places the order, pre-pays the money and then fetches his food from the service counter to his table himself. The cost of the waitors and other such staff are eliminated and this arrangement makes the customer pay more attention to the clean-up. You always place the tray at the designated spot while going out, don't you?Some clever people applied the same concept with some modifications to the web and Wiki was born. People collaborate online and add, delete, modify, re-organize content to the website. No one person does it, no content writer is hired and the websites evolves instead of being made. Can you imagine what would happen if we apply this model back to our McDonalds? It's akin to McDonalds allowing you to use their tools to make your own burger for yourself and others. Ofcourse one can change the ingredients list and include poison in it. But the underlying idea is that atleast one good samaritan will take notice and change it back to the original copy. The idea says that there are more good people in the world than bad people hence the overall progress will be positive. What Britannica took so many decades to collect and classify, Wikipedia did in a few years. You don't believe me? Check out Wikipedia, Algorithmist and our newest entrant in this field, the MSDN Wiki. By the way, did you know that Wikipedia is one of the Top 100 most popular websites in the world?As software became complex, new delivery platforms had to be evolved. Suppose you have a product which you give away for free download. It gets slashdotted (or digg'ed) and soon swarms of users come your way and your site goes down. What can you do about that? Buy new servers, take up dedicated hosting, larger bandwidths and scale up. And all that only for the initial excitement about your product. What happens when it dies? You're stuck with your new servers, your expensive hosting plan and all the bandwidth in the world that you'll never need. Enter the community...BitTorrent & Jidgo. The idea...give out torrents for the file that you intend to distribute. The users connect to each other to send and recieve your software and a central tracking server manages these peer-to-peer interactions. Send and recieve is simultaneous to utilize network bandwidth and the protocol is designed to perform better as the number of users interested in the software increases, unlike traditional methods.Ever seen Rent-A-Coder or e-Lance? They used the community to bring together the people who needed work and the people who wanted some work done. It was successful but the model had an inherent deficiency. It was hard to scale up. It was unsuitable for larger projects as there was no control over the developer and no fixed model of interaction. TopCoder took this further. They developed a model of interaction between the company and the individual. They presented the interaction in the form of a competition to build out the best possible effort from the individual. To attract the best possible talent, they gave away huge amounts of prize money for algorithmic programming contests. So they gave this proposition to the companies needing software...At TopCoder, the individuals from a world-class community compete to design, develop and review software for you. Other revenue streams for TopCoder were sponsorships from companies for their tournaments and employment services.Traditional search methods used algorithms to identify the relevance of documents for certain keywords. They can't be accurate all the time so the ideal would be to employ people to classify the information. But this is limited because the web is too big. Enter Tagging, the community joins hands to tag their favorite websites and the model evolves to include all relevant website. If a hundred people say that this particular site is great for cricket information, then it has to be so. Folksonomy was born. Examples are del.icio.us, ma.gnolia for general purpose searching and Flickr for photos.So the conclusion is that for all these websites, the service automatically improves as more and more people use it. Orkut, TopCoder, Bittorrent, Wikipedia wouldn't exist if the community didn't exist. They use the community to harness collective intelligence (Wikipedia, del.icio.us, Flickr, Cloudmark), collective bandwidth (BitTorrent, Jidgo) and collective development (TopCoder, Sourceforge) to grow themselves.So what will your website do? Does it stay the same way if five people use it or if five thousand people use it everyday? If yes, then it should not. That's the way to survive in the Web 2.0, just think about it. [Less]

As I mentioned in my last post, I downloaded the ISO image for the Ubuntu 6.06 "Dapper Drake" Release Candidate. Maybe I haven't mentioned it ever but I run Linux as my primary operating system since the last six months. And the Linux distro of ... [More] choice has been...you guess it...Ubuntu Linux.Warning: The following few paragraphs are meant for techy, nerdy, geeky, FSF breathing, Linux speaking (life is mostly about technology) people.[begin geek stuff]The website said that breezy (that's the name of the older version) users can upgrade easily by first making sure that they have updated to the latest "update-manager" utility through Synaptic. Then the only thing you need to do is issue the command:gksudo "update-manager -d"Now begins the tale. I did what they told me. I applied all the latest security updates that had been pending for a while (around 90 megs of patches!). Then I issued the required command. Behold! It showed me that a new version of ubuntu is available and a button to upgrade, which I promptly clicked.The installation process started and it downloaded some files and then a message saying "Cannot upgrade..." popped up. Another one told me "An error occurred while calculating dependencies...". So all this came to halt. Ok, maybe the only way now is to download the ISO and upgrade using that. It took me around 16 hours to download the ISO. Now without a CD-writer, how am I supposed to install it. I didn't want to wait to get it fixed. So I decided to take a bold step. An experiment to install from the ISO without burning to a CDROM. Reminded me of the good olden days, when all I did was to experiment around with my system...I mounted the ISO image on my filesystem. Until that time, I didn't know how to do that...for those who still don't, here is the command:mount -o loop -t iso9660 /opt/somename.iso /mnt/somefolderThen I added the following line to my /etc/apt/sources.lst:deb file:///mnt/dapper/ dapper main restrictedand ran apt-get update. Then apt-get upgrade started the upgrade process. When I had got that error from the update-manager, I should've known that something must be wrong and if something goes wrong during this process, it's going to trash my working, lovely, 6-month old operating system. Now as it turned out, it did :(It hanged while trying to upgrade the pcmcia package. A reboot didn't help either. I tried trying to boot from the ISO image from the GRUB prompt. No luck. Then I used the breezy install cd to boot and tried using the bootfrom: parameter. Again no luck. Tried to mount the ISO in between various phases of the breezy install process but that couldn't be done because a lot of clashing was taking place between the differing versions in the breezy install CD and the dapper install ISO image.Finally, I was left with only one option. I did a default install from the breezy install CD, and then upgraded all packages from within synaptic after adding the ISO into the sources.lst file. That worked somehow and here I am using a brand new ubuntu 6.06 release candidate.It's pretty cool with the new art-work and a more polished theme. Some menu items have been moved around to my discomfort. For example: the take screenshot menu item has been moved to accessories and the network tools launcher has been moved from the Accessories->System tools menu to the System->Administration menu.And now I am left with downloading and configuring softwares like Anjuta, Eclipse, Netbeans, Sun JDK (eesh...I almost forgot about it) and ofcourse vlc, mplayer, some totem codecs and Limewire. Interestingly, Synaptic removed Openoffice during the upgrade process. So I'd have to install that too. I also need to download the Linux-image-686 smp kernel. I wonder why ubuntu doesn't ship that with their CDs since almost all modern processors are P4 based and the 686 kernel is for P3 and P4 SMP.With the dapper release, Sun JDK is going to be a part of the multiverse apt source list. There's a lot of furor in the free software community about Sun's java distribution license. It's surprising that despite the popularity of Java, JDK does not ship with any major Linux distro.[end geek stuff]Hmm...If you're not familiar with most of the keywords used above...(even then you actually tried to read it all??)...I'd be happy to put together a starter guide to Linux someday. [Less]

Yesterday I got an invitation to try out the new Krugle Beta 6 search engine. For the lesser internet freaks, Krugle is a new kind of search engine. It is aimed at developers looking for technical content and, most importantly, code.I signed up for ... [More] the beta program immeadiately after I heard about Krugle through the MoMB (Museum of Modern Betas) about two months ago. So you can imagine the kind of excitement this newbie is generating among the developer community. This new search engine was listed in the hottest 100 betas list at the MoMB. And I think with good reason.Let me show you around a bit. Krugle crawls and indexes public source code repositories like Sourceforge and the like. It also indexes source codes of many major open source projects such as Apache, Debian, Ubuntu, Fedora, you know what. So you can search for keywords such as Hibernate specifying the code language as Java. You can see the results here:Clicking on any search result, opens it in another tab where you can also see the whole code, the project to which this code belongs and other related sources in that project in a tree view. For browsing anything you never have to go out of krugle.Krugle also searches for Technical content, whenever you search for a particular keyword, it shows you the technical articles relevant to your search term. You can also choose to search only for technical articles or browse source code for an open source project.Perhaps the most important utility feature that krugle offers is the ability to annotate, bookmark and share your search results. For example, a friend asks you to find file upload code for servlets. You can search for this term, find the most relevant code and technical article, add notes to them and share for him to browse.Looks good on first sight. I'll post a detailed experience when I...err...get some experience. For now, I have my ERP test coming up in another 2.5 hours. Also, I am downloading the Ubuntu 6.06 "Dapper Drake" Release Candidate which is the almost final release up for beta testing. The final stable release will be up for grabs in June. Another blog posting coming up shortly. Keep watching. [Less]