Lucene and SOLR Get Commercial Support 47
ruphus13 writes "Two of the technical leads and core committers of the Lucene Project have launched Lucid Imagination, a venture backed company now offering commercial versions of Lucene and SOLR in the hopes of making it the de facto choice of search technologies used by companies within their products. 'The Lucene search library ranks amongst the top 5 Apache projects, installed at over 4,000 global companies. Although OStatic is primarily Drupal-based, our site's search is based on Lucene. According to Lucid Imagination officials, the Solr search server, which transforms the Lucene search library into a ready-to-use search platform for building applications, is the fastest growing Lucene sub-project...Lucid's business model is roughly comparable to Red Hat's very successful model, in that it centers on support and services for free, open source software.'"
oookay. (Score:5, Insightful)
Nice press release but.. what does it do? O_o Five million dollars and they couldn't even buy a one sentence description of their product. Standards are slipping.
Re:oookay. (Score:4, Insightful)
"...in the hopes of making it the defacto choice of search technologies used by companies within their products. 'The Lucene search library ranks amongst the top 5 Apache projects... According to Lucid Imagination officials, the Solr search server, which transforms the Lucene search library into a ready-to-use search platform for building applications...
I agree, it could have been more explicit in giving a brief description, but was it really that difficult to glean what it does from the summary?
Re: (Score:1)
So your reaction to something you don't understand is denigration? I hope you don't expect people to accept you as a girl in training, then. It would be hypocritical.
Re: (Score:3, Informative)
No. It's a search engine for your website. It's not quite as simple as a SELECT query. Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform. [apache.org]. That does quite a bit more than a SELECT query could hope to do.
Re: (Score:1, Troll)
That, and at the least anyone peripherally involved with non-MS development should probably know what Lucene is. It's that awesome.
Comment removed (Score:5, Informative)
Re: (Score:3, Insightful)
Re: (Score:1, Insightful)
I read the summary twice and it just made my head spin.
There's a big presumption in the summary that we've heard of Lucene before. I don't even know what they do. Do they search... the web? ...your LAN? ...your desktop?
Re: (Score:2)
That's still illegal, though Bush was working on it behind closed doors.
Re: (Score:1)
I don't even know what they do. Do they search... the web? ...your LAN? ...your desktop?
In short: yes.
Re: (Score:3, Informative)
Lucene is a full-text indexer and search library. Solr is a full-text indexer and search server, based on Lucene.
full-text search (Score:5, Informative)
You mentioned SQL SELECTs elsewhere. Full-text search isn't like a SELECT. It's more like what what happens when you google something: many documents are searched in a split second, and complex queries can be done, like documents containing a phrase, but not this one, or documents that mention X with Y within a few sentences of that, or documents that mention X and Y, but not Z. Yes, SQL lets you do that, but not for text, except in very inefficient ways.
From what I've seen of it (which is very little), Lucene lets you, as a programmer, index data using your own field names. So, say you're indexing word documents and HTML documents. You can extract most of the text and index it as "maincontent", but seperately extract the author, title and subtitle, indexing those individually. This lets you query attributes, like: "space nasa and not genre:sci-fi". Full text search also does ranking based on the occurences of different words you query by, etc. Presumably Lucene would let you specify which fields/attributes are included in a search, and which ones have the highest scores in search results, for instance.
Yeah, I don't get where $5m USD went on that either. I didn't think it was THAT big a problem. But maybe it is. Personally, I'm holding out for a decent Triple API, which hopefully make all but the indexer of this obsolete.
Re: (Score:2)
Yeah, I don't get where $5m USD went on that either. I didn't think it was THAT big a problem.
Getting it right, and doing it as well as Lucene does (which is spectacularly well), really is THAT big a problem.
Re: (Score:2, Informative)
From what I've seen of it (which is very little), Lucene lets you, as a programmer, index data using your own field names. So, say you're indexing word documents and HTML documents. You can extract most of the text and index it as "maincontent", but seperately extract the author, title and subtitle, indexing those individually. This lets you query attributes, like: "space nasa and not genre:sci-fi". Full text search also does ranking based on the occurences of different words you query by, etc. Presumably Lucene would let you specify which fields/attributes are included in a search, and which ones have the highest scores in search results, for instance.
You've certainly hit close to the mark. I work on a site that uses Solr and it does work just as incredibly as others have said. You can tell it what fields you want to search. You can tell it what order you want results sorted in (and you can sort on more than one column in cases of relevancy ties). You can tell it you want matches in one column weighted more than another. You can tell it you want the terms to be within X words of each other. And you can tell it what words should not be in the result
Re: (Score:2)
One of the most interesting fields where Lucene is useful (probably also for you) is Wikipedia. Remember how painful it was to search something on Wikipedia some months ago?
Well now, thanks to Lucene, Wikipedia (and its sister projects) don't have to use the in-build MediaWiki search engine (which really is crappy). Probably the best feature Lucene brings is "Did you mean ...". Google is still better, but Lucene was a big step for Wikipedia.
Re: (Score:1)
Wikipedia has been using Lucene for a few years by now. The recent changes were improvements to how it was used, but it was being used the whole time. Out of the box, MediaWiki uses whatever fulltext search is available from the DBMS being used -- in MySQL's case, that means using MyISAM, which is impossible for a site the size of Wikipedia (all selects, updates, deletes, etc. take out table-level locks).
Re: (Score:1, Offtopic)
Ummm, nope. There Can Be Only One.
interesting (Score:1)
Re: (Score:2)
possible alternative: xapian (Score:2, Interesting)
Good to have choice, i guess.
Re: (Score:2, Informative)
Re: (Score:1)
Re: (Score:1)
They typically wouldn't. Python bindings for C or C++ libraries are usually nothing more than pointers to the correct shared library calls.
SOLR has several advantages (Score:4, Informative)
I agree, Xapian is nice, and we considered it for a while. However, in the end, the decision was made to use SOLR because of one overriding factor in its favor: it takes care of all the nasty details to enable concurrent access, which makes developing web applications just so much easier. With SOLR you just don't have to worry about who might currently be reading or writing to the index, and the index replication features are very powerful, too.
That, and facet searches are very nice, too (e.g., searching for a keyword and then automatically displaying the # of hits per category, and refining per category).
SOLR has Python bindings, too, by the way. They currently are not in the official repository, but recently maintenance on them has picked up, and they work in a very Pythonic way.
About to move to the Java port of Lucene... (Score:5, Informative)
We're currently using the Zend PHP port of Lucene. It was nice, because we were able to use all our existing code for loading our PHP objects from the database for indexing. It worked fine, as long as are indexes stayed small.
Now we have several indexes weighing in at around 300+ megabytes, and Zend Lucene has proven to be absolute crap. It takes seconds of CPU time, and hundreds of megs of ram to process simple queries against these indexes. When tested in Luke [getopt.org], the same queries against the same indexes finish in milliseconds with minimal memory usage. Either the Zend port, or PHP itself is clearly unsuitable for production use on large indexes.
Either way, we're going to switch it out for Solr ASAP, and we anticipate the development overhead should be minimal -- we'll keep using the same code to load our objects, and pass them to Solr via JSON.
Re: (Score:2)
Either the Zend port, or PHP itself is clearly unsuitable for production use on large indexes.
You phrase this in such a way as to imply an exclusion, when really both are often true. We've ported our PHP application to Rails (which provides a different, but workable, set of problems), and we've rid ourselves of the Zend engine in return for Ferret; I'm a proponent of replacing that with SOLR, but we've yet to go down that path.
Re:About to move to the Java port of Lucene... (Score:4, Insightful)
That's because the Zend Lucene library is written in pure PHP, ergo... _really_ slow. Either use a C module or get SOLR to get it fast. In my simple tests the Python lucene libraries were about 100-500 times faster than the Zend PHP version, it's really one of the worst Lucene libraries around (in terms of speed).
Re: (Score:1, Informative)
I found the original Java libraries to be plenty fast as well. We index millions of records, and it's always been plenty fast returning even the most complex queries. Granted, it probably isn't as fast as the C library, but it is the most updated and feature rich. And, many of those later features that the C library lacks makes it COMPLETELY worth it.
Re: (Score:2)
Yes the Lucene php version is very very slow (very)
I recently switched to sphinx (http://www.sphinxsearch.com/) its written in C and compiles nicely on my linux servers, indexes documents at crazy speeds and theres piles of options
I highly recommend above (use it on 200,000 queries a day vertical search engine for one of our sites)
Re: (Score:2)
> I recently switched to sphinx (http://www.sphinxsearch.com/) its written in C
Minor nit - it's in C++. But yeah, it's totally awesome - fast when indexing, easy to scale horizontally, powerful query language, custom stop word lists, etc, etc. The APIs (I use the Ruby one, Riddle) make it easy to do nifty excerpt formatting (for example, note the highlighting around the word 'battle' [militarypr...glists.com]), and there are a couple of different ways to integrate it into a Ruby on Rails app.
Speaking of Sphinx and Rails, here's
Re: (Score:1)
Re: (Score:2)
What I ended up doing for various webapps (PHP and Python, although Python's port of Lucene actually loads the Java runtime, and is fairly fast) is create a simple local server that a PHP script can communicate with over sockets and a trivial protocol.
This is fairly straightforward for me since most of the time I just want Lucene to return a list of document IDs. I use those IDs to create a temp table that I can do additional queries against in SQL.
Running it as a separate server allows me to use the origin
Both Spring and Hibernate have Lucene modules (Score:1)
I'm looking into adding search on my site so I should probably check it out. There's a new "In Action" [amazon.com] book out for using the Hibernate Lucene add-on -- I might have to pick that up.
Re: (Score:1, Informative)