GigaOM » Apache Lucene

Emerging trends in the non-relational database market

augusttechgroup — Thu, 06 Sep 2012 20:15:48 +0000

The market for non-relational databases is a crowded one. Technology leaders looking to extract competitive advantages from their data must now familiarize themselves with this market. This report examines the current marketplace, providing a focused view of three products from across the current non-relational spectrum: Cassandra, Neo4J, and Datomic.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The Case for Open Source Search in the Enterprise

Haydn Shaughnessey — Mon, 01 Nov 2010 17:49:52 +0000

With enterprise data predicted to grow 650 percent between 2009 and 2014, the pressing question among IT departments is how to manage this data and make it easily searchable. While Google’s Search Appliance and offerings from search market leader Autonomy are options, these proprietary models often mean an increase in cost, either with the number of seats on a search software license or with the volume of searches in a given year, and some enterprises are seeking alternatives.

Otis Gospodnetic, co-author of Lucene implementations primer “Lucene in Action,” says, “You would be surprised to know how many services and applications use Lucene. For example, did you know that Apple uses Lucene in Safari and iTunes?” He adds that Lucene powers Spotlight, the search functionality in OSX, and browser history in Safari. CNET, Netflix, Cisco, the Guardian, LinkedIn and MTV use it as well.

What’s driving a wider adoption of open source? The open source community promotes the increased scalability, flexibility and speed over enterprise-specific behemoths like Autonomy, Microsoft and Google. From a company’s perspective, Enterprise 2.0 and the socialization of knowledge create both unstructured data and a demand for knowledge as content.

Other factors include:

Enterprise 2.0. Enterprises face a growing array of data types. On top of relational databases, companies now generate data through wikis, legacy bulletin boards, internal online communities, email, customer interaction points, PDFs, video, images and Microsoft Office files. Estimates of these volumes vary, but a ratio of 80 to 20 unstructured-to-structured data in the enterprise seems to be sticking. An inability to organize this unstructured data and simultaneously access different structured data files is a problem for business intelligence, adaptability, responsiveness and compliance.

People and video. Cisco’s Pulse project, which uses Lucene/Solar as a platform, is a good example of an enterprise search application made possible by open source. It allows employees to find one another based on their current interests, even down to their latest corporate video appearance. Pulse tags content and media across enterprise networks and makes them searchable in real-time. The benefit of Lucence/Solr as a platform for the project, according to Cisco, is its scalability and speed.

The cloud. Cloud computing and SaaS will present new challenges for data discovery and management as enterprises migrate data into cloud environments and expect immediate access to their assets through search.

There are barriers to the adoption of open source. Many enterprises have poor search and discovery platforms that don’t index and return data well, often because of poor index or topology management. Many have platforms with price plans that were designed when data access rights belonged to small teams, as opposed to the larger open source community.

However, the ability for customers to open up the search box and modify it for their own purposes and to include advanced features like filtering at no cost should see open-source search emerge as a market force in the enterprise, the cloud, SaaS and in e-commerce. Ultimately, the biggest driver may prove to be demand on the part of employees to make the enterprise environment look and work like the consumer-driven web.

Read the full post here.

Image source: Flickr user stuckincustoms

Will Yahoo Use Xoopit's CloudQuery to Help Usher in the Real-time Web?

Geva Perry — Thu, 23 Jul 2009 00:52:37 +0000

While Xoopit’s flagship email product is being touted as the reason why Yahoo agreed to buy the company, I think the underlying technology used by Xoopit is far more interesting. Xoopit has even exposed this infrastructure technology as an on-demand service, which it’s dubbed CloudQuery, with the tagline “Search as a Service.” But I suspect that, once the acquisition closes, CloudQuery will get buried deep in the bowels of Yahoo.

Essentially Xoopit’s CloudQuery allows you to search-enable any application. It combines the functionality of full text search with more structured database-like queries, which have the advantage of being aware of the underlying data structure of the application and thus provide more accurate search results and in near real-time. It is very similar to the open-source project Apache Solr; both are based on the popular Apache Lucene.

Suppose I start a dating site that I want to equip with a sophisticated search engine. I have certain well-defined data fields, which I want to be searched as such (and not with a Google-style free text search); in other words I want people to be able to search for “gender is female,” “age is greater than 30 and less than 40″ and “location is less than 5 miles from Zip 94107.” But I also want to enable a full text search that says “long walks on the beach.” With the Xoopit CloudQuery service I can quickly index my application and enable these sophisticated search capabilities, something I can’t do with Google ( s goog) Custom Search, for example.

Because of their near-instant indexing capabilities, technologies such as CloudQuery and Solr will be key in implementing the notion of the real-time web that has been popularized by services such as Twitter. Not only will I be able to perform the above query, but I will be able to set a real-time alert for it. To see a very simple but interesting use of this kind of technology, check out Dave Rosenberg’s “A-Z List of Underrated Bands,” which he created using Twitter Search. If you click on the link you’ll see that the search query at the top says: “underrated bands” from:daveofdoom. This query essentially says “find all tweets that contain the words ‘underrated bands’ posted by the user ‘daveofdoom’.” if you were to leave that page open while Dave was adding to the list, you would have seen it updated on-the-fly — hence, the real-time web.

While Twitter has developed its own solution with significant development resources, Xoopit’s CloudQuery enables any application owner to very easily index and search-enable their application using its hosted cloud service and APIs. In addition to CloudQuery and Solr, other projects that are attempting to provide similar capabilities include the Ferret project for Ruby apps and Sphinx Search. As another indication of the level of interest in this area, Lucid Imagination, the company behind Solr (and Lucene support provider) recently received more than $6 million in funding from Granite Ventures, Walden International and In-Q-Tel, the CIA’s venture fund. But Xoopit remains the only player that has implemented this type of technology as a cloud service.

Yet it appears Yahoo is acquiring Xoopit as part of its strategy of focusing on a handful of core apps — among them email — and turn its email application into a platform, around which it hopes developers will create an ecosystem of complementary apps and plug-ins, utilizing the APIs that Yahoo has published.

But as happens so often in this business, a lost opportunity for one company could prove to be a new opportunity for another.

Geva Perry, who writes the Thinking Out Cloud blog, is an adviser to startups and enterprises on cloud computing strategy and marketing. Follow him on Twitter: @gevaperry.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.