You’re absolutely correct that it’s very hard to accomplish anything
like a true real-time workflow with Hadoop or indeed with any MPP
system.

One approach that you point out is to separate the actual analysis of
data (say using a Hadoop process) from its use (the instantaneous
lookup).

While this is certainly a step in the right direction it solves only
half of the problem and does not attack the fundamental issue of
accomplishing the analytics itself in a high-throughput low-latency
fashion. The reason for this is quite simple: MPP systems like Hadoop
are designed for very high scalability and accomplish this via
brute-force parallelism. As a result while it’s possible to get very
high throughput from such systems, the associated overheads (mainly
communication) are crippling unless amortized over very large batches
of data.

At Truviso (http://www.truviso.com) we are pioneering a continuous
analytics approach attacks both halves of this problem by decoupling
the analysis and the use of the data much as you point out with your
suggestive search example.

Sailesh Krishnamurthy
Truviso