GigaOM » database

Now anyone can buy the NSA’s database tech

Derrick Harris — Wed, 19 Jun 2013 16:18:43 +0000

Say what you will about the National Security Agency, but you can’t say it doesn’t know how to share — or how to build technology that can scale. In fact, Accumulo, the petabyte-scale database technology the agency built, has been available as an open-source project for a couple of years. Now, however, a more-polished version of Accumulo is up for sale to the general public thanks to a Cambridge, Mass.-based startup called Sqrrl.

On Wednesday the company announced the general availability of its product, Sqrrl Enterprise, which is a cleaned-up and more-functional version of the open source Accumulo software. That means users will get an experience a lot more similar to what NSA data analysts get than what the core database code allows.

How do we know this? Because Sqrrl’s co-founder and CTO Adam Fuchs helped build Accumulo and the applications that run on top of it during his previous life working for the spy agency. (If you want to know more about the history of Accumulo and the types of massive graph analyses the NSA is using it for, you can check out my coverage of the NSA citizen-spying scandal from two weeks ago (here and here).) So, instead of just downloading an open-source take on Google’s BigTable data store, Sqrrl users get things like built-in analytic functions and search; support for JSON data structures; and data encryption both at rest and in motion.

The Sqrrl architecture

It’s the latter features around security that Sqrrl Co-founder and VP of Business Development Ely Kahn said have many early Sqrrl users most excited. Health care companies, in particular, highlight an ideal use case for security features like those that Sqrrl provides. Because of its cell-level security and access control, Kahn explained, providers can try to do new things around data sharing while still keeping compliant with regulations such as HIPAA and the data requirements that come along with the Affordable Care Act.

But the applications of Accumulo and Sqrrl could be much broader across industries. Because it’s based on Hadoop, Sqrrl gives companies peace of mind when it comes to storing big data securely, Kahn said, which has been a big reason that many companies are afraid to do Hadoop in production. And Sqrrl’s analytic capabilities make it easier to analyze all that data, including log files and network data that could help a company track down the causes of any cyberattacks they might suffer.

At this point, said Kahn, who was previously director of cybersecurity strategy at the National Security Staff in the White House, that should be a major concern. For most organizations, he said, it’s not a question of whether they’ve been breached but “a question of whether they know that they know they’ve been breached.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Stealth-mode 28msec wants to build a Tower of Babel for databases

Derrick Harris — Tue, 11 Jun 2013 13:00:01 +0000

28msec is not your average database startup but, then again, neither is its mission. The company — still in stealth mode (until our Structure Launchpad event on June 20) after about seven years of existence — has created a data-processing platform that it says can take and analyze data from any source, and then deliver the results in real time.

The company took so long to officially launch, CEO Eric Kish told me, because it took such a long time to build. The 28msec history goes like this: The early investors are database industry veterans (one was employee No. 6 at Oracle) who, at some point in 2006, envisioned an explosion in data formats and databases. Their solution was to create a platform able to extract data from any of these sources, transform it into a standard format, and then let users analyze it using a single query language that looks a lot like the SQL they already know. 28msec is based on the open source JSONiq and Zorba query languages and will be available as a cloud service.

That’s about all Kish is willing to spill right now with regard to the technology.

As for the company itself, it has been staffed thus far primarily by Ph.Ds. in query technologies from ETH Zurich in Switzerland, where co-founder Donald Kossmann is a professor. Every year since 28msec was founded, it has hired one of his graduates to help build the product. The company brought on Kish, a serial entrepreneur, as CEO in 2012.

28msec was originally based in Zurich, but is in the process of shifting its base to Palo Alto, where Kish lives. It has raised $5.5 million in capital from friends and family, and already has paying customers.

As for the name, 28msec, it’s a reference to the time it takes for a database to access data stored on a hard disk. After the headquarters, maybe that name will be the next thing to change given the prevalence of flash and RAM as database storage media. “Seven years later,” Kish acknowledged, “it’s not relevant anymore.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Under the covers of the NSA’s big data effort

Derrick Harris — Sat, 08 Jun 2013 02:15:19 +0000

The NSA’s data collection practices have much of America — and certainly the tech community — on edge, but sources familiar with the agency’s technology are saying the situation isn’t as bad as it seems. Yes, the agency has a lot of data and can do some powerful analysis, but, the argument goes, there are strict limits in place around how the agency can use it and who has access. Whether that’s good enough is still an open debate, but here’s what we know about the technology that’s underpinning all that data.

What is Accumulo?

The technological linchpin to everything the NSA is doing from a data-analysis perspective is Accumulo — an open-source database the agency built in order to store and analyze huge amounts of data. Adam Fuchs knows Accumulo well because he helped build it during a nine-year stint with the NSA; he’s now co-founder and CTO of a company called Sqrrl that sells a commercial version of the database system. I spoke with him earlier this week, days before news broke of the NSA collecting data from Verizon and the country’s largest web companies.

Adam Fuchs

The NSA began building Accumulo in late 2007, Fuchs said, because they were trying to do automated analysis for tracking and discovering new terrorism suspects. “We had a set of applications that we wanted to develop and we were looking for the right infrastructure to build them on,” he said.

The problem was those technologies weren’t available. He liked what projects like HBase were doing by using Hadoop to mimic Google’s famous BigTable data store, but it still wasn’t up to the NSA requirements around scalability, reliability or security. So, they began work on a project called CloudBase, which eventually was renamed Accumulo.

Now, Fuchs said, “It’s operating at thousands-of-nodes scale” within the NSA’s data centers. There are multiple instances each storing tens of petabytes (1 petabyte equals 1,000 terabyes or 1 million gigabytes) of data and it’s the backend of the agency’s most widely used analytical capabilities. Accumulo’s ability to handle data in a variety of formats (a characteristic called “schemaless” in database jargon) means the NSA can store data from numerous sources all within the database and add new analytic capabilities in days or even hours.

“It’s quite critical,” he added.

What the NSA can and can’t do with all this data

As I explained on Thursday, Accumulo is especially adept at analyzing trillions of data points in order to build massive graphs that can detect the connections between them and the strength of the connections. Fuchs didn’t talk about the size of the NSA’s graph, but he did say the database is designed to handle months or years worth of information and let analysts move from query to query very fast. When you’re talking about analyzing call records, it’s easy to see where this type of analysis would be valuable in determining how far a suspected terrorist’s network might spread and who might be involved.

Stewart Baker, former NSA general counsel under George W. Bush, wrote on his blog Thursday that this type of data could also be used for for general pattern recognition — the kinds of stuff that targeted advertisers love to do. Only, instead of the system serving someone an ad because of what they’ve been searching for and the operating system they’re using, Baker presented the hypothetical of “[an] American who makes a call to Yemen at 11 a.m., Sanaa time, hangs up after a few seconds, and then gets a call from a different Yemeni number three hours later.”

The big legal question here is around probable cause and whether the government should further investigate this caller based on call patterns similar to those of known terrorists, but the big data question is around false positives. Baker’s hypothetical might appear pretty cut and dry but, data scientist Joseph Turian explains, call records in general probably don’t offer too strong of a signal and could lead to situations where innocent behavior patterns looks a lot like nefarious ones. “But once you start connecting the dots with other pieces of information you have from other sources,” he said via email, “you can start making more predictions.”

This is where a program like PRISM, the NSA’s reported effort to collect data straight from the likes of Google, Facebook and Apple could come into play. If you’re able to tie a name or web account to a phone number, you can figure out all sorts of information. If you can prove that certain people are radical Islamists, for example, you can start to infer more things about the others in that social graph.

And if Sqrrl’s capabilities are any indicator of what Accumulo is supporting within the NSA, the agency can perform a lot of simpler functions on its data as well. In addition to graph processing, said Ely Kahn, Sqrrl’s co-founder and VP of business development, their product includes pre-packaged analytic capabilities around SQL queries and full-text search, and also supports streaming data. This means Sqrrl’s version can support any number of interesting use cases — from processing data as it hits the system to keeping a massive index that can be searched in the same way someone searches the web.

How much data is the NSA collecting? Follow the money

We’re not quite sure how much data the two programs that came to light this week are actually collecting, but the evidence suggests it’s not that much — at least from a volume perspective. Take the PRISM program that’s gathering data from web properties including Google, Facebook, Microsoft, Apple, Yahoo and AOL. It seems the NSA would have to be selective in what it grabs.

Assuming it includes every cost associated with running the program, the $20 million per year allocated to PRISM, according to the slides published by the Washington Post, wouldn’t be nearly enough to store all the raw data — much less new datasets created from analyses — from such large web properties. Yahoo alone, I’m told, was spending over $100 million a year to operate its approximately 42,000-node Hadoop environment, consisting of hundreds of petabytes, a few years ago. Facebook users are generating more than 500 terabytes of new data every day.

Using about the least-expensive option around for mass storage — cloud storage provider Backblaze’s open source storage pod designs — just storing 500 terabytes of Facebook data a day would cost more than $10 million in hardware alone over the course of a year. Using higher-performance hard drives or other premium gear — things Backblaze eschews because it’s concerned primarily about cost and scalability rather than performance — would cost even more.

Even at the Backblaze price point, though, which is pocket change for the NSA, the agency would easily run over $20 million trying to store too many emails, chats, Skype calls, photos, videos and other types data from the other companies it’s working with.

Actually, it’s possible the intelligence community is taking advantage of the Backblaze designs. In September 2011, Backblaze CEO Gleb Budman says, he met with CIA representatives who discussed that agency’s five-year plan “to centralize data services into a large private cloud” and how Backblaze’s technology might fit into it. Its plans for analyzing this data, as illustrated in the slide below (and discussed by CIA CTO Ira “Gus” Hunt at Structure: Data in March), seem to mirror what the NSA has in mind.

Whatever type of gear the NSA is using, though, and how ever much it’s spending on the Verizon data or PRISM specifically, we do know the agency is spending a lot of money on its data infrastructure. There are those dozens (at least) of petabytes of overall data in Accumulo, and the agency is famously building a 1-million-square-foot, $1.5 billion data center in Utah. It recently began construction on a 600,000-square-foot, $860 million facility in Maryland.

Policies are in place

Sqrrl’s Kahn — who previously served as director of cybersecurity strategy at the National Security Staff in the White House — says even with all the effort it’s putting into data collection and analysis, the NSA really is concerned about privacy. Not only are there strict administrative and legal limitations in place about when the agency can actually search through collected data (something Stewart Baker explains in more detail in a Friday blog post), but Accumulo itself was designed with privacy in mind.

The system itself is designed to make sure there’s not a free-for-all on data, another individual familiar with Accumulo said.

It has what Kahn and Sqrrl CTO Fuchs described as “cell-level” security, meaning administrators can manage access to individual pieces of data within a table. Furthermore, Fuchs explained, those policies stick with the data as it’s transformed as part of the analysis process, so someone prohibited from seeing it won’t be able to see it just because it’s now part of a different dataset. When data would come into the NSA from the CIA, he said, there were policies in place around who could see it, and Accumulo helped enforce them.

Even agencies within the Department of Homeland Security are using or experimenting with Accumulo, Kahn added, because proposed legislation would put them in charge of ensuring privacy as cybersecurity data exchanges hands between the government and private corporations.

It’s ironic he acknowledged, but Accumulo actually flips the presumed paradigm that stricter security and privacy regulations mean less sharing. That might be a shallow victory for citizens concerned about their civil liberties, but data collection and sharing don’t seem likely to stop any time soon. At least it’s something.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Here’s how the NSA analyzes all that call data

Derrick Harris — Thu, 06 Jun 2013 22:13:39 +0000

The National Security Agency might not have the names of Verizon’s wireless customers, but the agency probably can figure out what they’re up to if it’s so inclined. The metadata Verizon has provided the NSA — phone numbers, numbers called, duration of calls, location — is a veritable treasure trove to an organization with the right analytic skills and the right tools. The NSA has both.

There are numerous methods the NSA could use to extract some insights from what must be a mind-blowing number of phone calls and text messages, but graph analysis is likely the king. As we’ve explained numerous times over the past few months, graph analysis is ideal for identifying connections among pieces of data. It’s what powers social graphs, product recommendations and even some fairly complex medical research.

My LinkedIn social graph

But now it has really come to the fore as a tool for fighting crime (or intruding on civil liberties, however you want to look at it). The NSA is storing all those Verizon (and, presumably, other carrier records) in a massive database system called Accumulo, which it built itself (on top of Hadoop) a few years ago because there weren’t any other options suitable for its scale and requirements around stability or security. The NSA is currently storing tens of petabytes of data in Accumulo.

For a more thorough description of Accumulo and the NSA infrastructure, read our post “Under the covers of the NSA’s big data effort.”

In graph parlance, vertices are the individual data points (e.g., phone numbers or social network users) and edges are the connections among them. In late May, the NSA released a slide presentation detailing how fast fast Accumulo is able to process a 4.4-trillion-node, 70-trillion-edge graph. By way of comparison, the graph behind Facebook’s Graph Search feature contains billions of nodes and trillions of edges. (In the low trillions, from what I understand.)

So, yes, the NSA is able to easily analyze the call and text-message records of hundreds of million of mobile subscribers. It’s also building out some massive data center real estate to support all the data it’s collecting.

How might a graph analysis work within the NSA? The easy answer, which the government has acknowledged, is to figure out who else is in contact with suspected terrorists. If there’s a strong connection between you and Public Enemy No. 1, the NSA will find out and get to work figuring out who you are. That could be via a search warrant or wiretap authorization, or it could conceivably figure out who someone likely is by using location data.

Having such a big database of call records also provides the NSA with an easy way to go back and find out information about someone should their number pop up in a future investigation. Assuming the number is somewhere in their index, agents can track it down and get to work figuring out who it’s related to and from where it has been making calls.

Presumably, agents could begin with location data, too. If a bomb went off at Location X, bringing up all the numbers making calls from towers in that area might be a good starting point for investigation. Tracking someone’s movement from location data could be helpful, too.

If this all sounds a little creepy, maybe it should. After all, the world’s biggest, baddest intelligence agency can pretty much figure out who you are, who you know and where you go. And unlike web and retail companies that collect and analyze so much data about us, the government can put you in jail.

It might be even creepier when you consider how much other data law enforcement agencies can collect about you without a warrant.

However, someone familiar with NSA policy told me, the good news is that the vast majority of people are still anonymous even in this sea of data: There’s just too much data to care until someone pops up in the bad guys’ networks or gets on the agency’s radar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

IBM throws its weight behind MongoDB for mobile apps

Derrick Harris — Tue, 04 Jun 2013 21:17:35 +0000

IBM helped propel SQL, Linux and Java into the mainstream, and now it’s looking to do the same for MongoDB. The company said it’s working with MongoDB creator 10gen on a new standard that will let mobile apps built atop the NoSQL database connect with data stored in business-critical systems.

At its core, the new standard — which encompasses the MongoDB API, data representation (BSON), query language and wire protocol — appears to be all about establishing a way for mobile and other next-generation applications to connect with enterprise database systems such as IBM’s popular DB2 database and its WebSphere eXtreme Scale data grid. MongoDB is already immensely popular among web and mobile developers who must deal with semi- and unstructured data, but its lack of transactional integrity (among other things) means MongoDB isn’t often deployed for “mission-critical” applications that require ACID compliance and consistent performance.

In theory, the new standards would MongoDB-based applications easily and securely access mission-critical database systems. This could usher in a new wave of flexible applications that add significant value by spanning multiple data systems. According to a press release, “Customers can begin to use these new features later this summer by pairing eXtreme Scale with MongoDB, and by running their MongoDB applications on DB2 directly.”

The companies are also seeking participation from other parties interested in developing standard methods for interacting with MongoDB.

However, there’s a bigger shift at play here than the development of a new database standard, and it has everything to do with IBM’s planned acquisition of cloud provider SoftLayer, also announced on Tuesday. If IBM wants to remain relevant as server sales and application platforms move to the cloud, it has to embrace the new business and application-development models that come along with cloud computing. IBM’s stable of enterprise developers might not be deploying mobile apps on Parse or Google any time soon, but they will look for alternative platforms if IBM doesn’t at least try to keep up with a changing landscape.

Coincidentally, SoftLayer and 10gen already have a strong partnership around hosting MongoDB applications in the cloud.

If IBM is still an IT kingmaker, that bodes very well for MongoDB, as well as for the OpenStack cloud computing platform that IBM is also backing. If IBM’s influence in this realm is slipping, though, one could argue that IBM needs MongoDB and OpenStack more than they need it.

I am awaiting comment from IBM and/or 10gen for more details on the scope of their partnership and this standard, and will update when I hear more.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Why Hortonworks is riding a faster Hive to the bitter end

Derrick Harris — Wed, 29 May 2013 23:14:51 +0000

Hortonworks isn’t about to get off the Apache Hadoop elephant just because everyone around it is now trying to ride impalas. The company released version 1.3 of its Hortonworks Data Platform on Wednesday, a major aspect of which is an improved iteration of Apache Hive that the company claims runs 50 times faster the previous version. Over the next year or so, Hortonworks expects to improve the speed of Hive by 100x its previous limits — this while its competitors are all but leaving Hive in the dust in favor of newer, faster analytic systems.

If you’re unfamiliar with Hive, it’s a project that Facebook developed in 2008 to make Hadoop function more like a traditional enterprise data warehouse. Hive stores data inside the Hadoop Distributed File System in structured format, and then allows users to query it using a language very similar to SQL. Until very recently, Hive has been the de facto method for querying (in a traditional sense) data stored in Hadoop, and it has proven immensely popular as more companies have begun tackling their big data woes with Hadoop.

Hive wasn’t built for speed

However, as more companies got used to Hadoop, they also began to notice its shortcomings. One of them is around MapReduce, a powerful but not-exactly-speedy method of processing data that requires running the job across every node in the cluster in order to find the right data. Although the Hive interface is that of a SQL query, it relies on on MapReduce as the processing engine.

(For more on how Hadoop and its flavor of MapReduce came to be, read this post on the history of Hadoop. To see me speak with Google Fellow and MapReduce creator Jeff Dean about how far Google has moved from a MapReduce-centric computing model, come to Structure next month.)

Users wanted faster, more-interactive query processing on top of Hadoop, similar to what they had grown accustomed to with data warehouse systems such as Teradata, Greenplum and Netezza. Hadoop vendors such as Cloudera (with Impala), MapR (with Drill), IBM (with Big SQL) — as well as a spate of startups — have obliged with their own new technologies that in various ways blend the familiarity of SQL with the scalability of Hadoop. EMC Greenplum, now Pivotal, has transplanted its existing database system inside of Hadoop.

Even Qubole, a cloud-based startup from Hive creators Ashish Thusoo and Joydeep Sen Sarma, is keeping an eye on how projects such as Impala and Shark (from the University of California, Berkeley’s AMPLab) might factor into its plans.

Giving Hive a better “Stinger”

Hortonworks, the Yahoo spinoff dedicated to driving the Apache Hadoop bus, is sticking with Hive. But is has a plan, and a point.

Essentially, VP of Products Bob Page told me during a recent briefing, “It just makes more sense from our view to have everything done in one place.” He means that Hive is already the method by which most people are already comfortable using SQL to access Hadoop data, so there’s no use rocking the boat by adding yet another technology into the mix. Hortonworks will just make Hive faster to the point (100x) where it’s at least in the ballpark of what these entirely new systems are capable of doing, but where users still use the same tools for interactive and batch queries.

It has in place a three-phase plan, under the “Stinger” codename, in order to make this happen. The first phase, now available as part of the Hive 0.11 release, is a new set of analytic functions and a columnar file format that Page says has resulted in a 50x performance increase over the previous version. The next phase is to move ~~YARN~~ Hive off of MapReduce and onto a still-under-development processing framework called Tez.

“You’ll see phase two come to bear later this year,” Page said, once YARN — a new resource manager that lets Hadoop clusters run multiple processing engines simultaneously — is ready for production.

The third phase is a whole new vector query engine for Hive and new tools for intelligent query planning. Page didn’t have a target date in mind for that phase, except to note that “we’re not talking about a five-year cycle.”

SQL isn’t the end game for Hadoop

It would be easy to dismiss Page’s and Hortonworks’ optimism about Stinger as a sweet lemons type of rationalization — the company was founded around Apache Hadoop and can’t really go about developing entirely new products outside that foundation — but they also appear to have their eyes focused on a future where SQL isn’t too big a differentiator.

SQL is the way folks used to data for the last 30 years can see how Hadoop fits in their environment, Page said, but the compelling thing about Hadoop “is it really unlocks a new way about how one thinks about storing and processing data.” Once YARN is ready to go, he added, there will be new avenues of innovation in areas like graph analysis and stream processing.

Page comes from a place of credibility when he talks about this evolution in thinking. Before coming to Hortonworks in March, he was vice president of analytics platform and delivery at eBay, a company that knows its way around big data. When people get all their data in one place, they want to do more things with it, he explained. The thinking becomes less about using Hadoop to lower cost and more about “How do I use Hadoop to increase my top line?”.

Besides, Page noted (echoing the sentiment of just about everybody else in the Hadoop space, including Cloudera CEO Mike Olson), even as companies turn Hadoop into their primary data store, it’s difficult to see Hadoop ever entirely replacing high-value relational data warehouse systems like Teradata. One could argue, then, that there’s no real purpose in trying too hard to match those systems in terms of capabilities.

At eBay, he said, they ran an in-depth analysis to see if it was economically or technologically feasible to collapse its big data workloads onto a single system. eBay has dozens of petabytes stored in Hadoop and possibly more within various Teradata appliances. The result: “We just couldn’t find a way in which we could justify collapsing everything we do into one system.”

Feature image courtesy of Shutterstock user vblinov.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

As robots get smarter, they’ll be pouring our coffee (and beer)

Derrick Harris — Wed, 29 May 2013 17:43:38 +0000

Humanoid robots. Useful in theory, not so much in practice. They’re kind of creepy, too. But if you’re desperate to have a silicon-powered helper pour your beer, have no fear — that day might be closer than you think.

Scientists are hard at work creating robots that are able to sense and predict human actions, which should make them better performing tasks and look more natural while doing so. Researchers at Cornell University have trained a robot to recognize objects in its line of sight, as well as certain human actions, and then assign probabilities to the next set of possible actions. The video below (ignore the campiness of it) shows the robot in action trying to refill a cup of coffee — it must recognize the book and coffee cup on a person’s desk, predict which one he’s going to pick up, predict him taking a drink, then predict him setting the cup down for long enough for a pour to occur.

It’s kind of like IBM’s Watson system, only it’s predicting human actions instead of the correct answers to questions. In fact, both rely on databases full of relevant information — human movements and objects in the case of the robot — in order to make predictions. And instead of accepting natural-language queries like Watson does, the Cornell robot uses a Microsoft Kinect camera to visually detect what’s going on. (Interesting side note: Microsoft researchers used a Kinect and machine learning to train an elevator in the company’s research building to detect the difference between someone who intends to get on and someone who’s stopping to chat in front of it.)

Teaching robots to predict human gestures isn’t just about saving us a trip to the coffee pot, refrigerator or elevator control panel, though. As Stacey Higginbotham explained last week when covering a similar experiment by Disney Research, the more human-like the robot, the more comfortable people will be interacting with it. Important if you’re in the theme-park business, yes, but also if you’re trying to automate some of the caregiving functions that aging baby boomers will require in the next couple decades.

Feature image courtesy of Shutterstock user Ociacia.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

How to make a less creepy robot? Simple, just add data

Stacey Higginbotham — Mon, 20 May 2013 21:22:00 +0000

Disney’s research arm has solved a problem that you probably didn’t even know robots have — their inability to accept objects from people in a natural way. The Disney Research team, working with funding from the International Center for Advanced Communication Technologies (interACT) at Carnegie Mellon and the University and Karlsruhe Institute of Technology (KIT), believe that robots who can’t naturally accept “handoffs” of objects from people are creepy. In a paper presented this month, Disney and its partners detailed how they used several motion-sensitive cameras, a database of gestures and some fancy algorithms to solve this handoff problem.

From the press release announcing the findings:

“If a robot just sticks out its hand blindly, or uses motions that look more robotic than human, a person might feel uneasy working with that robot or might question whether it is up to the task,” Katsu Yamane, Disney Research, Pittsburgh senior research scientist explained. “We assume human-like motions are more user-friendly because they are familiar.”

Despite the robot pictured on the Disney page touting this research looking like the mechanical, blue-haired skeleton that haunted my childhood nightmares, its attempts to grab the purse from the person do seem reactive to the human’s gestures, as opposed to the robot just sticking his arm out there and the person having to accommodate it. And that sort of naturalism will be important as we bring more robots into our homes and workplaces.

For example, an MIT group used a dancer’s motions to build a robotic bartender in a quest for naturalism — even though that robot doesn’t interact with people.

Today, designers try to endear robots to us with quirky noises (like R2D2) and maybe light displays or LED faces — anything to help anthropomorphize them. But as robots become more human-looking they can also become more sinister — achieving that same uncanny valley that Disney and other content companies have struggled with in animation. Remember the dead-eyed stars of the Polar Express that you probably couldn’t empathize with? The jerky movements of a home health robot might engender similar feelings — or worse — they may scare people.

Building the natural gestures of the Disney robot took the creation of a hierarchical gesture database that the robot can access as it detects the person passing something to it. In the Disney paper research, the robot is not only able to reach for the handbag, but when the human attempts a fake pass to the robot, the ~~blue-haired monstrosity~~ robot is able to adapt. From the release:

To enable a robot to access a library of human-to-human passing motions with the speed necessary for robot-human interaction, the researchers developed a hierarchical data structure. Using principal component analysis, the researchers first developed a rough estimate of the distribution of various motion samples. They then grouped samples of similar poses and organized them into a binary tree structure. With a series of “either/or” decisions, the robot can rapidly search this database, so it can recognize when the person initiates a handing motion and then refine its response as the person follows through.

Even if you don’t have an opinion on how naturally robots should move, this research brings home the awesome amount of work it takes to build computers and robots that mimic the capabilities of a person. Much like computer visualization, the science of robotic interaction takes a problem the size of a mountain and has to chip it down into grains of sand using a toothpick to find solutions. It’s a testament to human curiosity that people are willing to try.

Also, I expect Disney might be lured by the idea of natural-looking robots roaming its theme parks. My only question is would they be dressed up as characters or working the cash register at the gift stores.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Database startup Drawn to Scale is closing down

Derrick Harris — Fri, 17 May 2013 21:24:03 +0000

Database startup Drawn to Scale, creator of the SQL-on-Hadoop technology called Spire, is closing down. Co-founder and CEO Bradford Stephens officially announced the closure in a blog post on Friday.

The company’s product, Spire, which provided full SQL support on top of the HBase NoSQL database, was one of the first products to try to blend Hadoop’s scalability with the robustness and familiarity of SQL. That’s now an increasingly crowded space (and has grown since that linked graphic was created). In March, Drawn to Scale expanded its support to MongoDB, as well.

I wasn’t shocked when Stephens told me the news — questions about the four-year-old company’s financial health had been swirling for a while — but to hear of its financial woes was a bit surprising. His account in the post pretty much echoes what I had heard from others:

“It seemed we had everything going for us — paid customers such as American Express, Orange Telecom, Flurry, and 4 others. Our technology worked brilliantly, we had a big hiring pipeline, and we had great media presence against our competitors who raised 10-100x more cash.”

He added:

“Yet five days before we signed term sheets for a big A round or sold the company, we started getting hit by a series of black swans — and we just didn’t have what we needed to recover. I’ll leave the public detail at that level, but I will say that paying employees’ health insurance out of your meager savings is a powerful incentive to change course.”

Up to this point, the company had raised $925,000 from RTP Ventures, IA Ventures and SK Ventures. There’s no word yet on what will come of the company’s intellectual property.

As Stephens — who’s now doing an entrepreneur-in-residence gig at Ping Identity and helping out other startups (including popular wardrobe app Cloth) — succinctly put it during a phone discussion, “We just don’t have the horsepower to keep running the company.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

How data warehousing is now a cost-effective solution for businesses

nraden — Mon, 13 May 2013 06:55:34 +0000

The new economics of data warehousing provide attractive alternatives in both costs and benefits. While big data gets most of the attention, evolved data warehousing will play an important role for the foreseeable future. In order to be relevant, data-warehouse design and operation need to be simplified, taking advantage of greatly improved hardware, software, and methods.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.