Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud and data first-quarter 2013: analysis and outlook
• How to use big data to make better business decisions

That’s the back story to Friday’s SAP Startup Forum held at the hack/reduce facility in Cambridge, Mass.’s Kendall Square neighborhood. There SAP talked up HANA, the company’s analytics database as a development platform data (or big data) applications to more than a dozen startups including Hadapt, Entagen, Diffeo, Objective Logistics, InsightSquared, Luminoso, Sqrrl, and Veracode. Those startups, in turn, were able to tout their business plans and demonstrate their products to an audience of reporters, VCs and others.

Wanted: (big) data startups

“We’re pitching to the startups and they’re pitching to us,” Scott Jones, SAP’s senior director for startup training and enablement told about 100 attendees. HANA, which debuted three years ago, has given SAP traction in an audience beyond its usual big-company ERP customer base and SAP fully intends to press that advantage.  SAP Ventures, the company’s VC arm, is increasingly active in finding and funding data startups. And it would very much like them to build their technology atop HANA.

The SAP execs repeatedly talked up HANA ONE, which runs on Amazon Web Services, as if to say “this isn’t the traditional, big iron, expensive SAP” of another era. It costs $3.49 per hour to run HANA ONE on an AWS EC2 8-core cluster.

As for the startups, many were clearly intrigued by HANA’s capabilities although none of those I talked to had actually run it. The consensus was this type of event and the promised perks — Jones offered free “no strings attached”  licenses and training — are what cash-strapped startups need. Indeed, that may be only way for a large commercial software vendor like SAP to hook small companies born-and-bred in a world dominated by free or nearly-free open-source software and rentable AWS infrastructure.

Plea to local startups: Stay put

A gaggle of area VCs were also on hand to sweet talk entrepreneurs into staying local rather than decamping to Silicon Valley after graduating from Harvard or MIT — as has been standard practice. Chris Lynch, the former CEO of Vertica Systems, who has helped nurture a big data startup community in and around Boston, was on hand to talk up that effort.

And, Dr. Sam Madden, of MIT’s Computer Science and Artificial Intelligence Lab (CSAIL) was there to help out. “Boston is an awesome place for startups — there’s a spectacular pool of outstanding, hungry young talent [here].” It helps that the VCs that used to live way out on the Route 128 corridor have relocated in closer into The Hub.

In the past two years, Kendall Square has seen a huge building boom with growing presence from Microsoft, Google, Amazon, IBM, Oracle and others. It’s a hip area to work for young techies — many of whom don’t own cars and like how mass transit and bike-friendly the area is.

Still, not every attendee was buying either pitch completely. Cyrille Vincey, CEO and founder of qunb, a data analytics and visualization startup that does use HANA, extolled its features and performance, but had one suggestion: “HANA is simple and fast. It feels like open source. Why don’t you open source it?”

And Timothy Jones, CEO and founder of Buzzient said his company, which has been based both Cambridge and Boston but is now virtual, may relocate to the San Francisco area. “We can get office space cheaper there than in Kendall Square,” he noted.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How data warehousing is now a cost-effective solution for businesses
• Listening platforms: finding the value in social media data
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud and data first-quarter 2013: analysis and outlook
• Takeaways from the second quarter in cloud and data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011

The historic space — in the old brick Kendall Boiler and Tank building — will be home to big data practitioners from academia, private and public industry. Hack/reduce, which sports the motto “code big or go home”, aims to bring top talent together to train the next-generation of sorely needed data scientists, Chris Lynch,  co-founder of hack/reduce told me.

Hack/reduce co-founder Chris Lynch

The facility can accommodate 150 dedicated hackers and is fielding 50 applications per week for spots. The first residents are Sqrrll, a big data startup launched by former National Security Agency technologists. “These 7 young men out of NSA spent 5 years building a big data store in Washington and now we have it in Boston,” said Lynch, who co-founded Vertica.

Lynch said he was inspired to give back to the community by a conversation with Diane Patrick, the wife of Massachusetts governor Deval Patrick. ”Mrs. Patrick told me about the governor’s transition from a top position at Coke to public service and said he knew he’d be measured based on what he gave back versus what he earned,” Lynch said.

Lynch, whose first job was at Maynard, Mass.-based Digital Equipment Corp., was frustrated seeing so many young computer scientists and programmers — including his own son — go elsewhere for jobs. If hack/reduce can  attract enough talent, only goodness will result, he said: “As we incubate young people, some will go to work at Fidelity as consumers of big data and some will go to EMC or Microsoft and other companies.”

The other hack/reduce co-founder is Frederic Lalonde, an entrepreneur and math scientist from Montreal. “Hack/reduce is his brainchild,” Lynch said. He hopes Boston’s resurging status as a database hub will help the effort.

Hack/reduce sponsors include Microsoft, Google, IBM, EMC’s Green Plum division, along with top venture capitalists and the Massachusetts Technology Cooperative.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012

“We want to build a community and innovation around big data to make Boston a leader there,” founding executive director Abby Fichtner told me. Hackers with expertise in biotech, medical devices, consumer web, energy, IT, telecommunications, music and art are all welcome to apply.

Boston locals see big data as a way to recapture some the high-tech glory that faded after the minicomputer era, as the center of gravity moved to Silicon Valley and Seattle. Now the plan is to capitalize both on experienced veterans left from Boston’s booming minicomputer years and young talent from area colleges to rebuild that high-tech hub in a big data mold.

Hack/reduce was founded this summer with some state funding and support from Chris Lynch, former president of Vertica Systems, a local big data company now owned by by HP. The group just moved into a historic building in Cambridge’s Kendall Square. Fichtner, formerly startup evangelist for Microsoft’s New England Research & Development (NERD) center, came aboard as the hacker space launched in August.

Feature photo courtesy of Flickr user kirinqueen.

.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How data warehousing is now a cost-effective solution for businesses
• How to use big data to make better business decisions
• How search can unlock the power of big data

Investors include 500 Startups, Data Collective and SK Ventures.

The new cash will go towards doubling the head count which now stands at six, adding support for Android and JavaScript and building an open-source Rails-based analytics dashboard for Keen.io’s data visualization APIs,  according to company co-founder Kyle Wild.

As Derrick Harris wrote when Keen.IO won GigaOM’s Launchpad competition last month:

The thinking behind Keen.io is pretty simple: No one should have to build their own analytics infrastructure, especially not mobile developers. It’s a long, hard, complex process that’s only made more difficult in an era where developers want to track a lot more metrics than just page views. So Keen built an analytics infrastructure….

San Francisco-based Keen.io has set lofty goals for itself. One of its investors characterized its potential addressable market as: “One million developers, one billion connected devices, one trillion events per day.”

Voter Planet, an iOS app built using Keen.io technology by Campaign Junkie LLC,  lets voters find the candidates running for federal office in their district, according to The San Antonio Business Journal. It will:

create personalized rosters of candidates they’d like to follow, track the latest polls, read news articles about races they follow and take part in app-wide polls  [and] political campaigns can use the app’s advanced features to connect with constituents via social media and offer live-streamed campaign events, fund-raising efforts and polls.

Keen.io sometimes competes with Flurry and in some situations when it’s used for non-mobile applications. “We’re like Twilio in that people use us rather than reinventing the wheel in house — they use us for analytics whereas they use Twilio for telephony,” Wild said.

“Our real competitors are in-house teams and technologies. People use us to do really, really custom stuff. And sometimes those in-house teams build on things like Omniture and Vertica. [But] our market is really mobile,” he said. Omniture is a web analytics service now owned by Adobe Systems; Vertica is an analytical database that HP bought last year.

Here’s Keen.io’s video from the TechStars’ Cloud conference in April:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• How data warehousing is now a cost-effective solution for businesses
• How to use big data to make better business decisions

Build or buy?

What stands out most when looking at Zynga’s gaming infrastructure is that Zynga prefers to buy software or use commercial open-source software rather than build its own. Its zCloud is built upon cloud computing software by Cloud.com (now part of Citrix), its analytics platform is primarily a Vertica database (now part of HP) and its application servers run the Couchbase NoSQL database. And although it has significantly lessened its reliance on the public cloud, Zynga still runs 20 percent of its gaming workload on Amazon Web Services.

Facebook? It runs its own Hadoop distribution, created the Cassandra database and has built countless tools for automating processes and speeding applications. It has scaled MySQL to extremes heretofore thought impossible. Heck, it even builds its own servers and storage as well as designing its own data centers from the ground up. It’s all very impressive, but is it money well spent?

Facebook spends so many man-hours building new software and hardware because it can tune them to the platform’s unique demands and because it can save untold millions on new gear, licenses and energy costs annually as the the infrastructure scales to handle nearly a billion users.

So why buy?

When I asked Zynga CTO Allan Leinwand about his company’s buy-over-build mentality, he told me it was a matter of what’s strategically important. Zynga focuses on deploying infrastructure that “satisfies the needs of our business,” he said, and Zynga’s core business is delivering an engaging gaming experience. “The most important thing for us is building infrastructure that runs our games in the most effective way,” he said, adding that he’d run Zynga on his iPhone if that provided the best user experience.

And it’s not as if Zynga’s zCloud didn’t require some heavy-duty engineering work. As Leinwand explained, the company wants its platform “to be available to everyone on the planet all the time,” and getting to that point meant lots of fine-tuning across the computing, storage and network infrastructure. It has gotten to a point where one of Zynga’s own servers can handle three times the load of a comparable machine image on AWS. Actually, being able to effectively utilize the public cloud for spikes in workload is itself an impressive feat for a company of Zynga’s size.

Fair enough. In fact, buying over building is an approach I’ve championed in the past (although one that I’ve softened on), especially for web startups. But Zynga is a billion-dollar company. If it grows as it wants to — and as its investors no doubt expect it to — the costs of license costs and limitations of commercial software could get cumbersome.

The numbers tell the story

Zynga’s latest financial results illustrate why the company is evolving into a platform provider, but they also suggest the company needs to nail that transition. Yes, the company earned just more than a billion dollars in 2011, but it ended up actually losing $405 million on the year. About $238 million (or 22 percent) of Zynga’s earnings went toward building out its infrastructure. Roughly $445 million went to Facebook as the cost of doing business on its platform.

Facebook, on the other hand, earned more than $3.7 billion in revenue in 2011 and netted a respectable $1.7 billion before taxes. It spent $606 million (or 16 percent) of its earnings on building out its infrastructure. And while Zynga said it expects to spend “in the range of $140 million to $160 million” on capital expenditures in 2012, Facebook is planning more than 10 times that amount as it builds out its data centers around the world.

Assuming it can bring its hundreds of million of users — as well as a vibrant developer ecosystem — to Zynga.com, perhaps Zynga can get back in the black. But what if it grows too fast (or too slow), or what if it wants to get a more balanced revenue stream between in-game purchases (currently about 90 percent) and advertising (currently about 10 percent)? Advertising revenue, after all, is the lifeblood of most web-platform companies (subscription required).

Facebook’s work on its own infrastructure technologies has enabled it to keep scale right along with its user base and their increased activity on the site. Facebook (as well as other web platforms such as Yahoo and Google) operates a massive Hadoop cluster that helps the company build features and target advertising based on what it’s able to learn about its users. Zynga has to hope its off-the-shelf pieces can scale — and do so affordably; if it needs to expand its analytics efforts, it might need to complement the Vertica system with a significant Hadoop deployment.

Being a major web platform is serious business on the infrastructure side, something Zynga no doubt understands. And while its current technology infrastructure is impressive and innovative, history doesn’t seem to be on Zynga’s side if it wants to grow into something as big as Facebook with its current system. Leinwand seems to get this. Zynga is always looking at new technologies and new ways to innovate, he said, because “forever’s an awful long time” to commit to anything on the web.

Feature image courtesy of Flickr user randomduck.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The capex connection: Why we pay for privacy on the Web
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• A field guide to cloud computing: current trends, future opportunities

Boston has some pretty good database DNA dating back to Digital Equipment Corp.’s venerable Rdb and a raft of small object-oriented database firms that popped up 20 years ago. Those good genes are showing up again in a fresh crop of database companies clustered around the Boston-Cambridge nexus.

The fact that database pioneers Michael Stonebraker (famous for Ingres, Postgres, Streambase and Vertica) is in the area, as is Jim Starkey (instrumental in Rdb, InterBase and Netfrastructure), helps draw database talent from elsewhere and keeps local prospects coming out of MIT or other area schools where they are.

In the traditional, relational database world, the power sphere remains in Silicon Valley, with Oracle, but even that company has been drawn to Boston,  where it beefed up its database portfolio with acquisitions like Endeca. Hewlett-Packard, another valley giant, bought Tewksbury, Mass.-based Vertica and is moving its “big data” operations to Cambridge. Clearly, the area is a center of gravity for a lot of NewSQL, NoSQL and other new-age database startups.

Here are five up-and-coming database companies — most with a “big data” flavor — to watch.

1: Akiban Technologies: This startup’s goal is to speed up database queries by putting data tables into logical groups to make it much faster for users to perform database joins. (Joins are common database operations that combine records from two or more tables.) The basic idea of table-grouping technology, which will work with standard MySQL databases, is to co-locate information that is likely to be involved in common queries together to streamline the process.

Akiban co-founder and CTO Ori Hernndstat uses an online dating site to illustrate. “Say I’m interested in all the people in my area with red hair and a fascination with dogs. When you put that in a relational system, it explodes into many tables — that’s the process of normalization.  You need all users joined with are-they-online-now, joined with regions, joined with hobbies — dogs. That is a four-way join.  [Our] technology … captures all the tables that belong to a given object or profile, and then any queries of those pieces [are] very efficient,” he said. In this example, speed is of the essence. The idea is to see who’s online right this minute that meets all these other criteria.

Hernndstat and co-founder and CEO David McFarlane, formerly with IMLogic and Nexaweb, started the company — then called Akiba Technologies — in July 2009 and at that time  year netted $6.5 million in Series A funding from Northbridge Venture Partners and Foundation Capital. It has 20 employees, all but one in Boston.

2: Paradigm4: Here’s a company that seeks to analyze “highly dimensional” data: satellite data, geospatial data, genomic data — stuff  that tends to display more as an array than as rows and columns.  Paradigm’s CTO is the peripatetic Stonebraker. CEO Marilyn Matz likes to say that Paradigm4 targets “multistructured” data where the key is to be able to look at not just the data itself, but the associated meta-data.

“When you have all this machine-generated or other data, what makes it more interesting is you also have other information about the data. Information about when it was generated, what machine it was produced on, was there a bad data point and did someone go back in and correct it? All that meta-data is valuable, and you want to keep it tied in to the data itself,” she said in an interview.

The result is often matrices and 2-D arrays — data structures that, in Matz’ view, need a better framework for complex analytics. “In operations on matrices and arrays, we’ve proven over time is, if you store all the data natively, the analytics you use will give you high performance,” she said.

Paradigm4, unlike IBM’s Netezza analytics appliance, with which it will compete, is a software-only solution, now in early beta. It will be made available in both free, open-source and commercial versions. The Waltham, Mass.-based company, with 18 employees, is backed by Sigma Partners and Kepha Partners. Matz will talk more about the big data analytics question at GigaOM’s Structure:Data conference later this month.

3: Cloudant: Historically, if a cable company or a telco wanted to know what’s going on in its far-flung network — how customers are being serviced, if they’re paying the right amount for what they get — it would collect all that data at the zillions of local endpoints, replicate the data back at a central location and then push it through some analytics.

With Cloudant, a lot of that shlepping goes away, said Derek Schoettle, CEO of the six- three-year-old company. Cloudant lets this customer instead put an instance of Cloudant at the endpoint edge, where it collects and crunches the data and then sends back the processed information rather than the unwieldy original data set.

Schoettle describes the product as a scalable “data layer as a service” built atop the open-source Apache Cloud database, JSON – a common data exchange format — and MapReduce, a popular framework for distributed data.

Cloudant’s data layer can be deployed around the world to collect, store, analyze and distribute applications — to place its processing and aggregation power where it’s needed. The company was founded by three MIT physicists and late last year brought in Schoettle, a former VP at Vertica Systems, as CEO. (Cloudant named Andy Palmer, Stonebraker’s co-founder at Vertica, to its board this week.)

4: Parelastic: How many times have we heard about an e-commerce site brought to its knees by an unexpected spike in demand?  What that usually means is that the relational database powering the site simply can’t scale up to process all those orders. Parelastic wants to crack that problem of database inelasticity while retaining the useful life of existing SQL (specifically MySQL) databases.

Towards that end, the Waltham, Mass., company built middleware that layers atop existing MySQL databases to distribute that workload as it arises. The advantage is that the thousands of people who already have MySQL databases (and skills) can keep using them.

“The problem with SQL systems was that everything went virtual except for the databases. You had all these processors, but everyone kept banging against the same database, which is why people started to abandon SQL for NoSQL. You rebuilt your database atop NoSQL to replicate what you had in SQL, but then you had to find people to deal with databases no one had heard of,” said John Landry, of Lead Dog Ventures, a Parelastic board member and investor. Parelastic seeks to make SQL databases elastic and run in parallel.  ”Hence the name Parelastic,” Landry said.

The company was founded in 2010 by Amrith Kumar, formerly VP of Dataupia, which built the Satori Data Warehousing platform, and Ken Rugg, who was SVP at Progress Software.

5: NuoDB: Barry Morris, founder and CEO of NuoDB said his company has another, NewSQL, take on distributing workloads. It uses peer-to-peer messaging to route tasks to as many nodes as needed to get the job done. “We work almost more like BitTorrent than traditional databases,” he said.

“We use the example of a flock of birds that takes off together, flies at the same time, but there’s no one in charge. It’s peer to peer. Lots of players doing very simple things, but the overall effect is impressive,” he said.

Older SQL databases are great if you can pre-provision your load, he said. They’re not great if there’s an unforeseen spike in a load. If your product is mentioned in The Wall Street Journal and your orders spike 1000-fold, that is the type of workload that is tough for traditional databases to handle. NuoDB’s messaging infrastructure divvies up and provisions loads fast across multiple nodes in a peer-to-peer fashion. NuoDB is in beta now and is slated to ship in a few weeks,  he said.

NuoDB’s co-founder and CTO is Starkey. The company is backed by Hummer Winblad Venture Partners, with Mitchell Kertzman, the former CEO of Sybase, taking an active role, as well as Longworth Ventures.

Note: This is by no means a comprehensive list of hot database companies in or around Boston. Just a few worth checking out.

Feature photo courtesy of Flickr user John Stracke

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• How data warehousing is now a cost-effective solution for businesses
• How to use big data to make better business decisions

Much has been written about the “big data” phenomenon — the petabytes of machine data from computers, sensors and other equipment; social networking data; scientific data — is a rich but unwieldy trove that is available for the taking. The big data problem is that the sheer amount and diversity of this data outmatches the abilities of traditional relational databases like Oracle, SQL Server and DB2 to handle effectively. With the Hadoop distributed data file system and MapReduce processing power, that data can be aggregated. The next step is finding tools to analyze it further.

It’s that analytics problem that has Andy Palmer excited. Palmer is a serial database entrepreneur who co-founded Vertica Systems (now part of Hewlett-Packard) and VoltDB and was a founding board member of Bluefin Lab, CloudSwitch (now part of Verizon), and Recorded Future.

“The real purpose of big data is to enable big analytics. The most compelling companies out there, I think, are those that attack that problem,” Palmer told me this week.”I really do believe that big data is, in and of itself, a tool. The real story is more about big analytics. Once you aggregate the data you then have to ask really hard questions.”

Andy Palmer

The surging interest in data analytics and visualization tools supports his take. Splunk last month filed for its IPO, and Tableau is well on its way.  Another analytics player, QlikView went public last summer, and its stock has doubled since launch, as Derrick Harris reported in GigaOM. All of these companies aim to help users make sense of all that data.

Palmer, who often works with database pioneer Michael Stonebraker, shares Stonebraker’s view that the sheer variety of data formats and the types of operations to be performed on them call for a variety of specialized databases.

There is a real need for database technology that can handle multi-dimensional data arrays — data sets that often come out of astronomy and other scientific research, Palmer said.  ”When you represent data in traditional relational databases, you can compromise the inherent nature of the data. And if you integrate a lot of data together, ultimately that data looks like a large array. Representing an array in a traditional database is really an unnatural act,” he said.

He is backing yet another Stonebraker company, Paradigm4, that is attacking that problem. In the past, the big database powers were able to shoehorn new types of workloads into their relational model. For example,  a decade or so ago, there was a raft of small, innovative object database companies — Object Design, Ontos and others — that built their businesses on the premise that relational databases could not handle objects which did not fit well into the rows-and-columns world of relational databases. Over time, however, the big data base players pushed and shoved at least some object capabilities into their databases, and those smaller companies disappeared.

Palmer and others in the big data world said this won’t happen again — that big data cannot be co-opted the same way — it would be way too expensive and resource intensive for traditional databases to try to churn through all this stuff. That’s why Oracle et al. are coming out with specialized big data products.

And when it comes to big data, the data itself will be meaningless unless the right analytic tools are available to sift through it and there are people who know what questions to ask. Big data, and the big analytics used to make sense of it, will be hot topics at GigaOM’s Structure: Data conference next month in New York City.

Photo courtesy of  Flickr user Il conte di Luna.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How data warehousing is now a cost-effective solution for businesses
• Sector RoadMap: Social customer service in 2013
• Why the big data startup boom will likely be short-lived