Jodok Batlogg used to be the chief technology officer for StudiVZ, Germany’s big Facebook clone of the late 2000s. It was there that the Austrian discovered the joys of scaling, and where the seeds of his current startup Crate — a zero-administration data store firm and a finalist in this year’s Structure Launchpad competition — were sown.
“We were growing from a few million users to 16 million users. We had around a thousand servers and had to solve some big data problems [handling] billions of small files, connections between users and high concurrency,” Batlogg told me. After he left the social network, Batlogg set up a consultancy specializing in high-volume traffic, and ended up spinning out the back-end technology as its own product.
Crate is an open-source tool for setting up and automatically managing a distributed, self-configuring, self-healing data store with a SQL interface for real-time query and search. It builds on components such as Elasticsearch and Presto, and the focus here is on simplicity (Batlogg says he was particularly inspired by Elasticsearch in that regard) and scalability achieved through a “shared-nothing” distributed architecture.
“Our main target was high concurrency, where you have multiple read and write at the same time – millions up to billions of records – and all of this on commodity hardware,” Batlogg said.
The internet of things is a key sector for Crate, though it’s gunning for any vertical that involves a lot of concurrency and fast results, such as user-facing apps that draw on big data. Batlogg and co-founders Christian Lutz and Bernd Dorn (Batlogg is picture above at left with Lutz) claim typically sub-second response times – faster than Hadoop but a bit slower than in-memory technology, which is in any case generally a pricier option.
As an open-source product, Crate is of course free to download and use. The company charges for support, but on a per-cluster rather than per-node basis in order to be more scaling-friendly. It also gives discounts to startups, charging $199 per cluster per month rather than the usual $499.
Crate’s approach certainly impressed backers Sunstone and DFJ Esprit, who put $1.5 million into the company in late April. You can see what Batlogg has to say on Wednesday, 18 June, at our Structure conference in San Francisco.