Comments on: Amazon preps Data Pipeline service to automate big data workflows

By: amuletc

amuletc — Sat, 01 Dec 2012 07:35:13 +0000

Whoa, now that’s something I can use – data pipeline, great idea.

By: Derrick Harris

Derrick Harris — Thu, 29 Nov 2012 23:26:47 +0000

Actually, workflows don’t need to touch EMR. If you have in-memory DBs in EC2, or are using Redshift, for example, you can move data there too. Or they could go from EMR to an in-memory system. It’s really whatever flow you want, I’ve been told.

By: Mark Chmarny ✔

Mark Chmarny ✔ — Thu, 29 Nov 2012 21:38:51 +0000

As in-memory data grids become the backbone of next-generation on-line applications, their dependency on any specific data storage technologies will become less relevant. AWS’s Data Pipeline Service could cross the divide between local/cloud data and allow HDFS to become the consolidated data storage platform of choice.

More on this here: http://mark.chmarny.com/2012/11/hdfs-has-won-now-de-facto-standard-for.html

By: Chris Taylor

Chris Taylor — Thu, 29 Nov 2012 18:52:16 +0000

This is a great step forward for those who store Big Data sets in the Cloud. It will reduce the administration required to do periodic batch runs of data sets, for sure.

But is Amazon looking at ways to offer in-memory services that would reduce the need for batch schedules? That would be an interesting proposition.