Speed is the driving factory behind the in-memory analytics push that spans everyone from classic Teradata rivals like SAP and Oracle to startups such as MemSQL. Estimates vary as to the exact speed difference between data access in RAM versus hard disk, but Teradata is claiming RAM is 3,000 times faster. The speed difference between RAM and solid-state drives or flash memory is smaller, although still significant.

Of course, cost also comes into play, as the speed and cost tend to go hand in hand when it comes to storage media. That’s one reason Teradata says its new technology, called Intelligent Memory, doesn’t operate fully in-memory like some competitive offerings do. Rather, it places only the “hottest” data in memory for super-fast analysis and spreads the rest between solid-state drives and disk within a Teradata environment.

This concept of intelligent data placement has been around a while in the storage space (it’s part of EMC’s new ViPR software-defined storage platform, too), but the advent of big data and abundant flash has given it some new life. Many companies desire a tiered system in which they can pay more for fast access to their important or hot data, while saving some cash on lower-performance for their older and less-accessed data. Facebook is really pushing the envelope here with its cold storage initiative — something VP of Engineering Jay Parikh will likely discuss at our Structure conference June 19 and 20 in San Francisco.

In analytics, though, RAM, not flash, is the fastest medium out there. Whether someone goes all-RAM or a tiered approach like Teradata pushing probably depends on how much performance they need across how much data, as well as how much they’re willing to pay. But if you’re doing interactive analytics in the next decade, they’re almost certain to be in-memory to some degree.

Feature image courtesy of Shutterstock user Hellen Sergeyeva.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Red-Hot Data Warehouse Market: Who’s Buying Next?
• The importance of putting the U and I in visualization
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?

The deal is focused around 600,000 smart meters in Sweden. But E.ON is one of the largest power and utility companies in the world and operates in 30 countries with 26 million customers, so clearly its a relationship with a lot of potential.

Ericsson competes with GE, Siemens, ABB and others to sell software, services and hardware to utilities and power companies. Increasingly utilities and power companies are interested in buying more data analytics tools as their networks are becoming IT networks that need to monitor, digest, and analyze data to run more efficiently. As global electricity grids add more clean energy (solar panels and wind turbines), and energy storage (battery farms), better data analytics and predictions tools will also be needed to keep the grid stable.

Some of the power gear vendors are turning to the big data startup leaders for partnerships around the latest analytics. For example, earlier this week Siemens announced that it’s working with data warehouse veteran Teradata.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Key technologies for the smart city
• Big data meets the smart grid
• Cleantech, meet connectivity: a new era of energy efficiency

Teradata sells software that can pull together disparate data from different sources and make it available to be analyzed. The company already works with utilities Southern California Edison and Oklahoma Gas and Electric. Utilities can use Teradata’s software tools to analyze data around things like blackouts, power supply and demand, weather conditions, and energy efficiency programs to help them manage the grid better.

Siemens is one of the largest power grid technology suppliers in the world, and in recent years has been bulking up its grid software assets. That move has included acquiring venture capital-backed eMeter, which manages smart meter data, as well as partnering with some of the leading grid software companies.

The union between the big data player and the power grid gorilla shows how the power grid is slowly adopting the network technologies of the internet and other IT-based networks. With grid assets like smart meters and substations increasingly becoming connected, the power grid could be one of the largest “internet of things” type networks in the world.

And to manage those connected things, utilities will need to embrace the most cutting edge data analytics tools.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Green IT Q1: Cleantech Breaking Out — and Bracing for Hard Times
• Networking the Smart Grid

Teradata also announced the new Active Enterprise Data Warehouse 6700, which comes with fast Mellanox InfiniBand networking gear and Intel Xeon X5 processors The box provides 40 percent better compute performance than the previous model, the 6690, and can handle up to 61 petabytes of data.

Teradata’s Enterprise Access to Hadoop is easier to use than mere Hadoop connectors, said Chris Twogood, vice president of product and services marketing. Business analysts can easily transfer data easily on their own, without calling on Hadoop experts, he said. At the same time, mission-critical data can stay inside the data warehouse and other data, such as tweets and log files, can stay in the Hadoop cluster.

The connection between a data warehouse and Hadoop distribution is helped along by a partnership Teradata formed with Hortonworks last year.

Several other vendors offer support for running SQL queries on Hadoop, including EMC’s Greenplum, IBM’s Netezza and Microsoft with its SQL Server. One moving part here is whether to split up appliances for data warehousing and Hadoop implementations. For Teradata, the answer to the splitting question is a resounding yes. With customers as large as Apple, eBay and Wal-Mart running the company’s gear, the Teradata way should hang around for a while.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Big data 2013: key trends and companies to watch
• Takeaways from the second quarter in cloud and data

But whatever you think about Teradata’s approach to handling big data workloads, make no mistake about the company’s clout: It has been around for decades, and it’s still analyzing boatloads of data for some of the biggest names in business. I spent a day in February touring the Teradata Labs facility in San Diego, and although I heard all about the technology and the company’s vision for a Teradata-Hadoop-Aster analytics super-environment, the thing that stuck out most were the users. Walmart, eBay, Continental … Apple.

Here’s how they’re all using Teradata and at what scale (try not to faint when you think of the bill):

• Apple: Apple is operating a multiple-petabyte Teradata system (that became apparent during its iCloud launch in 2011) and, I learned, was Teradata’s “fastest ever customer to a petabyte.” Apple uses the data warehouse to get a better understanding of its customers across product groups. Now every piece of identifiable information — and those iTunes interactiona generate a lot of data — goes into the system so the company knows who’s who and what they’re up to.

Rows of Teradata appliances.

• Walmart: The retail giant deployed Teradata’s first-ever terabyte-scale database in 1992, and it has grown, uh, a bit since then. Its operational system was at 2.5 petabytes as of 2008, and is certainly leaps and bounds bigger by now — likely well into the double digits when you consider it operates separate ones for Walmart and Sam’s Club as well as a backup system. The analytics efforts have essentially helped Walmart become a massive consignment shop. It tells suppliers, “You have three feet of shelf space. Optimize it.” And then it gives them any data they could possibly need to determine what’s selling, how fast and even whether they should redesign their packaging to fit more on the shelves.
• eBay: eBay (e ebay) has two systems in place, and they’re both big. Its primary data warehouse is 9.2 petabyes; its “singularity system” that stores web clicks and other “big” data is more than 40 petabytes. It has a single table that’s 1 trillion rows. Yes, this is smaller than the 50 petabytes worth of Hadoop capacity eBay added last year, but Teradata is quick to point out that all of its systems support data into and out of Hadoop, so it’s not as if eBay is operating two entirely distinct data environments.

Of course, Teradata has lots of other petabyte-scale customers, with Verizon, AT&T and Bank of America among them. Here are a few more interesting use cases:

• Harrah’s (now part of the Caesar’s Entertainment casino empire) understands how much money particular gamblers can afford to lose in a day before they won’t come back the next day.
• Disney is rolling out new bracelet tickets equipped with GPS and NFC that track everything visitors do while inside Disney’s amusement parks. The New York Times detailed the privacy implications of this move in a January article.
• A manufacturing customer generates 20 terabytes of data per hour while testing products, although that volume is ultimately reduced to about 1 terabyte after the valuable data is filtered out.
• At some point, Continental Airlines decided it wanted to keep its customers happy and began assessing them by lifetime value (which, it turns out, is often inversely related to frequent-flyer status) and began making alternative arrangements for them as soon as the airline realized flights would be delayed.
• A luxury car company used Aster Data to analyze the pattern of failures for various components inside its cars. It found out that lighting, seats and infotainment often failed together (they’re on the same circuit) and began inspecting all three when a customer comes in for service on any of them.

None of this means Teradata is destined to continue being a huge name in analytics (Scott Yara, co-founder of rival EMC Greenplum, recently called data warehouses this generation’s mainframe), but it’s still interesting to learn how big companies are analyzing their data, regardless what they’re running on. And with exabytes worth of data no doubt residing in customer systems across the world, Teradata isn’t going anywhere soon.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Takeaways from the second quarter in cloud and data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The fourth quarter of 2012 in cloud
• Putting Big Data to Work: Opportunities for Enterprises

As is usually the case with AWS, there is a free tier of usage available for those wanting to test the waters:

And then there’s a paid tier for production workloads:

In announcing Data Pipeline plans at AWS: Reinvent, last month, Amazon CTO Werner Vogels painted it as a way to help customers create automated and scheduled workflows of data — from Amazon’s own DynamoDB database service or S3 storage to Elastic MapReduce or wherever the data is needed (here’s where Redshift comes in.)   He promised pre-integration with AWS data sources and “easy connection” to third-party and on-premise data sources as well. It’s not clear from the post what connectivity there is to those third-party data sources now although there is mention of  copying on-premises MySQL on the list of Data Pipeline templates.

.

Also new from AWS: Fat, new instance types

On Friday, Amazon also announced a new “high storage EC2 instance family”  tailored for data-intensive jobs that need lots of storage density and fast sequential I/O handling. Such applications include data warehousing (hello again Redshift), log processing etc.

According to the blog post announcing the new family:

“High Storage Eight Extra Large (hs1.8xlarge) instances are a great fit for applications that require high storage depth and high sequential I/O performance. Each instance includes 120 GiB of RAM, 16 virtual cores (providing 35 ECU of compute performance), and 48 TB of instance storage across 24 hard disk drives capable of delivering up to 2.4 GB per second of I/O performance.”

The new instances are available immediately from AWS’ US East facility and will roll out to other regions later. Pricing for on-demand instances starts at $4.60 per hour but users can also buy one- and three-year reserved instances with prices listed on the EC2 pricing page.

Both the Data Pipeline and fat new instances show that, as customer applications continue to generate tons of data — both relational and non-relational — Amazon is determined to attack the biggest and some of the toughest big data applications around. A combination of the Data Pipeline and Redshift, if it works as advertised, could mean serious problems for big, pricey data warehouse solutions from Teradata, Oracle, and Hewlett-Packard /Vertica

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• A near-term outlook for big data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Case in point: On Wednesday at the AWS: Reinvent show, Amazon announced the preview of its Redshift data warehousing service to considerable hubbub — pitching it as a much-less costly competitor to Oracle, IBM, Hewlett-Packard and Teradata solutions. A few hours later, startup BitYota got up on stage to plug its data warehouse-as-service running atop AWS. You see where I’m going with this, right?

BitYota CEO Dev Patel

Now, BitYota CEO Dev Patel, a former Yahoo exec, says there are data warehouses and then there are data warehouses. Bityota built its Software-as-a-Service offering from the ground up with its own technology and crafted it so users won’t have to sweat how to configure compute instances or storage. And doesn’t include Hadoop, so it will sport a performance advantage there. And they won’t have to hire Hadoop eggheads, who are expensive and hard to find.

BitYota, which just disclosed $12 million seed and Series A funding from Globespan Capital, Andreessen Horowitz and others, uses Amazon EC2 compute, elastic block storage (EBS), but the rest is its own technology. A preview of AWS Redshift, which licenses technology from Paraccel, is available now. Companies can use existing analytics products from MicroStrategy, Jaspersoft, Cognos with Redshift, Andy Jassy, AWS senior vice president said at AWS: Reinvent on Wednesday.

BitYota and Redshift aren’t the only Amazon-based data warehousing games in town. Treasure Data also runs on Amazon infrastructure.

One thing is clear: Data warehouses in their current incarnation are big and expensive to build and maintain and these cloud providers all see opportunity disrupting that apple cart. ”I wouldn’t want to be Vertica right now,” said one wag at the show, referring to the data warehousing company HP bought last year.

But perhaps a bigger question for these startups running atop AWS is whether their platform provider will be more of a competitor than a partner going forward.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• Putting Big Data to Work: Opportunities for Enterprises
• In Q3, Big Data Meant Big Dollars

Redshift will cost roughly 1/10th of what old-school data warehouses would, Andy Jassy, SVP of Amazon Web Services (AWS) said Wednesday morning in his AWS:Reinvent keynote.

RT @GeorgeReese: Oracle is screwed. #AWSReinvent < LOL. Wait until Amazon does CRM and HR. :)—
James Urquhart (@jamesurquhart) November 28, 2012

Data warehouses are too expensive and too hard to scale for big companies and just plain too expensive for small businesses, Jassy said. For that reason, AWS wanted to build a service that is easy to get started on, would be self service and can scale up and down as needed.

The company tested out Redshift on Amazon.com’s giant retail business and found that it worked out to about $32,000 a year versus “millions” spent on its old-school data warehouse. Traditional data warehouse applications can cost $19,000 to $25,000 per GB per year compared to $1,000 per GB TB per year for Redshift. Update: Redshift builds on technology licensed from Paraccel.

Of course the proof is in the pudding. Customers can sign up for the preview now.

Amazon execs are hell bent on showing that their services are ready for primetime even for mission-critical applications. But last week,  two IT execs, one from Diebold and another from a big US bank, both told me that no one in their companies, not even developers, can use AWS at all for compliance reasons So there’s still some work to do there.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• 2012: The Hadoop infrastructure market booms