President and COO George Mathew says the company’s mission is to be a one-stop shop for statistical analysis. It wants to be the one place where analysts and data scientists can blend their data, model it on it and then visualize it. Often, he noted, that same process might require two or three separate products.

Another feature that Alteryx hopes will set it apart is its collection of prebuilt models in what the company calls an analytics gallery. Users can share their own work or find models others have built for tackling similar issues. Alteryx also offers up its own pre-formatted datasets for analysis, often public data such as the U.S. census.

“The canvas for creating an analytics application should never be blank for the analyst when they’re getting started,” Mathew explained. They often need to understand external data as well as their internal data, so Alteryx’s software gives them easy access to it.

Because it’s based on the R statistical-programming language, heavy R user Walmart has been able to transition some workloads to Alteryx when employees need an easier user experience. McDonald’s uses it to analyze data about franchisees and about its growth strategy in China, and Bloomin’ Brands (parent of company of Outback Steakhouse and other restaurants) is using it to help build menus that take into account what diners in various parts of the country prefer to eat. Nine of the 10 leading top wireless providers providers are also users, Mathew said, trying to blend actual call data with traditional sources such as customer service data.

Mathew compares Alteryx’s current growth as analogous to software-as-a-service applications like Salesforce.com in the CRM space, or even Tableau in the traditional business-intelligence space. In a business world increasingly driven by at least the idea of big data, one might expect any vendor pushing a more consumer-like purchase and consumption experience to get interest from companies tired of dealing with legacy software or never wanting to experience it in the first place.

“The disruption that’s happening is creating a new space for ourselves,” Mathew said, “without having to go head to head, frankly, with the a status quo out there.”

Feature image courtesy of Shutterstock user ramcreations.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Internet of things: creating tomorrow’s health care
• Dissecting the data: 5 issues for our digital future
• Connected world: the consumer technology revolution

“We’re thrilled,” Tableau co-founder and CEO Christian Chabot told me during a call after the market closed. One should hope so.

Chabot and his fellow co-founders stand to make a lot of money if today’s closing price holds up, as does its sole investor NEA. The firm put $15 million into Tableau since it launched in 2003, and has rode that sum to profitability and more than $127 million in annual revenue.

Here’s a quick chart (made using Tableau Public) showing who owns how many share and what they’re potentially worth.

The company didn’t really need more capital to operate, Chabot said, but one of the primary drivers was to raise awareness of the company. It has about 12,000 customers, he said, but there are millions more possible users. As part of attracting them, the company is going to expand globally and is working to improve its reach across mobile devices, the cloud and the Mac operating system.

“I don’t believe in the this whole ‘or’ philosophy with computers,” Chabot said. “It’s ‘and’” — meaning people will use desktops and tablets and smartphones.

More prominence and more users singing its praises might also dispel the notion that Tableau is just about visualization. It has some fairly advanced features under the covers (as a commenter to my earlier post about the company’s influence pointed out), even if they’re hidden by the relatively simple user experience.

“Tableau is not a visualization company, per se, it’s really an analytics company,” Chabot said.

However, if the company really wants to expand its reach to everyone one who wants to gain knowledge from data — something Chabot calls a “timeless human need” — it might actually need to get simpler. More marketing can let potential business users know about new features like forecasting and data-extraction, but it won’t make a dentist is Des Moines better at formatting his data.

After raising $254 million in its IPO, though, Tableau is in a good place to do whatever it has to.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• The importance of putting the U and I in visualization

That’s an apt ticker symbol for a company that is in some ways a bellwether for the current fascination with all things data. Tableau isn’t a big data company, per se, but its visualization software breathes life into many big data calculations. Its focus on making software that’s easy to use and that creates visually captivating charts has turned people from numerous professions into amateur data analysts. (I’ve even used it in the past, including for the first time in 2011.)

Christian Chabot

As Chabot told me during a conversation in 2011, “In any field of human endeavor … there are a hundred to a thousand more people who understand the data of that field more than they understand reporting and analytics.”

Anytime you read about a hot new visualization or analytics startup promising the moon, you’re also seeing the results of what Tableau has sown in terms of the user experience. Many of those same companies will be quick to tell you how limited Tableau’s capabilities are. It’s memory-bound, it doesn’t have a database, it’s not available in the cloud (or on the Mac operating system), it can’t do predictive analytics. All true.

Of course, if it raises the kind of capital it expects to by going public, it can build and buy a lot of those capabilities. If pricing stays flat all day Friday, Tableau stands to make $155 million from its 5 million shares. Previous estimates had Tableau’s market cap at around $1.7 billion at a price of $29 per share (the company’s S-1 filing is available here).

If investors have really bought into the company and the concept of a data-driven world, then who knows. Machine-data expert Splunk wnet public in 2012, flying the big data banner, and saw shares peak at 91 percent above its original asking price of $17.

I’m not suggesting Tableau is the biggest name in data, or even that it will some day become it. This next-generation analytics field is very young, with startups and larger vendors alike sometimes competing against themselves to win wholly new accounts than trying to displace legacy vendors within large enterprises. And every month, it seems, I come across some new startup that was built with the same principles in mind as Tableau, but with the advantage of having today’s best practices baked into its software.

But Tableau definitely commands a lot of the mindshare. How it fares as a public company could be a strong indicator of just how powerful the data movement is, and how well it capitalizes on a new influx of cash will determine how long it stays on the top of customers’ minds.

This post was updated at 7:01 p.m. to include previous estimates of the company’s market capitalization and a link to its S-1 filing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 4 iPad apps to help wrangle data
• Health care and big data in 2012

Storage, databases, query processing and algorithms are all extremely important — heck, visualization is next to nothing without them — but in a data-driven world where is obsessed with insights, they’re just the foundational layers. They are to big data what server and network configurations are to mobile-app development on platforms like Parse. If you’re going to find out new things from massive and highly complex data sets, or going to give new types of people the ability to analyze even simple data, the presentation of that data and the ability to create consumable presentations are critical.

With that in mind, here are six startups I’ve seen trying to fundamentally change the way that data is visualized. Some are highly complex under the covers, some are not and none are perfect, but they’re all doing their part to make us rethink what it means to look at data and make spreadsheets and static charts look like relics. (And this list is by no means exhaustive, so feel free to add your favorite visualization tools in the comments.) We’ll be highlighting data visualization at our design-focused RoadMap conference in San Francisco in November (sign up here to get first access to tickets this Summer).

Ayasdi

The idea of network graphs isn’t new, but Ayasdi’s approach to it is. Under the covers, there’s an HBase data store, a technique called topographical topological data analysis and hundreds of machine learning algorithms to churn through complex data sets and determine the similarity among the data points. To the end user, though, there’s a map of the data set that looks a lot like a network graph (only it’s probably not network data) highlighting clusters of related data points that analysts might want to investigate further.

BeyondCORE

BeyondCore actually operates under the same basic premise as Ayasdi — show users the significant correlations so they don’t have to think of the queries that will uncover them — but it uses some different techniques to get there. It uses a different visualization method, too: BeyondCore sticks to standard charts, but actually offers the option of having an avatar talk users through the correlations the software has discovered.

ClearStory

ClearStory has a pretty unique product in the works — even if it’s keeping many details and all of its screenshots under lock and key until its formally launches. Essentially, though, it’s trying to tell stories via visualizations that display mashups of numerous data sources, update automatically when the source data changes, and invoke collaboration and social concepts. Here’s Co-founder and CEO Sharmila Mulligan explaining the idea behind ClearStory at Structure: Data in March.

Datahero

Unlike so many data startups, Datahero isn’t trying to woo people fed up with business-intelligence software or the difficulties of getting insights from Hadoop data. Rather, it’s trying to let people with simple business or personal data make simple charts without ever having to enter an Excel function or worry too much about how their spreadsheets are formatted. Early on, Datahero’s visualizations are still pretty commonplace (bars, pies, plots, etc.), but it’s the ease of creating them that’s so unique.

Platfora

Platfora has undertaken the ambitious task of trying to make analyzing mountains of data stored in Hadoop clusters as easy as analyzing their own Stripe data might be for developers using Datahero. It’s based on a foundation of Hadoop and massively parallel query processing, but is presented like an HTML5 version of current visualization golden boy Tableau that’s all about dragging, dropping, and visually slicing and dicing through data. The latter capability is actually critical in a big data world where there are likely more data points than you can ever digest at once.

Zoomdata

Zoomdata is far from the only analytics company to support mobile devices, but it’s one of the few I know of (Roambi also comes to mind) designed primarily for them. Zoomdata connects to standard business data sources, but takes advantage of touch screens and the D3.js visualization project to offer up some visually interesting charts that are designed to be manipulated like an artist’s palette.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Cloud computing and trickle-down analytics
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Taken alone, ScraperWiki isn’t that big of a deal, but it’s part of a huge revolution that has been called the democratization of data. More data is becoming available all the time — whether from the government, corportations or even our own lives — only it’s not of much use unless you’re able to do something with it. ScraperWiki is now one of a growing list of tools dedicated to helping everyone, not just expert data analysts or coders, analyze — and, in its case, generate — the data that matters to them.

After noticing a particularly large numbers of tweets in my stream about flight delays yesterday, I thought I’d test out ScraperWiki’s new Twitter search function by gathering a bunch of tweets directed to @United. The results — from 1,697 tweets dating back to May 3 — are pretty fun to play with, if not that surprising. (Also, I have no idea how far back the tweet search will go or how long it will take using the free account, which is limited to 30 minutes of compute time a day. I just stopped at some point so I could start digging in.)

First things first, I ran my query. Here’s what the data looks like viewed in a table in the ScraperWiki app.

Next, it’s a matter of analyzing it. ScraperWiki lets you view it in a table (like above), export it to Excel or query it using SQL, and will also summarize it for you. This being Twitter data, the natural thing to do seemed to be analyzing it for sentiment. One simple way to do this right inside the ScraperWiki table is to search for a particular term that might suggest joy or anger. I chose a certain four-letter word that begins with f.

Surprisingly, I only found eight instances. Here’s my favorite: “Your Customer Service is better than a hooker. I paid a bunch of money and you’re still…” (You probably get the idea.)

But if you read my “data for dummies” post from January, you know that we mere mortals have tools at our disposal for dealing with text data in a more refined way. IBM’s Many Eyes service won’t let me score tweets for sentiment, but I can get a pretty good idea overall by looking at how words are used. For this job, though, a simple word cloud won’t work, even after filtering out common words, @united and other obvious terms. Think of how “thanks” can be used sarcastically and you can see why.

Using the customized word tree, you can see that “thanks” sometimes means “thanks.” Other times, not so much. I know it’s easy to dwell on the negative, but consider this: “worst” had 28 hits while “best” had 15. One of those was referring to Tito’s vodka and at least three were referring to skyline views. (Click here to access it and search by whatever word you want.)

Here’s a phrase net filtering the results by phrases where the word “for” connects two words.

Anyhow, this was just a fast, simple and fairly crude example of what ScraperWiki now allows users to do, and how that resulting data can be combined with other tools to analyze and visualize it. Obviously, it’s more powerful if you can code, but new tools are supposedly on the way (remember, this is just a beta version) that should make it easier to scrape data from even more sources.

In the long term, though, services like ScraperWiki should become a lot more valuable as tools for helping us generate and analyze data rather than just believe what we’re told. Want to improve your small business, put your life in context or perhaps just write the best book report your teacher has ever seen? It’s getting easier every day.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• 4 iPad apps to help wrangle data
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?

So it took that data and tweaked its approach. Now when it sees a slowdown it reaches out to the client to try a new plan or just make sure the clients don’t cancel. In the future it may offer new pricing plans to adjust for slack usage.

That’s just one way connected devices and the data they offer can be used for benefitting a business. But the value of constant connectivity to a firm goes far beyond that — and could change the way businesses operate. Even after a product goes out the door, the company responsible can still keep an eye on it. That has big repercussions for business and consumers — and not all of those repercussions may be welcome.

Always be talking … to your device.

For example, the constant contact can also help tweak a design or improve the function of a product — even out in the field. In a recent conversation, Splunk’s Tapan Bhatt walked me through a few examples such as the one above, where the company’s machine logging data helped businesses adjust. For example, the makers of the Nest thermostat use Splunk to analyze data uploaded from hundreds of thousands of homes, and tune their algorithms for energy performance.

Medical device manufacturer iRhythm uploads remote monitor data to Splunk to make sure devices run as expected, as well as help ensure that patients can use the devices intuitively. In many ways this isn’t new. Jeremy Conrad at Lemnos Labs pointed out to me in a conversation last month that many manufactured devices are tweaked again and again after the first manufacturing run to smooth out perceived and real flaws in the design.

The shift is that it can now happen constantly and that the changes might be implemented weeks or months after the product has been manufactured. Advertising firms and online publications have been using such data to refine their products for years. The Huffington Post’s love of A/B headline testing is well documented, while the use of eye tracking in web site design is a common practice. But more connectivity in devices means the fine-tuning and easy tracking that are common in digital products are now available in the real world.

Want to tweak a feature? Send out some software

Connected devices not only offer you the ability to get data from your goods (while software like Splunk’s helps you log and later analyse it), but it also allows you to change how they feel and function. For example, Orbotix, the company that makes the Sphero not only knows the moment someone activates one of the Bluetooth-controlled balls, but can give it new abilities with an over the air update.

This connectivity and resulting data can also help with business goals, like improving manufacturing, anticipating demand and even holding reviewers accountable for their articles as was the case when The New York Times and Elon Musk, the CEO of Tesla got in a public battle over a poor review of the electric car.

The board at the Orbotix HQ that tracks all the active Spheros in the wild.

At Orbotix a billboard in the office tracks how many Sphero’s were activated that day, that month and even over longer periods of time. If you stand in front of it for a few moments the numbers will change. Paul Berberian, the CEO of Orbotix told me that during the holiday season the numbers were changing so fast it was hard to keep up. During the rest of the year evenings and weekends were popular times for seeing the numbers flip more rapidly.

Just in time manufacturing gets a data infusion

As this data accumulates he’s finding that he can better anticipate demand and plan inventory to meet it. Perhaps if he wanted to, he could implement a similar program to that elevator company, watch the data from individual Sphero’s and when interest seems to wane perhaps the company sends a notification to the user about a new app available for the ball.

But it’s not always about the customer — this data can be used to monitor manufacturing partners or suppliers. For example, Electric Imp, which makes a tiny module that device makers can insert into their products to give it connectivity (it’s a radio with access to a cloud back end), connects its modules as they come off the line. One of the final steps in the packaging process is each module gets an ID laser-etched onto it.

This process requires the module to “wake up,” connect to its virtual machine in the cloud to get its ID number, and then tell the laser etching machine (which has its own Imp module) what number to print on it. As part of this process Electric Imp’s management can track all of its modules off the manufacturing line and get key information about yields and even product theft.

Of course the flip side of this constant connectivity is the disquieting sensation that even as you enjoy a product it’s not yours. It’s features might change at any point. Perhaps things you love about the product or even features you’ve purchased, might suddenly disappear. As a consumer, the idea of dynamic pricing can seem exciting if you don’t use something a lot, but it becomes a source of higher costs if you have a building with very active elevators, for example.

And perhaps most unsettling is the realization that these products can act as a doorway into your home, sharing information that perhaps you’d rather it didn’t. Your car tracking your trips. A toy that knows if you’ve skipped school to play video games. It’s unsettling enough that this happens on the web and with our phones. As this capability hits more devices, we may find ourselves taking the stairs instead of a connected elevator or playing with an old-fashioned doll instead of a Bluetooth enabled ball.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Internet of things: creating tomorrow’s health care
• Connected world: the consumer technology revolution
• GigaOM Research highs and lows from CES 2013

While we may never get the full extent of government spying on citizens in machine-readable formats, the Census, FDA testing, EPA and myriad other data will offer a treasure trove of information for years to come. And by making it open and machine readable, it will presumably be harder to bury such data in physical warehouses or behind crazy fees.

The executive order is here. Essentially it requires the government to do the following:

• Figure out what data the agencies have, and make an index
• List all of their publicly available data in a public place
• List all of the information that could be made public, even if it is not yet available

But the order attempts to address an almost existential question about moving from an organization where data is assumed TO BE hidden to one where it assumed to be open. John Wonderlich over at the Sunlight Foundation put it well in his blog on the announcement:

“Most importantly, though, the new policies take on one of the most important, trickiest questions that these policies face — how can we reset the default to openness when there is so much data? How can we take on managing and releasing all the government’s data, or as much as possible, without negotiating over every dataset the government has?

How can the public (or policymakers) request what they don’t know exists? How can CIOs manage what they haven’t surveyed?”

He concludes that this order will address a lot of these issues, and I hope that will actually happen. As someone who has submitted Freedom of Information Act requests only to get back boxes of redacted and almost meaningless documents, the hunt for government information — or information that is supposed to be publicly available — can be daunting, exhausting and ultimately fruitless.

Of course, as was shown when the SEC started making its records available online using XML, the greater visibility of those documents, notably the Reg-D filings that indicated a private company had picked up funding, prompted the agency to include less information in those documents. They still made them public online, but also made them less useful in some cases.

Also, this order notes that privileged information, law enforcement information, national security information, personal information, or information that agencies can’t disclose because it is prohibited by law, are all off the table when it comes to the order. This isn’t unexpected, but it can be used to create loopholes where agencies (or private companies working with the government) can attempt to hide data it doesn’t want to share.

But, as the White House release notes, government data such as Global Positioning System data and weather data have been open for decades and have helped create some awesome new services for citizens. I’m sure that today’s news will open up plenty of great data sets that entrepreneurs can start using to build amazing new apps.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Readers weigh in: future prospects for Twitter
• AWS Storage Gateway jolts cloud-storage ecosystem
• 4 iPad apps to help wrangle data

Our mobile devices are amazing at capturing real-world data — location, temperature, movement, sound — that just goes to waste if we don’t put it to use. It’s easy enough to get a personalized experience on the web, but these types of data might make it possible to get one in traditionally more-static places such as retail and radio as well. At the least, perhaps we can expect content, price tags and experiences that cater more to our actual tastes than those of station programmers and a fashion designer’s idea of what people should be willing to pay.

Location is the key to everything

Retailers already have a pretty good sense of what people are buying and even how they’re moving through stores, but they don’t really know where customers are going once they leave. This knowledge could be very useful, however: If you want to improve your store or figure out how to market your company, knowing what else your customers are up to could go a long way. This type of data is starting to become available thanks in part to a Seattle-based startup called Placed.

We’ve been covering Placed for about a year, since it launched its first product targeting developers interested in learning where users were accessing their applications and mobile sites. The company has since expanded its operations to include a Panels service that lets the company track around the clock, on behalf of paying businesses, the physical location of customers who have downloaded the app (usually in exchange for a small monetary reward). It also has its own Panels app, unaffiliated with commercial customers, that allows Placed to provide market data on the physical movements of some 70,000 consumers.

This week, the company released a report highlighting some national findings from the first quarter, including, for example, what departments stores are most popular with what demographics, what business categories experienced the most increases in traffic, and what businesses have the highest and lowest affinities (i.e., people who visit one also visit, or don’t visit, the other). If you’re willing to pay, Placed will tell you pretty much anything you want to know, founder and CEO David Shim told me, broken down by geographic region, business type, demographic, you name it.

Shim noted a couple of actual users and potential users that I think highlight why this type of data is so valuable. One is a high-end retail business that found out that while female millenials enter its stores a lot, they don’t buy a lot. Rather, the stores they visit next are usually discount retailers such as Burlington Coat Factory and Ross. The suggestion is clear: These shoppers want to see what’s hot and then buy a reasonable facsimile at a lower price.

He also noted that some Las Vegas casinos are interested in running their own Placed panels to figure out what restaurants their guests are eating at once they leave the casino grounds. Now, if casinos can figure out where else on the Strip people are spending their money, they can make better choices when it comes time to swap out their own restaurants and shops.

In both cases, it’s possible the answer to the question of how to get more of these customers’ money is to drop prices. If a 10 percent price reduction leads to a 14 percent increase in sales, that’s a win-win situation.

Rethinking radio

Location data becomes even more valuable when combined with other data, though, such as sound. Consider the implications of knowing not just what radio stations people are hearing — which is essentially how the Arbitron ratings work — but what songs they’re actually listening to. Just because you hear the Latino station for an hour at the taco shop during lunchtime or the top 40 station at the gym, that doesn’t mean you’re listening to them or like listening to them.

But the songs you choose to listen to in your car, for example, probably tell a lot about what you actually like. And the technology exists to determine that. Last month, I wrote about how Gracenote is able to use the internal microphones on tablets and smartphones to recognize the songs playing on people’s televisions or stereos. It can also detect reactions such as cheering or booing, and likely whether someone turns up the volume.

Arbitron’s Portable People Meter

Now, all of a sudden, one can envision a world in which programming managers at radio stations can figure out on a song-by-song (or artist-by-artist) basis what people are actually listening to, and when and where they’re listening. If all it involves is someone downloading an app, they can presumably do it at a larger scale than requiring people to wear special additional sensors or fill out a diary. Broadcast radio can never be as personalized as something like Pandora, but it could start sounding a lot more like what listeners would choose if left to their own devices.

Digital radio could get downright great, even better than what Pandora can currently offer. I might never add Disney theme songs or the Sesame Street favorites to my preferences list, but if that’s all I listen to when I’m in my car between 4:30 p.m. and 5 p.m. — and it is — maybe a service could hook me up with some new songs every day. If I’ve turned up the volume on a Taylor Swift song three times this week while I was at home, maybe I actually like it and want to hear more even if I won’t admit it.

Not just data, but good data

As great as all this might sound (it does to me), it’s the advent of big data that makes it all possible. Placed’s analytics are so accurate because it has special algorithms to determine where a person actually is — even if there are numerous options within a small area — and its models are constantly being trained. Shim said his company gets 15,000 responses a day to surveys asking Panels users whether it had them at the right location, and it has already validated 3.5 million of the the 13 billion locations in its database.

Gracenote, for its part, has audio and metadata for millions of songs that it keeps in memory so it can access them in a hurry for the sake of real-time recognition. It can group music into dozens of categories based on genre, artist, geography or even just how the songs sound. It wants to build an in-car system that can change songs based on driving conditions fed to the stereo from the car itself.

I acknowledge this all sounds a little creepy, but, ironically, it also sounds like the beginning of the end for some concerns over privacy. Heck, Shim said, about 500,000 people have already downloaded the Placed Panels app.

Really, it all comes down to value. If handing over a little bit of data actually provides value in return — in the form something better than just targeted ads — it appears people will be willing to do so. People tell Amazon about their purchases, let Google Now access their email and tell Placed which store they’re at out of five possibilities because they think they’re getting a worthwhile service in exchange.

The data-collection genie is already out of the bottle. Now it’s just a matter of making it work for us instead of at our expense.

Feature image courtesy of Shutterstock user Vadim Georgiev.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

Obviously, data is nothing new to the automotive industry — companies have been trying to optimize supply chains and analyze sales numbers for decades — but the advent of big data, as well as related technlogies such as sensors and smartphones, is changing how companies are thinking about data. Ford isn’t alone in its quest to take advantage of these new technologies, either. For example, General Motors collects data from its OnStar system to help lower drivers’ insurance premiums, and also collects lots of data on its Chevrolet Volt electric car that it feeds to drivers via a mobile app. We recently noted how a luxury automobile company used big data software from Aster Data Systems to determine the relationships between malfunctions so it could provide a more thorough and beneficial service-department experience.

But in an industry notoriously unwilling to talk about information technology, Ford’s experiences might shed a lot on what other companies are thinking and doing, as well.

Building a better experience through data

According to John Ginder, manager for systems analytics with Ford Research & Innovation, the company has been doing advanced business modeling for about 20 years, but big data is something else. Today’s technologies are allowing Ford to handle larger, more-diverse datasets than ever before possible, and its efforts are already beginning to bear fruit in numerous places — including in the cars themselves.

The most obvious example of data influencing the driving experience might be the types of data car companies are actually giving back to drivers. At Ford, its Energi line of plug-in hybrid cars generate 25 gigabytes of data per hour that’s then processed and given back to drivers via a mobile app. It tells them about battery life, the nearest charging stations and other data about the vehicle’s performance.

The MyFord mobile app architecture.

Ginder said all that data is the result of a “convergence of need and opportunity.” The opportunity is a way to experiment with collecting and presenting vehicle data on a group of early adopters that’s probably more interested in this type of advanced technology. The need has to do with what Ginder calls “range anxiety” — when drivers are getting used to electric vehicles, they need reassurance they’re not going to run out juice.

However, Ginder said, the company is just scratching the surface of what’s possible, because there aren’t that many of the electric vehicles on the road yet. The goal is to better understand how drivers are using the vehicles and use that information to continuously improve the vehicles and the overall experience. Ford’s Super Duty line of pickup trucks also offers a “crew chief” package that lets bosses monitor the fuel consumption, engine performance and other data about their fleets of vehicles.

Mike Cavaretta, technical leader for predictive analytics and data mining with Ford Research & Innovation, added that Ford is really interested in collecting more data from more vehicles, but noted there’s also a privacy concern that could come into play. The potential of someone knowing where and how you’re driving might not appeal to the mainstream just yet (just look at all that data Tesla collects about its cars and can present if it really wants to), but as with the Energi, data does present some opportunities to improve the customer experience.

The test cars in Ford’s research labs are collecting about 250 gigabytes of data per hour from high-resolution cameras and an array of sensors, Cavaretta noted, and the company is trying to find out what data is most useful and how it might be rolled into production vehicles.

Building betters cars through data

Of course, sometimes the best data isn’t the stuff you see, but the stuff that just makes your car better. Cavaretta said Ford analyzes a lot of social media and other external data in order to figure out, for example, what customers are saying about their vehicles compared with other makes and what problems they’re having.

Opens with the touch of a foot. Source: Ford

In one recent case, the product development team was curious as to whether the Ford Escape sport-utility vehicle should have a standard liftgate (i.e., it opens manually and the rear window can flip open) or a power liftgate in which the glass and the gate are one piece. In the latter option, the gate opens automatically by tapping under the rear bumper with your foot, but the window doesn’t open at all. Regular surveys hadn’t addressed the question, so Cavaretta and his team took to social media, where people were actually talking about it quite a bit and seemed to heavily favor the power liftgate in most cases. It’s now a feature.

Back in 2004, Ford built a self-learning neural network system for its Aston Martin luxury brand that maintains proper engine function by recognizing engine misfires and particular driving conditions and adjusting warnings and performance accordingly.

Ginder said his team has been improving on that technology ever since and actually expanded its use into a system, called Smart Inventory Management System, that lets dealers ensure they have the optimal stock of vehicles and features on their lots. Historically, he said, some dealers were very sophisticated about inventory management, while others were more reactionary (“They just sold a red Mustang,” he joked, “so they think they need to go order another red Mustang.”) With SIMS, all sorts of data about vehicle sales and other locally relevant data from across the country is aggregated in Ford’s big data platform, and the neural network algorithms learn the current patterns so Ford can make better recommendations — whether or not dealers choose to heed the advice.

Selling big data internally

Cavaretta characterizes the division in which he and Ginder work as “an Ernst & Young, but just for Ford,” an internal consultancy (as opposed to Ford’s more-traditional research and development division) in charge of solving business problems via analytics. About 80 percent of those problems come directly from those lines of business, while about 20 percent are the research division’s own ideas. However, although he’s excited about how big data can help his team answer these questions in novel ways, it’s not always an easy sell with other parts of the company.

Mashing up data sources such as social and sales in order to find insights is a pretty easy sell, Cavaretta explained, but getting people to put sensors in everything and collect data every second or with every transaction can still be a bit challenging. In part, this is just a lingering effect of the constraints that legacy technologies imposed on the company. It wasn’t possible to store all this data, so people just got accustomed to the status quo of summarizing data hourly, for example.

Source: Ford

Now, however, he’s pushing them to “dial it down” and collect data at the lowest level possible and as often as possible. In manufacturing alone, he explained, there are between 20,000 and 25,000 parts in any given vehicle, and there’s a supply chain that spans from parts suppliers all the way up to dealerships. Getting a complete view of this process could help drive serious efficiencies and, Cavaretta said, “We don’t see anything but big data technologies that can get us there.”

Other areas where Ford is collecting, or wants to collect, more real-time data is from websites, call centers and the company’s credit-processing arm, he added.

Building big data internally

In order to accomplish their lofty goals, the Research & Innovation analytics team relies heavily on open source technologies, most prominently Hadoop. However, Cavaretta said, they’ve been experimenting with a variety of natural-language processing tools, too, and even did a proof-of-concept with SAP’s HANA in-memory analytic database. The NLP tools were first turned on text analysis of internal surveys and dealer network documents, but now are used pretty heavily on social media and other web data.

Their team has some systems numbering in the dozens of nodes in its own building, but on weekends it’s able to borrow high-performance computing cycles from Ford’s Numerically Intensive Computing Center next door in order to model recommendation engines and other tasks that demand serious computing power.

But as a part of a specialized research division, the work that Ginder, Cavaretta and their team do on everything from Hadoop to visualization with tools like Tableau isn’t automatically ready for primetime. In fact, Cavaretta said, it looks at “what’s the art of the possible” and tries to show the value of it. It’s like a vanguard, he added, going out and seeing what’s ahead and then reporting back.

At that point, projects are often handed off to Ford’s central IT team that actually puts the technologies into production. A system that took the research team weeks to deploy and start deriving insights from might take IT months to make production-ready. However, Ginder added, his team can’t just throw stuff over the wall and abandon it — it has to collaborate with the IT team and individual departments throughout the project’s lifecycle.

An important part of this cross-company relationship — and something many CIOs have likely heard before — is having data scientists on board that can see the world through the eyes of both technologists and businesspeople, two groups that often have different concerns and goals in mind. “We look for people who can bridge those worlds,” Ginder said. “It’s hard to find these people, but they’re hugely important to organizations.”

Feature image courtesy of Shutterstock user PhotoSmart.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• The importance of putting the U and I in visualization