And now, if you have up to a million rows of these types of data and are on a newish Windows machine, GeoFlow will let you plot the data over Bing Maps. The visualizations actually look pretty good, and time lapse capabilities are always good when time is a variable.

I also like Microsoft’s focus on using the tool for storytelling, citing the ability to share findings through “cinematic, guided video tours.” Even though visualizations are getting much better, more impressive and interactive — especially online — telling a good story with complex data is still pretty difficult.

Of course, GeoFlow is only in Preview mode, so it’s possible there are kinks to work out. And, who knows, maybe early users will roundly dismiss it.

One of California ISO’s massive control rooms

But it’s definitely trying to solve an increasingly important problem — something you can see at a far greater scale in the work the California ISO is doing with a startup called Space-Time Insight. Other startups — like TempoDB and SpaceCurve (see disclosure)– are even dedicated to building databases to address the oncoming deluge of this data.

GeoFlow actually comes from much larger-scale project, too, for what it’s worth. In a Microsoft Research blog post on Thursday, the company talks about its roots as part of the Worldwide Telescope project that let users explore a high-resolution, interactive map of the universe.

Disclosure: Reed Elsevier, the parent company of science publisher Elsevier, is an investor in SpaceCurve as well as Giga Omni Media, the company that publishes GigaOM.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• The 2013 task management tools market

We’ve covered Neo and its technology, a commercial version of the open source Neo4j graph database, before. Graph databases store data based on their relationship to one another, making them ideal for organizing massive amounts of information such as social networks, the connections between various web pages and web data, or, in the case of Neo Technology customer Cisco, its master data management system. Neo4j, Neo Founder and CEO Emil Eifrem told me last year, is also ideal for serving transactions and as part of a Java application stack, which helps its explain why VMware’s SpringSource division is backing the project.

Neo’s customer base also includes Deutsche Telekom, Lockheed Martin, BAE Systems, LexisNexis, Adobe and Pitney Bowes, as well as a slew of web startups.

Aside from graph databases, those focused on storing data relating to time and place are also catching fire. In August, spacial-temporal database startup SpaceCurve raised a $3.5 million second round for its technology that the company claims can make short work of uncovering patterns in complex geospatial and time-series data. TempoDB, a startup offering a cloud database service for time-series data like that thrown off of sensors and electrical equipment, has raised almost $900,000 in seed funding. Space-Time Insights, a company that specializes in visualizing organizations’ geospatial and time-series data just raised $14 million.

This, of course, is only a sampling of the types of databases that are garnering attention as we generate more data from more sources and need new ways of storing, accessing and analyzing that data. The relational database isn’t going away any time soon, but it looks like it might soon be sitting alongside a variety of databases designed for other tasks within a lot of corporate data centers.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Emerging trends in the non-relational database market
• AWS Storage Gateway jolts cloud-storage ecosystem
• A near-term outlook for big data

We first covered Space-Time last year, when the California Independent System Operator deployed the company’s software to power a massive control room featuring an 80-foot screen that displays map views and real-time information from the ISO’s energy infrastructure. Among the ISO’s duties is managing approximately 25,000 miles of power lines, and Space-Time helps it get a view of what’s going on across all that distance.

The California ISO mission control room

However, Steve Ehrlich, Space-Time’s senior vice president of marketing, told me during a recent phone call, Space-Time is starting to see traction in other related areas such as oil & gas exploration and transportation. If organizations have critical assets that are geospatially located, he said, Space-Time can help them take in weather, social media, enterprise, and other data and correlate and visualize it “in ways they’ve never thought about before.”

Once they get started, they just keep thinking of new analyses they’d like to do. For example, Ehrlich explained, a utility might begin by trying to locate which regions are having the most problems by analyzing the general readings from its infrastructure in each area, and then drill down ever further to the point of looking at specific neighborhoods and external data associated with those neighborhoods.

Space-Time CEO Rob Schilling told me one big customer recently summed up the value of the company’s product thusly: “If you can’t see it, you can’t understand it.”

Despite the broadening interest, though, the energy sector is still “absolutely” the driving force behind Space-Time’s growth, Schilling said. The company’s new investors underscore that reality. EnerTech Capital, Novus Energy Partners and ClearSky Power & Technology Fund joined existing investors Opus Capital Ventures and Start Up Farms International in this round.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How to Make Cloud Computing Greener
• Smart Algorithms: The Future of the Energy Industry
• Cleantech and investment in 2013

As I explained when the company launched in February, SpaceCurve’s technology is “able to discover the underlying patterns in multidimensional geodata rather than trying to work around the complexity in the data values. The goal is a product that gives users a familiar database experience on data that are not at all ideal for traditional databases.” In theory, that should mean anybody collecting data from sensors, mobile devices or other devices spitting out location data will be able make use of it in ways and at a scale never before possible — and on scale-out commodity hardware instead of specialized big iron systems.

One possible use.

SpaceCurve CEO John Slitz told me during a call on Monday that the company already has some “major, major” pilot customers in the works. One of them is a Fortune 50 company that has spent tens of millions trying to solve a particular problem, but has been constantly thwarted by problems with the scale and complexity of its data. In fact, Slitz noted, customers are thinking about a handful of “off the wall” and eye-opening uses for SpaceCurve’s technology.

They’ll have to wait a while before they can test those ideas out, however. SpaceCurve will start delivering code in the fall, Slitz said, followed by a close of its beta period in December and general availability around March 2013.

Feature image courtesy of Shutterstock user Hasloo Group Production Studio.

Disclosure: Reed Elsevier, the parent company of science publisher Elsevier, is an investor in Giga Omni Media, the company that publishes GigaOM.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

In a blog post on the Facebook Engineering page today, Facebook’s Karan Mangla describes how it took some serious big-data and architectural handiwork to add broad geo-tagging capabilities. It was actually relatively easy to find locations for Facebook Places when geo-location was only possible with updates done via mobile devices broadcasting GPS signals, but letting users add such data to any updates — even to do so retroactively — meant opening up the range of possible places beyond a user’s current physical location.

Dealing with a larger data set for every update meant Facebook had to change its infrastructure in order to make each update as efficient as possible, lest it risk a slow user experience. Here’s how Mangla said it accomplished this at the database layer:

Depending on the use case, results can be biased toward places closer to the user or provide completely global search. Since the candidate set of places increases significantly in this new system, ranking each of these places to find those that are most relevant to the current user becomes very important. With each place in the index, we store a large set of features to improve ranking, including check-ins and likes received by the place and our best estimate for the opening/closing time for the place. Then we run models generated by machine learning to select the most relevant places based on these features.

With any complex platform such as Facebook, though, there’s always a catch. Mangla noted that while displaying chronological information only requires a server to grab the last few dozen pieces of a user’s activity, displaying geospatial information requires  grabbing all the activity associated with any given place on the user’s map. Facebook had a fix:

To manage this data load, we created infrastructure to farm out data fetching to multiple servers. On every page load, a single server fetches the IDs of all pieces of content that can be displayed for the current user. This server then breaks up this data into smaller chunks, and each chunk is sent in a request to another server to actually fetch the data and do privacy checks. The responses from these servers is then combined to create the timeline map display.

The rest of Mangla’s post provides some deeper insights into how Facebook went about predicting locations for previously uploaded photos, as well as its plan for launching APIs that will let third-party apps update users’ timeline maps or feed users relevant information based on where they are at any given time. It’s well worth reading.

The bigger picture, though, is the further confirmation of how much work goes into making a web platform as sticky as possible to ensure maximum user and developer engagement. That means not only adding new features, but also making them perform as well as possible. That’s why, as I’ve written before, anyone expecting to buy Facebook stock and get rich quick might be in for a surprise. It takes a lot of time and money to get this stuff right, and Facebook plans to spend whatever it takes to keep itself relevant over the long haul.

Of course, assuming that all of this location data will eventually make its way to advertisers who’ll then be better able to target users with relevant ads, Facebook should still bring in enough money to keep everyone happy.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Infrastructure Q3: OpenStack and flash step into the spotlight

Technology is at the core of the discussion. In this case, officers used the GPS to gather data about a suspect’s whereabouts every 10 seconds for 28 consecutive days. Physically tracking someone for the same period without a warrant wouldn’t have raised an issue, but it would have consumed considerable resources. In fact, in its brief to the Court, the government cited only one previous instance of a 24-hour surveillance mission, and it lasted only two days.

A GPS, however, does all the work while the police can just sit back and collect the data. Hence, most warrantless police-surveillance activities are relatively brief, maybe following someone for a few hours, or from Point A to Point B. Chief Justice Roberts astutely summarized the difference: “Well, you’re talking about the difference between seeing the little tile and seeing a mosaic. The one gives you information, the other doesn’t.”

His observation echoes the sentiment of the federal judge in the D.C. Circuit appellate court finding the GPS tracking to be a violation of the Fourth Amendment. And while it’s true that the only real issue at hand is whether a search warrant is required to undergo this type of surveillance, the federal judge’s opinion spells out why some find warrantless tracking so troublesome:

It is one thing for a passerby to observe or even to follow someone during a single journey as he goes to the market or returns home from work. It is another thing entirely for that stranger to pick up the scent again the next day and the day after that, week in and week out, dogging his prey until he has identified all the places, people, amusements, and chores that make up that person’s hitherto private routine.

It’s not about GPS at all

In an article earlier this year about big data and the law (when Jones was known as U.S. v. Maynard), attorneys Nolan Goldberg and Micah Miller took a broad view of the appellate court holding that applies well beyond GPS devices:

Maynard suggests that a person’s reasonable expectation of privacy in a data set arises, at a minimum, when the collection or compilation of the data set would not have been reasonably expected. The results of that collection reveal information specific to those individuals that cannot be discerned from the individual pieces of data constituting the set.

That interpretation would seem to apply to any situation in which the government gathers enough individually innocuous datum to start piecing together personal details of a suspect’s life. The kind of information one could obtain, for example, by obtaining someone’s cell phone records and the location data collected by their carriers.

As IBM’s Jeff Jonas explained on his blog back in 2009, the data our wireless carriers gather about subscribers, when taken as a whole, can be incredibly valuable, especially when combined with today’s advanced analytic techniques and high-powered systems that have the entire IT world abuzz about big data and how it will transform their businesses.

And here’s the kicker: That information is, for the most part, obtainable by police without a search warrant. If you’re wondering what providers store and for how long, this little chart unearthed by the ACLU earlier this year is pretty revealing.

During oral arguments in Jones, Justice Sotomayor certainly seemed skeptical of using wireless location data without a warrant, asking the government’s lawyer the following question:

So — under your theory … you could monitor and track every person through their cell phone, because today the smartphones emit signals that police can pick up and use to follow someone anywhere they go.

Your theory is so long as … what’s being monitored is the movement of person, they have no reasonable expectation that their possessions will not be used by you.

The Court needn’t stretch the issue much further to find that numerous other practices once allowable look a lot more questionable since the advent of big data technologies.

In analytics, more data is better

There’s a saying in the analytics world that more data trumps better algorithms; the idea being that one can more accurately detect trends by collecting and analyzing more information than by collecting less information and investing resources in writing better algorithms. Assuming that’s true, law enforcement is in great shape when it comes to finding out information about suspects without ever first obtaining a search warrant.

Phone records, location data, credit card records, email in many cases, online user accounts: all are obtainable without a warrant. And with the right set of technologies in place, all are capable of being analyzed to paint a clear picture of who a suspect is, who he interacts with, where he goes, where he spends his money, what his interests are, you name it. But technology changes everything, as the Court has noted as far back as 1989.

In their article, Goldberg and Miller note a case in which the Court noted “there is a vast difference between the public records that might be found after a diligent search of courthouse files, county archives, and local police stations throughout the country and a computerized summary located in a single clearinghouse of information.” That same logic could easily apply to Jones, and perhaps should.

Justice Alito said during oral arguments that “the heart of the problem that’s presented by this case” is that technology has made it to amass information about individuals that would have taken extraordinary legwork in the past. “So, how do we deal with this?” he asked. “Do we just say, well, nothing is changed, so that all the information that people expose to the public … is fair game? There is no [warrant required] because there isn’t a reasonable expectation of privacy? But isn’t there a real change in this regard?”

If the Court punts, will Congress step up?

Clearly, the Court gets the bigger picture beyond placing a GPS device, but there are questions over whether it’s the Court’s place or Congress’s to decide issues around data collection and new technology. What seems certain, however, is that the Court’s opinion in Jones will at least address these issues and will influence any future cases in which judges are asked to rule on similar issues.

Unless, of course, Congress steps up and does its job by enacting laws that address the gray area between existing data-privacy laws and new technologies. Some claim existing laws are already threatening our civil liberties, as well as affecting burgeoning industries such as cloud computing.

However it happens, something has to give — and soon. We can cry until the cows come home about consumer privacy and what Facebook can and can’t do with our data, but what Facebook knows about users isn’t putting anyone in jail. Whatever the Court or Congress decide, both officers and citizens deserve to know what’s fair game and what’s not before it’s too late.

Feature image courtesy of AdventureJay.com.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Dissecting the data: 5 issues for our digital future
• The importance of putting the U and I in visualization

At GigaOM’s Structure Big Data conference on Wednesday, Jeff Jonas, a IBM Distinguished Engineer explained the concept with some startling examples. With roughly 600 billion data transactions from cellular phones on a daily basis, adding space and time to traditional data objects can help predict where someone will be on a given day and time with up to 87 percent accuracy, for example. Adding space-time works because, oddly enough, of physics, says Jonas.

Expert counting is a traditional way to observe data, but the same thing cannot be in two places at once. Including space and time observations removes ambiguity. For example, the last 10 years of address history, taken in context, can tell if a person is the same or not, when digging through billions of rows of data.

Why would companies want to add the physics of space-time to their data efforts? More context is needed, Jonas says, because the amount of captured data is rising faster than the creation of algorithms to make sense of the data. That’s creating a gap in the understanding of vast information stores and adding space-time adds much more context. Observations are where the sense-making begins but without context, it’s like trying to build a jigsaw puzzle without the picture of the final product.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future