The company focuses on historical data that might need to be stored for long periods of time and isn’t likely to change. In some cases that might be data stored for regulatory compliance, while in others it might be machine data such as server logs that would never be written over in the first place. RainStor has also beating the drum pretty loudly around big data, where it certainly has a compelling proposition.

Because RainStor utilizes massively parallel processing, it can ingest and query data in a hurry. The company claims ingest speeds of 30,000 to 50,000 records per second per core. Query results also return faster because the analyses are taking place over such a smaller volume of files.

RainStor also can sit atop Hadoop for users that want to churn through unstructured data via MapReduce or Pig jobs as well as run more-traditional SQL queries. This type of hybrid system is becoming a hot topic in big data circles, and is the premise of a handful of other products, including Boston-based startup Hadapt. I’ll actually be speaking about the intersection of Hadoop and SQL at our Structure: Europe conference on Oct. 16 with Cloudera’s Amr Awadallah and NuoDB’s Barry Morris.

RainStor says it can provide 50 to 80 percent smaller Hadoop clusters and increase performance by up to 100 times. These are both big concerns as data volumes continue to explode. Quantcast created its newly open sourced distributed file system so the company could bring operational expenses under control while also lowering the time it takes to process its ever-expanding data set.

You can watch RainStor Chief Architect Mark Cusack discussing big data at our Structure: Data conference below.

Feature image courtesy of Shutterstock user z0w.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Takeaways from the second quarter in cloud and data
• Amazon’s DynamoDB: rattling the cloud market

The sum of all these parts is a huge privacy violation

For example, when viewed in a single presentation, claims made by Netronome — a networking chip maker — that it can process massive flows of information in real-time in order to intercept unlawful packets might not be worrisome. However, when juxtaposed against the violation of human rights that the word “unlawful” can hide, it becomes part of a broader story of surveillance and what many in the U.S. would rightly regard as a violation of their privacy. From the intro to The Spy Files:

Intelligence agencies, military forces and police authorities are able to silently, and on mass, and secretly intercept calls and take over computers without the help or knowledge of the telecommunication providers. Users’ physical location can be tracked if they are carrying a mobile phone, even if it is only on stand by.

But the WikiLeaks Spy Files are about more than just ’good Western countries’ exporting to ’bad developing world countries’. Western companies are also selling a vast range of mass surveillance equipment to Western intelligence agencies. In traditional spy stories, intelligence agencies like MI5 bug the phone of one or two people of interest. In the last ten years systems for indiscriminate, mass surveillance have become the norm. Intelligence companies such as VASTech secretly sell equipment to permanently record the phone calls of entire nations. Others record the location of every mobile phone in a city, down to 50 meters. Systems to infect every Facebook user, or smart-phone owner of an entire population group are on the intelligence market.

WikiLeaks' leader Julian Assange

What people can do in reaction to the information

But will the shock of seeing the tools of our digital surveillance society aggregated together with tales of abuses be enough to stop governments from implementing ever more technical means of gathering, analyzing and using information? My colleague Mathew believes the transparency here will help address the problem, but I have less confidence. While the outrage currently aimed at CarrierIQ in the U.S. over its software that tracks users via their smartphones is intense, in a month it may well be forgotten. Lasting and real change is harder to come by when it comes to these issues, because it’s hard for enough people to sustain the outrage needed to go against something that benefits the government.

That the CIA or FBI can put GPS devices on cars without a warrant (since questioned by the courts) or attempt to access the data about web sites or on hard drives is a huge temptation even in the freedom-loving U.S.. Because of the ephemeral nature of our digital transactions, many citizens are unaware at how concrete and traceable their bytes really are. Just because you don’t recall a tweet you made six months ago, doesn’t mean it can’t be found and dredged up against you. Likewise, search data sticks around for nine to 18 months; your phones are a black box of information that the feds would love to crack; and the government and law enforcement don’t want to restrict their access to any of this digital gold mine. Unlike the Founding Fathers, who were protecting their rights after a revolution, our politicians are protecting the status quo and see surveillance as a tool to help maintain that.

As for sales to dictators and other countries that might use access to Internet monitoring, phone tracking, listening devices and GPS logging software as a means of repressing their citizens or activists, the practice may be morally reprehensible, but its unclear if it runs afoul of U.S. laws. Yes, there are laws that prevent U.S. firms from selling technology to certain governments, but the list of governments and technologies is by no means comprehensive.

But, other than shining a light on the practice and writing stories, what is there to be done? In the U.S., the courts are the best method of counteracting overreaching governments, which means someone has to get caught up in some zealous digital manhunt. The Wikileaks Spy Files may help someone realize this is going on, but it’s unlikely we’ll see laws designed to really protect privacy in the digital age. So then how should citizens, public policy groups and others take this data dump? More than the information that DPI inspection gear from Arbor Networks or data retention and analysis tools from Netezza or RainStor are used in some cases to target or suppress citizens or activists, these are the questions the report evokes.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4
• IT spending update, third quarter 2012
• 2012: The Hadoop infrastructure market booms

* Scale Computing just raised $9 million in funding in its second round that was led by by Benchmark Capital.

* Pivot3, a company that provides virtualized servers and storage, raised $25 million from Focus Ventures and other investors.

* RainStor, a structured data software company, today raised $7.5 million in Series B funding from Storm Ventures and Informatica.

We’ve been keeping track of this trend for nearly two years. So in order to get more context, check out these posts from our archives:

* Storage Startups Turn Cache Into Cash
* Who Will Cache in on Cloud Storage
* The Future Is Big Data in the Cloud
* Why Big Data & Real-time Web Are Made for Each Other.
* How Yahoo, Facebook, Amazon & Google Think About Big Data