Amazon, Barnes & Noble and other e-reader companies are collecting data about your e-book reading habits, but they’re keeping their most interesting findings close to the vest. Read more at paidContent »
The results of a recently released survey from Hadoop-focused startup Karmasphere show that while Hadoop use is picking up among mainstream (read “non-web”) companies, it’s still far from the all-powerful and ubiquitous insight engine its supporters (myself included) believe it will become. Read more »
According to physicist Geoffrey West, the world’s cities have what one might call a growing problem. As they grow bigger, their problems grow worse, which means it takes an ever-faster pace of innovation to keep things in check. Big data techniques might provide the answer. Read more »
PoliticIt is a Logan, Utah-based startup that uses machine learning to gauge the popularity of political candidates by measuring their digital influence. Its system has proven remarkably accurate in predicting winners, but its real promise is in leveling the playing field between political haves and have-nots. Read more »
Big data has always had a place in the world of systems management, but it’s sweet spot might be in the cloud. Especially with a new model such as cloud computing, there could be a real value in learning from what your peers are doing. Read more »
If you haven’t heard IBM Fellow Jeff Jonas talk about how the right algorithm can help you figure out who’s who among a sea of data points, you’re missing out. The good news is Jonas’s vision is now a product. Read more »
As datasets get fatter and cumbersome, it’s becoming harder to move them around. Even the fattest pipes look like cocktail straws when you’re talking about petabyte databases. It’s getting more and more difficult to move these massive troves of data to the applications that use them. Read more »
In publishing, there’s a constant struggle to determine who’ll read what posts, what the ideal headline might is and when is the best time to publish. GigaOM is teaming with Splunk to find some answers via a Kaggle competition worth a total of $25,000 in prizes. Read more »
Discussions about the cloud now involve more than just the IT department. New developments in hardware architectures, more-energy-efficient data centers, regulatory concerns and simplifying analytics are all discussions currently circling through the industry. Here’s what to consider when thinking about your business in the cloud. Read more at GigaOM Pro »
When you mix a researcher, a massive online encyclopedia and a supercomputer, the result is a collection of insights and visualizations into what Wikipedia looks like mapped across time and space. It looks a lot like how our history books might look merged and graphed. Read more »
Now that businesses have collected and stored all of this data, how are they going to protect it? And most importantly, how are they going to use if safely and legitimately? ISF’s Steve Durbin outlines the five key issues surrounding big data and information security. Read more »
Now six years old, the Apache Hadoop platform for storing and processing huge amounts of data, perhaps the catalyst of the current big data movement, appears ready for its closeup. According to the companies leading the Hadoop charge, they’re already beating away customers with a stick. Read more »
Netflix’s algorithms for recommending movies to customers might not be perfect, but it isn’t for lack of trying. The company is capturing and analyzing incredible amounts of data, even from the videos themselves, to try and figure out what you want to watch next. Read more »
It’s no secret that Facebook stores a lot of data in Hadoop, but how it keeps that data available whenever it needs it isn’t necessarily common knowledge. Today at the Hadoop Summit Facebook Engineer Andrew Ryan highlighted that solution, which Facebook calls AvatarNode. Read more »
Amazon Web Services already has a winner with its Elastic MapReduce Hadoop service, and now it’s turning up the heat by adding MapR’s Hadoop distribution as an option. Users can take advantage of MapR’s performance features while also having integration with AWS’s suite of cloud services. Read more »
VMware is launching a new open source project, called “Serengeti,” that aims to let the Hadoop data-processing platform run on the virtualization leader’s vSphere hypervisor. VMware apparently smells a lucrative opportunity in Hadoop and isn’t about to miss out on getting a piece of the pie. Read more »
Online genealogy service Ancestry.com is trying to become like the Amazon or Netflix of family trees. Much like those companies use customer data to recommend products or movies customers might like, Ancestry.com is using machine learning to make learning about ancestors a lot less work. Read more »
According to a new survey by the Economist Intelligence Unit (commissioned by IT consulting giant Capgemini), corporate executives are starting to figure out that big data matters and how to leverage it, even if they haven’t fully come around on the concept. Read more »
Amazon S3 now hosts more than a trillion objects for its cloud computing customers, proving once again that AWS is the king of the cloud. That’s 142 objects for every person on Planet Earth or 3.3 objects for every star in the galaxy. Read more »
One year after launching into the Hadoop market with much anticipation, Yahoo spinoff Hortonworks finally has a product available. The company announced version 1.0 of its flagship Hortonworks Data Platform on Tuesday, as well as a High Availability version designed with new partner VMware. Read more »
Rice University researchers have built a web-based calculator that predicts the risks associated with hurricanes for an address. The tool uses historial and meteorological data to generate a risk profile for residents of Houston. This is what big data tools should do — offer users actionable intelligence. Read more »
Karmasphere CEO Gail Ennis told me recently she thinks “2013 is going to be the year when we see [Hadoop adoption] go a lot more mainstream and [turn] into a tornado.” I like the prediction, as much for its imagery as for its near-term certainty. Read more »
Earlier today I wrote about online bra retailer True&Co delaying delivery of its products to customers. It turns out that the supply chain was an issue, but only because the demand for the service outstripped what True&Co thought would be a full year’s supply of inventory. Read more »
Online bra retailer, True&Co, which launched last week to help women find the perfect bra, is having trouble delivering its products. As the Internet crosses over into the real world, not only websites must prepare for a launch, but the entire supply chain. Read more »
Two key members of the Facebook team that created the Hadoop query language Hive are launching their own big data startup called Qubole on Thursday. Qubole is a managed version of Hive that’s hosted on the Amazon Web Services cloud computing infrastructure. Read more »
In a move to expand its utility beyond simply finding better answers to known statistical problems, hot data-science startup Kaggle is now letting its stable of expert data scientists compete to tell companies how they can improve their businesses using machine learning. Read more »
If you just pay attention to largest Hadoop users, you might think the platform is just a way of powering search engines or analyzing customer behavior for ad-serving. Of course that’s not the case, but finding those broader use cases can still be kind of difficult. Read more »
According to new research by Twitter’s data science team, Twitter search is used often as a tool for finding breaking news in real time, which makes it difficult for Twitter to assign relevance to any given tweet or topic in the long run. Read more »
All right all you big data nerds — it’s time to suit up for the NIST’s Big Data workshop slated for next week. The event will focus on what state-of-the-art core technologies will drive big data and how to ensure accuracy of big data processes. Read more »
Privacy was often an afterthought for small companies, but that’s changing in the era of big data. As TRUSTe’s CEO, Chris Babel has seen the impact privacy can have on a startup — for better or worse. Here, he offers tips on how to avoid the pitfalls. Read more »
A startup called Datahero launched on Thursday with a new cloud service that makes visualizing data as simple as a few mouse clicks, and $1 million in seed funding. The company’s ultimate goal is to make big data something you or I could do. Read more »
The U.S. government is investing billions of dollars in big data technologies and research, and now it has a team of industry executives from IBM, Amazon and elsewhere ready and willing to share their views on how it can best transition into a data-driven institution. Read more »
Solariat Founder and CEO Jeffrey Davitz has a message for anyone trying to leverage social network data to make money: “The fundamental problem with social is yes, it’s big data, but it’s mostly big, sucky data.” Targeting users means deciphering what they really want. Read more »
Roambi is becoming a tool for more than creating cool, interactive data visualizations or real-time business intelligence reports your boss or colleagues will read on their iPad. Now Roambi is aiming to be a mobile app development/ publishing service with a new feature called ESX Platform. Read more »
The greater Boston area’s bid to be the go-to big data hub will get a boost as Intel and MIT will announce the Intel Science and Technology Center for Big Data as well as bigdata@CSAIL, a research group for MIT academics and industry researchers. Read more »
As more “open data” comes online, finding the right data, managing its access and workflow, and fostering collaboration, is the problem startup Junar wants to attack with its new Open Data Platform. Customers can try it out for free, starting now. Read more »
When a data scientist crunched enough numbers to predict that Sweden would win this weekend’s Eurovision Song Contest, he felt fairly confident. But he didn’t expect that the biggest noise would be the inaccurate prediction that Malta would do well — something he’s now apologized for. Read more »
If you’ve ever wondered what big data means at an individual level, this realization about sums it up: “I could either keep dying my hair or retire a year earlier.” It’s those types of realizations Intuit hopes its heavy big data use will help uncover. Read more »
NoSQL databases like MongoDB, Cassandra or CouchDB are a key foundation for web startups. But those companies might be better served using an old-fashioned relational database when it comes to their bread-and-butter transactions, according to Thrillist CTO Mark O’Neill. Read more »
According to some researchers, web companies such as Google, Amazon and Facebook are doing the research world a disservice because they won’t make their datasets available for peer review. These researchers have a point, but privacy concerns might always trump openness where openness matters at all. Read more »
