This week, both Facebook and Yahoo detailed new efforts to manage real-time data flows within their myriad systems. Yahoo’s work is an open source implementation of Storm designed to run on the same cluster as Hadoop and even share resources. Read more »
Google Image Search just got a whole lot better, and the company’s purpose-built machine learning system infrastructure is a big reason why. No surprise, Jeff Dean helped build it. Read more »
As more companies build their businesses on cloud infrastructure, it’s important to not only understand the technical decisions behind their architecture, but also the economic ones. That’s one of the topics we’ll explore at Structure. Read more »
GigaOM contributor James Urquhart shares some of the best books, blogs and other information on the concepts of devops and complex IT systems. Read more »
Google spent $1.2 billion on property and equipment in the first quarter of 2013, nearly doubling last year’s first quarter. Read more »
Former VMware CEO and current Pivotal Initiative leader Paul Maritz shares his thoughts on how the future of enterprise IT must mirror the practices of consumer web companies. Read more »
Microsoft is no joke when it comes to building web infrastructure and developing techniques such as machine learning. The company thinks its heavy investment in these areas will pay off big-time in the years to come. Read more »
Facebook has developed a new data cache called McDipper that’s essentially memcached rewritten to run on flash memory instead of DRAM, thus saving money while still delivering higher performance than disk. Read more »
eBay has released a trove of information about the efficiency of its data centers, and plans to do so quarterly as part of a mission to continuously track computing resources and tie them to bigger business goals. Read more »
Five years ago, LinkedIn was a shell of the technology company it is today. Here’s an inside look at where it came from, what it’s become and where it’s going. Read more »
Box VP of Engineering Sam Schillace talks about building the service that became Google Docs, then fighting for the service’s life, and now rethinking collaboration for the mobile web. Read more »
Google spent more than a billion dollars on infrastructure in the fourth quarter, representing the company’s second-biggest quarterly expenditure ever. As it competes against Facebook, Apple, Yelp and Amazon, the company can’t afford to stop building data centers now. Read more »
Zynga has deployed nearly 100 nodes of MemSQL, the hot new database from two former Facebook engineers. It might not be a magic pill for Zynga’s woes, but it could help the company boost revenue and even build new types of games. Read more »
China’s big four internet companies are big — huge, in fact — but they’re not yet technological innovators like their American counterparts. However, scalability is an an issue that knows no borders, which has spurred some cross-continental cooperation. Will it also inspire a Chinese tech awakening? Read more »
The “mobile first” philosophy is under way today. That means a new generation of mobile-centric data centers will arise over the next three years, with chips, servers, and power architectures customized for mobile workloads. Read more at GigaOM Pro »
At the AWS Re: Invent conference, engineers from Pinterest, Flipboard and Yelp detailed some of the strategies their companies employ in order to keep costs low as computing demand increases. The keys are keeping an eagle eye on usage and using the right types of resources. Read more »
AWS CTO Werner Vogels and I sat down at the AWS re:Invent conference yesterday to talk about whether large companies are actually using the cloud to innovate through new styles of applications. Vogels says they are, and has plenty of examples to prove his point. Read more »
Obama for America CTO Harper Reed had helped build Threadless, a site for selling hip t-shirts, but he had never done anything like this. Here’s how he and his team built a tech platform that might forever change how presidential campaigns are built. Read more »
Facebook has open sourced a new system called Corona for scheduling and managing Hadoop jobs. Corona attempts to do away with many of the problems that come along with massive-scale Hadoop operations, and soon looks to take Facebook’s Hadoop deployment beyond just MapReduce. Read more »
MongoDB startup MongoLab has built a seemingly solid business delivering the popular NoSQL database via the Platform as a Service model. The trick, CEO Will Shulman says, is knowing how to sell to developers and understanding why someone would use MongoDB in the first place. Read more »
When companies such as Google and Facebook design their own servers, switches and data centers, it’s more a business decision than it is a test of their hardware-hacking skills. Custom gear means lower power bills, better performance and the flexibility to adapt to unforeseen situations. Read more »
Facebook knows something about big data — it collects more data and has built more tools than almost anybody else. Here, Facebook’s Jay Parikh and Accel Partners’ Ping Li talk about what lessons big data startups can take from Facebook to build businesses that can succeed. Read more »
It’s not for everyone, but if you’re storing petabytes of data Hadoop, Quantcast thinks it has the cure to your woes. Its newly open sourced Quantcast File System promises smaller clusters and better performance, and it has proven itself over exabytes of data inside Quantcast. Read more »
Much of the data center industry is upset about a recen report exposing some wasteful energy practices. However, eBay’s Dean Nelson says the data center industry isn’t perfect and it’s up to companies like his to lead the charge on bringing everyone else up to speed. Read more »
Orbitz has transitioned a major system off of Oracle’s Coherence database and onto the NoSQL Couchbase Server, but the database giant still has a significant footprint in Orbitz’s data centers. It’s all part of being a big company trying to roll with the IT punches. Read more »
Hosting giant GoDaddy has completed its investigation of Monday’s outage and deemed it was not the result of a DDoS attack as originally rumored, but rather the result of network failures within GoDaddy’s system. The outage crippled hundreds of thousands of web sites. Read more »
Etsy shared the details of its hardware architecture on Friday, showing the world a whole lot of Supermicro servers running everything from web servers to Hadoop. At this point, software is the name of the game at webscale, so hardware openness is just welcome community service. Read more »
Netflix has open sourced Chaos Monkey, a service designed to terminate cloud computing instances in a controlled manner so companies can ensure their applications keep running when a virtual server dies unexpectedly. In the past year, Chaos Monkey has terminated more than 65,000 of Netflix’s instances. Read more »
Companies doing business on the web and in the cloud must spend a lot on infrastructure to stay running and to support new services. Here’s what Apple, Facebook, Google, Microsoft, Yahoo and Zynga spent in the recent quarter, and what it might mean. Read more »
If Twitter wants to remain opaque about its practices, that’s fine — but it shouldn’t expect any slack from upset users or investors. Blaming a two-hour outage on an “infrastructural double-whammy” after remaining mum on even where its data centers are located doesn’t exactly inspire confidence. Read more »
In cloud and big data, the second quarter of 2012 featured several high-profile deals and product launches that could reshape the marketplace for everyone. Google and Microsoft launched Infrastructure-as-a-Service offerings, software-defined networking took off, and all eyes stayed fixed on the continuing promise of data analytics. Read more at GigaOM Pro »
For better or worse, Hadoop has become synonymous with big data. In just a few years it has gone from a fringe technology to the de facto standard. But is the enterprise buying into a technology whose best day has already passed? Read more »
Last week’s AWS outage outage might have outsmarted Netflix’s Chaos Monkeys, but the content-distribution giant isn’t about to turn its back on cloud computing. It was a relatively small blip in what has been better availability since the company made the move entirely to the cloud. Read more »
AOL is taking its flexible infrastructure strategy to a whole new level of flexibility by building data centers that are about the size of French door refrigerators. Now, AOL will be able to deploy infrastructure where needed with little more than an electrical outlet required. Read more »
Lest storage vendors thought they were immune to disruption that open source hardware is having on the server industry, Netflix’s new Open Connect content-delivery network might make them think again. It’s inspired by open source storage designs first released by Backblaze almost three years ago. Read more »
10gen, the creator and commercial entity behind the popular MongoDB database has raised another $42 million and wants to take the technology to an application near you. The money will help 10gen double down on research and development to make MongoDB live up to its hype. Read more »
Facebook’s hyperinflated valuation heading into its IPO has everything to do with its promise, and very little to do with its actual profits. Here are some numbers we know about Facebook’s infrastructure that speak to its promise perhaps as much as its 900 million users. Read more »
As the world once again starts analyzing Yahoo’s myriad woes after Sunday morning’s ouster of embattled CEO Scott Thompson, I’m left wondering if its investment in Hadoop didn’t aid in the company’s demise, even if it’s a way down the long list of Yahoo’s mistakes. Read more »
If Facebook really is overvalued leading up to its IPO, privacy might be the underlying cause of the company’s missed expectations. As it turns out, pleasing both investors and users isn’t easy for a company that relies heavily on advertising and personal data. Read more »
If you think about it, Netflix’s metamorphosis into a company that runs its infrastructure completely atop cloud-based resources is truly remarkable. For many companies, such as site-optimization and CDN provider Yottaa, the bigger they get, the harder it is to justify the cloud’s cost and performance. Read more »
