Blog Post

Google: Livestreaming of record-breaking skydive used a quarter of the world’s bandwidth

[protected-iframe id=”299a720df8dcdbbeeb21b3405305e5fe-14960843-33105277″ info=”http://new.livestream.com/accounts/74987/events/2361507/videos/30326591/player?autoPlay=false&height=360&mute=false&width=640″ width=”640″ height=”360″ frameborder=”0″ scrolling=”no”]
Session Name: Google’s Infrastructure DNA, packaged up for you.

Chris Albrecht
Barak Regey

Chris Albrecht 00:03

Thank you very much. Good way to start the day. We’re going to move on to Google’s infrastructure DNA, packaged up for you. It’s going to be a presentation from Barak Regey, he’s the head of EMEA Cloud Platform for Google Enterprise. Please welcome Barak to the stage.

[music]

Barak Regey 00:20

Hi there good morning. Thank you so much for those of you have joined me here on the LiveStream it’s an early bright morning in London. I saw that most of you’re local but for those of you who are not local just be aware we did have a summer here up to two weeks ago – and it was a real summer. I’ve been living here for a year so I can tell you that it was a real joyful summer. I’ve mentioned EMEA Cloud Platform across the Europe, Middle East and Africa. I work for Google. My names Barak Regey and I’m here to tell you a story. I want to take you through a journey through our cloud and try and give you a little bit sense of what it is. What Google contributes to the cloud market and well as to the big data? I love first there, and I’ll relate to the Hadoop phenomena throughout my presentation. So many discussions have been made about the cloud and I’ve been attending so many conferences and when I think about–I never heard never. The same answer of what is cloud. You hear the different variations and definition coming in from different vendors obviously each one with his own pitch and his own appeal. But if you think about Google a fairly young company. We’re celebrating our 15th birthday this month – or next month actually, by design we’re a cloud company you might say.

Barak Regey 01:41

When we think about what cloud gives us – it gives us the ability to innovate. It really removes barriers, it gives the IT the real place where it needs to be and that’s to enable the workforce to become more productive and innovation. We’re very proud that cloud actually supports your business in transforming into a richer environment into enabling your information workers. Driving innovation and live in the modern workplace, because this is what it’s all about and the facts in the markets are out there. If we think about – I call this the geologic time frame that is taken for companies. So, I would assume that not everybody here is a 20 year old very cool developer and some of us have been out there in the market for a while. One of the amazing things that you see is that it took time for companies to build its credibility it’s presence and if you look at the standards in poor and 500 index, which basically includes the top 500. You would see that on average up to ten years ago it took you 75 years to build that business. Now if you look at the S&P 500 you will that the vast majority of the company vendor are less than ten years old. That’s how fast these companies evolve, and the next big S&P 500 probably sitting here in this auditorium or somewhere here in London or in the Silicon Valley and it’s yet to be formed or established. Let me tell you something that company will be based on the cloud and it’s entire IT infrastructure will be there as well. So think about the amount of innovation climbing in from those companies.

Barak Regey 03:25

Given the time I’m not going to talk about the obvious so you’ll see me sometimes run really quick or not even mention a certain slide, but I want to just focus here on one thing before I dive into what the Google story and give you a little bit sense where it is. I love this Amara’s law where it’s so obvious but we sometimes we need to reflect back and think about this statement. People tend to overestimate what can be done in one year and to underestimate what can be done in five or ten years. Think about that discussion or this statement and the essence of Hadoop or big data -technology that’s been out there on only for a few years. I hear a lot of discussions and you know the immediate the comparisons to data warehousing and traditional tools and SQL and all up services and OLTP services. I tell people to take a step back try and envision things beyond your time frame because sometimes it does take markets to mature, and it does take technology to hit main stream; the same goes for the cloud. There’s so much discussion going on the impact of cloud now. How you shift existing applications that you have bespoke, developed applications that you have been developing for last 20 years in your enterprise domain and how you shift them effectively to the cloud. I always say that’s not the core discussion of cloud. The cloud is about new workloads, it’s about the way to innovate and it’s always trying to sync beyond the one year time frame. I know it’s a hard statement what I’m saying because in this day and age that we’re living in and the pace of innovation – five years is – I must tell you that I’m pretty certain that the way that everything is evolved now, big data solutions cloud platform will not even look the same in five years. I’m pretty sure that most of you will agree with me on that.

Barak Regey 05:14

How does all of this all relate to Google and this whole cloud phenomena? It’s really important to give you a little sense in the next ten minutes I’ll try and do a good job in giving you that – what Google is about. In the last 14 years – next month I’ll update the slide to 15 years – but we have been building the words faster and probably the most powerful highest cloud infrastructure on the planet. I want to say a statement here not to upset anybody we didn’t build it to enable the cloud platform and really at the core essence. We built it in order to support our business. We built it to support assets like Google.com, YouTube, Gmail and a lot of infrastructure that I want to assume that some of you are using on a daily basis or hopefully some of you are using. The beauty about this is that this same infrastructure is basically what we now provide you. It has taken us some time, but we’re literally unlocking access to our most precious infrastructure – that’s the beauty of Google’s cloud offering. We are giving you access to the data center facility – the picture here from our Idaho Data Center – and it’s the same infrastructure that powers Google is now going to be able to power your organization. Your next web campaign, your next mobile app, you next–anything that you want do in the workload. And that’s the real beauty and the real differentiator of what Google is about.

Barak Regey 06:39

It’s not only about cloud, it’s about operational excellence it’s about commitment to environmental responsibility – we live in a green era and it’s really important. I think that only companies of that scale can really put an audacious goal of being 100% carbon neutral. It’s not a trivial task believe me with the amount of data centers with operate, but this is something we strive for and we invest tremendous amount. But beyond that we invest in very basic stuff from the hardware design up to Google’s network. We are very proud to be probably on the non-Telco company to operate marine sea cables. We have a dedicated line going in throughout the Pacific Ocean to connect Google platform as well as Japan and actually in Asia and we were very fortunate to complete that before the Gum Gum style explosion on the internet – so think about that.

Barak Regey 07:37

Think about other live broadcasts. I don’t know how many of you saw the live moon shot parachuting Red Bull, but for those who know or don’t know that was a live stream going on in the internet. And during that time that actual live stream consumed probably a quarter of the world’s bandwidth. That’s how big and audacious our network is in terms of really supporting mass scale beyond our imagination. And it’s beyond just the sea cable it’s also the global open-flow network. Has anybody heard about software defined networks, which are the big buzz and trend? Google has been doing it now–we’ve been very proud to be standardized on size DNA networks for the last two years and we keep on developing. All of our data centers across the globe are connected, we own the fibers between out data centers and what that gives us, it gives us the ability to give you consistency. The same consistency that experience as a consumer user using Google.com, YouTube, that’s the beauty of building multi-global applications that need to be distributed across the globe and that’s the power again that you get when gaining access to our cloud platform.

Barak Regey 08:44

The investment in our cloud infrastructure – again not to upset anybody – doesn’t go because of what I created or the demand of business consumers asking to spin up more instances on our cloud or run their core applications because what Google is. We provide services to the mass consumers to business and we see tremendous growth going through all of our assets, via it search, via it YouTube. We continue to invest around $3 billion on an average. We just announced and you can see presence or expansions into South America, here in Europe, the US in other parts of the world; that investment just keeps on pouring in order to make that a reality. But behind the hardware the fiber the processor the biggest pride I have in working for a company like Google is the innovations we make in software. I don’t know how many of you have read these white papers or know – I want to assume that a lot of you have heard about MapReduce. 2004 Google published the white paper telling basically the open source community how we handle big data internally.

Chris Albrecht 09:52

That’s been a decade now for us, we still have tons of MapReduce jobs running in Google but we are mostly now converted into Flume. A lot of you will see that the next wave of big data will be Hadoop as much as it’s new is relatively going to be faded away. That discussion that was here before me on the convergences of the SQL world in Hadoop I thinks that’s going to be even spinned further when new technologies will be introduced. But we’re extremely proud to see that in 2004 we introduced the MapReduce white paper and three years later Hadoop and the Hadoop evolved. If you look at 2006 big table, it’s currently the biggest noSQL database out there, but it’s also literally you guys use it if any of your Gmail user – Google.com – are entire web index is built on that. Yet again we’ve exposed just recently an instance of what we called big query which gives you a little bit of the taste has internally to run these massive platforms.

Barak Regey 10:56

A really quick two minute summary of what cloud computing is for Google. The cloud world has been dissected into three pillars; the infrastructure is a service; the platform is a service; and the software is a service. Google is in all fields. I look after our infrastructure as a service and platform as a service, our software as a service offering mostly dominated by Google Apps, or by Google Maps is different part or part of my extended family of Google enterprise brand. But I’m going to talk to you now about now a little bit about our infrastructure as a service and platform as a service.

Barak Regey 11:34

The way I want to try and outline it for you is a very simple manner because it’s early morning we didn’t all have breakfast. I didn’t have breakfast so you know the brain is still warming up. But think about what the red items are what you manage and grey colored stuff is things we manage for you, and that basically gives you an al la carte menu to do whatever you want. You can go the most extreme manner – I mean extreme in a way that you can leave it all for Google and get us to do your software the services I mentioned; your mail for example. But the decision when do I choose an infrastructure as a services versus a platform as a service is very specific to the scenario we speak about. I can tell you that the vast majority of start-ups that we engage with will usually start off with the platform as a service. And guess why? They have a very, very limited capacity in terms of the amount of head count they have or human resources and they want to leave all the thing that they don’t need to manage, leave it to somebody else. But then infrastructure as a service you usually find companies that are looking to – I’d say shift – existing workloads into the cloud. We’re looking for a little but more flexibility in way that we manipulate data, or we manage data. You will never run a Hadoop instance on platform as a service. You will probably most likely always run it, spin up your own VM, Linux flavor – whichever you want and run your Hadoop instance on that.

Barak Regey 13:01

So the decisions are usually or the way that infrastructure versus platform is defined is really based on the use case scenarios. This is my personal belief, we will see as we talked about or spoken before me about the convergence of Hadoop and SQL. I believe that IS and Pass will eventually find their way of becoming one. But that’s again, don’t think about the next year try and envision and don’t underestimate what we can do in five to ten years from now.

Barak Regey 13:33

The way build our cloud platform and offering is basically is simple and in a way for you to either leverage all of the assets some of the assets or a really small portion of the assets. We have the most basic compute elements from virtual instances that you can spend. It’s still in limited release we announced it only a year ago, but it’s advancing extremely fast. I’ll show you in a bit some use cases already of our compute instances but it’s coming strong into the market and it’s key differentiator and the way that we’ve seen the uptake of it is basically people love the consistency of spinning up instances really, really fast. As time would suffice and they would let me run–I usually run this presentation and I click a button in the beginning to show you during 20 minutes of talk how many virtual instances I can spin up. And you would be memorized with the number we reach after 20 minutes. But given I don’t have that time to do that demo I can share with you that we reach many, many thousands of virtual instances and first open running in 15 seconds, which is extremely interesting.

Barak Regey 14:42

Then the cloud offers new storage capabilities – that’s the yellow pillar in the middle – so many variances I’m not going to go into them. But think about that you have any type of storage you need, from blog store to relational database to mass noSQL data store or anything of that nature. The biggest element – not the biggest – but the coolest element would be our API’s. So many API’s that some of you know are wrapped under a cloud from offering and give you access really to utilize Google’s platform in the loosely coupled manner that it needs to be engaged.

Barak Regey 15:20

So that’s what Google cloud platform is, it’s bringing the magic of Google to your developers to your business and giving you the ability to leverage our infrastructure. The way that the whole evolution -and we’ve been now interacting with business over the two years now – we’ve seen major focus and this is my snapshot of what reality is from my eyes. That’s the current workloads that we see taken up to the cloud, it’s back-end for mobile, it’s the gaming company that really need to burst infinitely. It’s mass amount of storage, it’s the big data, it’s high performance computing and it’s–think about the digital marketing. The many sites that need to go viral within very, very short time frame. As of today beyond what we use internally we are very proud to host over three million active applications on our clouds. That’s a very, very big number and the fact that it’s developed in two years tells you a big story of what Google cloud platform is.

Barak Regey 16:22

Just to give you a sense of the stories that I cannot just leave you here with these big statements. Anybody heard about Snapshot? Anybody have kids that use Snapshot? That’s the more appropriate. Snapshot didn’t exist a year ago. Let me tell you something, after this talk go and check them out. Snapshot came up with an amazing concept of photo sharing between individuals but you as photo sharer can decide how much time you can view the picture and it never gets stored on your device. Pretty cool. In one year they are now sharing over 250 million photos per day, that’s literally and built entirely on the cloud. That’s how explosive a cloud can support. And this is the same phenomenal you see with other companies like Instagram or any other company that came into the public. And literally we never heard about the before they were bought or acquired by the bigger company.

Barak Regey 17:19

In other examples would be SongPop, but for the Europe crowd Eurovision was just broadcast. It’s the second most non-sport event televised live in world, 400 million spectators. The entire voting experience and web experience was all hosted on our platform and we were reaching a peak level – amazing amount of request per second. We’ve noted here 50,000, I know the numbers were a bit higher but the beauty of the consistency serving 90% of that request was less than 24 milliseconds of latency which is again staggering numbers for that. But it’s not only limited, the whole cloud is not only a story of the start-up, you see traditional retailers be it Best Buy in the US developing their entire mobile experience in social apps on the cloud rather than doing it on their infrastructure. And you see how they mention gaming company then traditional this is the largest e Commerce. If there’s any French people in the audience I’m pretty sure they’re familiar Cdiscount is the e Commerce site of France.

Barak Regey 18:29

Basically all types of companies are going into the cloud and really making a leverage out of it. I’ve been trying to be very fast; I usually take a little bit more time in diving in and giving a little bit more example and scenarios. I just want to leave you with the facts that beyond Google a massive platform we work with phenomenal partner that help us bring in the value of software, enhance our platform as well as obviously our customers are probably the biggest people that design our platform. Because based on their requirements we know how to improve instantly.

Chris Albrecht 19:06

That’s it you can see a lot of information on Google’s cloud platform at cloud.google.com. I’m going to be around here till mid-day so if anybody wants to have a chat with me you might find me. Thank you very much.

[applause]