Analyst Report: Evolving SDN: Tackling challenges for web-scale deployments

1 Summary

Customers want mobility, rapid change, and larger networks that work with less hassle through the use of powerful automation because this enables business to build speed in innovation. Software-defined networking (SDN) embraces these requirements with new dynamic networking features that enhance server value and user services while operating with an existing network.

Virtualization arrived in networking more than a decade ago in the form of virtual local area networks (VLAN). In the early 2000s, multiprotocol label switching (MPLS) network virtualization enabled vast global networks in the wide area network (WAN). Then, in the mid-2000s, device virtualization arrived and delivered virtual firewalls. Throughout this period of change, the network edge remained located at a fixed physical point with simple, static services.

Server virtualization enables data center mobility, while Wi-Fi and LTE networks enable user mobility. Yet our current network technology remains focused on fixed endpoints. The tension is leading to a change that adapts old requirements for static and stable connections to dynamic and variable forwarding methods. Today’s network is built from hundreds of individual devices that act like separate elements instead of a single platform.

So is SDN the best of everything? In examining that question, this report will also address questions including:

  • What are the major technology enablers for SDN?
  • How are carriers and enterprises implementing SDN to enable distributed applications and cloud infrastructures?
  • How can customers prepare for SDN given that the technology and marketplace are rapidly evolving?
  • What are the key industry standards efforts (IETF, ONF, NFV, etc.), and how do they differ?

2 The history of SDN

SDN has been evolving since 2005, when ideas based on network security evolved at Stanford University as a research project. In 2006, Stanford University graduates published a paper on a clean-slate security architecture, which they named SANE, and an “Ethane” software controller for network access control (NAC) that stated, “No user, switch, or end-host has more information than it absolutely needs.”

The ideas behind SANE faded as NAC proved unworkable in actual use, but the key technology programming flows into network devices were adapted in 2008 into OpenFlow as a switching specification hosted and supported by high-profile Stanford professors, including Nick McKeown, and Scott Shenker at the University of California, Berkeley.

The first use of SDN concepts was a Global Environment for Network Innovations (GENI) project that was intended to allow researchers to share the existing university networks for research purposes. Researchers could not afford to purchase networking assets and wanted to use software methods to share existing equipment with each having her or his own controller for handling a slice of the network. In 2009, MIT Technology Review named OpenFlow as the top technology of the year, and was established.

But in 2010, it became clear that OpenFlow enabled a whole new model of managing networking because the centralized controller has a unified end-to-end view of the network.

By 2011, the term “software-defined networking” was in common use to describe the concept of controller networking in which a combination of controller software and applications that “drive” the controller had become the focus of the technology. The concepts of using software applications to program flow forwarding in the network layer had reached wider appeal.

During 2012, it became clear that software networking on an x86 server was practical and, when combined with the move into server virtualization, could offer a viable alternative. Innovation in network hardware is measured in three-year product cycles, but the industry needed new answers immediately to handle the server mobility and network configuration using an application programming interface (API). This led to the development of overlay networking, in which innovation in the software switch of the hypervisor could be delivered without integration with the physical network through the process of encapsulation of ethernet in internet protocol (IP) packets.

In 2012, the European Telecommunications Standards Institute formed a working group called Network Functions Virtualization (NFV) to focus on SDN in carrier networks. As bandwidth moves to become a commodity, carriers are looking to add value to their core business with new services that are delivered from the network itself.

Additionally, vendors introduced new hardware devices to market that offer integration with overlay networking and a pathway for integration of existing networking to integrate with overlay networking. OpenFlow is no longer the technology keystone. The focus is now on the controller and the applications that configure, control, and operate the end-to-end services that the customer requires.

SDN timeline


Source: MyEtherealMind, GigaOM Research


3 The challenge

The broad strategy of building cloud computing services, either private or public, demands changes in all areas of IT.

  • Applications are changing from monolithic to distributed and must be much more interconnected and in real-time.
  • Servers have been virtualized.
  • Cloud computing presents irresistible opportunities for service providers and cloud infrastructure.
  • User access is becoming increasingly mobile, driving larger networks with dynamic configurations.

The current generation of network technologies does not adequately address the challenges of the next generation of computing services. Many network protocols were designed 10, 20, or even 30 years ago. Transmission control protocol/internet protocol (TCP/IP) remains largely unchanged since 1982, and routing protocols like open shortest path first (OSPF, 1998) and MPLS (2001) were designed to address challenges around the scale and resiliency of large-scale networks. They were not designed to support mobile devices or applications that were static.

In a data network, which is a single coherent system, each network device is connected to every other network device across the entire estate, so network changes are a serious business risk. To ensure reliability, each network router and switch acts autonomously to select the best path through the network and to select an alternate path in the event the best path fails. This process led to protocols that cannot allow loops and that must assume eventual path consistency instead of fast convergence. By today’s standards, these network systems are slow to react to change.

In recent years, the rising value of capital investment in building silicon fabrication plants combined with rapid product obsolescence is driving chip companies to operate at ever-higher volumes. In the past, chip fabrication tended toward small batches of chips that allowed customization to a specific task. Today, companies like Intel can manufacture large quantities of general-purpose x86 processors that have enough performance and capability to meet the requirements for increasingly real-time and interactive networking tasks.

Using x86 server hardware for networking means scale and flexibility. While this paper introduces software-defined networking and why it works, the practical outcome is that servers become the network edge. They’re integrated into an existing network fabric. Every additional server adds more performance and capacity but remains flexible because of the limited system impact at the edge. This echoes the MPLS value of a simple, fast core with a complex edge.

“Software-only” networking has two more unique features—speed of development and speed of change.

  • Speed of development means that software development for the x86 is so ubiquitous that tools and resources are readily available. The software is decoupled from the hardware and can progress independently of the hardware or device operating system. Compared to developing for custom network devices, product development is fast and cost-effective.
  • Speed of change refers to the use of application program interfaces (API) to apply operational changes with orchestration. Orchestration delivers on lowering the cost of operation and ownership by treating the network as one platform instead of many devices. SDN controllers now have deeper service integration than was previously possible because the virtual router (vRouter) intelligence is embedded at the edge of the network.

4 Network agents and overlays

vRouter technology

The use of vRouters, a software program that provides networking features that are part of the server operating system or hypervisor, enables network programmability. The vRouter interacts with the existing software commonly known as the network driver so that it can become a more complete part of the network platform.

The difficult business process of network operations resists change and has limited service flexibility. For business owners, the dilemma of information technology infrastructure library (ITIL) and information technology (IT) service management is that change is strictly controlled to manage risk and change control processes. For networking, the risk of a highly interconnected system is that routine network tasks are classified as high-risk and require long lead times. To reduce the perceived risk, network engineers spend many hours preparing and documenting even minor changes.

The rise of server virtualization and application networking has made this practice unworkable. Network engineers are prevented from meeting business needs by technology hardware limitations and ITIL processes. SDN provides answers to these problems by using mature, proven techniques in new ways, thus providing a network overlay that can be safely modified at any time using programmable interfaces.

Today, a network driver is simple software that transforms the data stream from the application into TCP sessions and IP packets. It has no awareness of the network itself or integration with it; it forwards packets to the switch or to the router for all handling.

Figure 1. Agent connects VMs to the physical network


Source: MyEtherealMind, GigaOM Research

Because the overlay network is abstracted from the physical network, data paths can be modified. The network fabric provides high bandwidth and consistent delivery while the overlay network can be changed as needed. The business process is enabled for low-risk network changes and makes network programmability possible as continuous delivery without change-control.

Network programmability

The vRouter and overlay networking are two key technology enablers for network programmability. Each server and hypervisor in the network uses the vRouter to provide access to the overlay network, which is independent of the physical network. The vRouter becomes the software edge of the network.

In networking, reliability and scale is consistently achieved by moving complex technology to the ingress point. Today, we add quality-of-service (QoS) policies to IP flows with the top-of-rack switch, and we manage WAN paths using MPLS provider edge (PE) routers. Complexity at the edge provides scaling of the network processing since each extra device provides more capacity to the overall system.

In a similar way, the vRouter moves the edge of the network inside the server operating system. In a major change for networking, the vRouter is not configured by a command-line interface (CLI), and the network paths are not determined by autonomous protocols like OSPF or border gateway protocol (BGP). Instead the vRouter is configured from a logically centralized software application known as a controller.

Overlay networking

Overlay networking is a general term for technology that creates a virtual network within the physical network; in other words, it separates the virtual network configuration from the physical network. Abstraction of virtual networking in an overlay allows for lower-risk network configuration and policy-based automation using programmable network interfaces.

The current generation of overlay networking uses tunneling protocols to connect a virtual switch in the server to another virtual switch. As data is received from the operating systems (OS), the local vRouter routes the data into a tunnel to the remote vRouter that connects to the next server. The physical network still switches ethernet frames and routes the IP packets to deliver a high-speed and reliable network fabric for the tunnel protocols.

Currently, the most common overlay protocol for a data center is VXLAN. VXLAN is optimized for use in ethernet/IP networks providing forwarding performance, load-balancing over multiple ethernet connections, and supporting multi-tenancy. In other overlay protocols, such as MPLS over generic routing encapsulation (GRE) or network virtualization using generic routing encapsulation (NVGRE), the tunneling protocol isn’t important as long as it is pervasive and technically suited to the underlying network. The overlay network that the protocol enables must be efficient, resilient, and flexible.

 Figure 2. Overlay network


5 The network controller

The controller uses an API to read and write from all vRouters in a single system. For example, the network controller can:

  • Write the configuration of the tunnels between the vRouters
  • Write the VLAN and IP networking data to the vRouter
  • Write the MAC address tables to vRouters based on the data of the network adapters presented to the VMs
  • Configure the vRouter when a VM moves between vRouters

The network controller has a complete end-to-end view of the overlay network and data about the vRouter’s capabilities. It can derive the location of all hypervisors, track the movement of VMs between hypervisors, program different network paths based on source and destination addressing, and will be able in the future to react to changing network conditions.

The network controller also becomes a network management application by reading data from the vRouter. In another significant change for networking, the network controller doesn’t rely on protocols like simple network management protocol (SNMP), which does not provide the features or real-time capabilities needed for today’s more dynamic application environments.

Figure 3. Controller and vRouters


Source: MyEtherealMind, GigaOM Research

Use case

The network controller can track the usage of a network port that connects to a virtual server by reading statistics from the vRouter. That server may move to a different hypervisor and connect to a different vRouter, yet the controller will still recognize the server and its network port and continue to record traffic statistics.

The network controller has greater potential than current network management tools simply because it has full- and real-time visibility of the entire network with data about the connected servers and their configuration. This access to such a complete picture of the network, the use of deeply integrated data sources using APIs, and data about the servers means that network controllers can provide more useful monitoring and status information than before.

Current platforms are based on read-only data using SNMP for static network devices. Most likely, controllers will replace existing network management applications over time. This change will begin in the data center and move into the WAN as controllers are proven.

The future of vRouters

Some people question whether vRouters have the necessary performance and capabilities to handle data centers. Reassurance comes from two key areas. First, this type of technology is deployed today in large-scale cloud environments, such as Rackspace and eBay. While these server farms are relatively simple compared to enterprise or carrier systems, scaling is generally accepted as proven.

A second concern is performance. For the last decade, two key factors have driven network performance—bandwidth and speed. The widespread deployment of 10-gigabit ethernet (GbE) is sufficient for edge network connections from server to network. Bonding of 10 GbE will be sufficient in most networks until 40-GbE and 100-GbE pricing becomes more acceptable in the next couple of years.

The vRouter does consume the central processing unit (CPU) and memory on the server. Tests suggest that performance of up to 10 GbE has a worst-case consumption of about 20 percent of a single CPU core in a server. Since most modern servers have quad-core architectures and CPU cycles are rarely the system bottleneck, this is a small trade-off for the significant advances in network flexibility.

New chips also may be added to server motherboards to support network processing. Today, the physical network adapter in a server uses a network processor to perform tasks such as TCP offload. In a similar way, the industry is starting to add silicon-switching fabrics to the server motherboard to better support vRouters.

6 Network services

In an SDN network, the vRouter can modify packets, select paths, perform simple firewalls, and complete load-balancing. Until today, delivering these types of network services has required physical appliances like load-balancers and firewalls. Managing all these dedicated appliances has made networking complex.

To understand network services, we must understand the key difference in the operation of vRouters. Current network technology is based on the premise of switching ethernet frames or forwarding IP packets. For a router, an IP packet is received, and the destination IP address is read from the packet header. The destination address is checked in the routing table to determine the output interface before dispatching the packet for transmission on the physical network according to the frame type.

An application is not aware of packets or network devices. The program simply opens a client network socket to a server network socket and exchanges a block of data. The segmentation of the data block into packets takes place in the network driver to allow the packets to traverse the network. Packets are used so that the network can be shared between many services to provide reliability in the event that data needs to be retransmitted and to allow path recovery without impacting the application.

The key point is that the data exchange is the required outcome. A data request may need to transmit a thousand packets to—and from—the same network source and destination. This session is commonly described as a flow. For a given server and client, the source and destination IP address will remain the same, and all of the packets for a given data exchange will remain the same, thus defining a flow.

vRouters use the concept of network flow to create the forwarding table. A network flow is the same for each server or client exchange and usually for multiple data exchanges when the client reuses the open TCP socket. Therefore, a network flow is usually defined as having the following five properties:

  • Source IP address
  • Source TCP port
  • Destination IP address
  • Destination TCP port
  • Protocol type

Example flow table for vRouter

Screen shot 2013-07-24 at 5.00.01 PM

Source: MyEtherealMind, GigaOM Research

An action is assigned for each flow in the table. Here we show the forward and drop actions. Other actions are possible, but they are beyond the scope of this report.

This is a rich feature set of forwarding decisions compared to existing methods. Today, a network switch or router will forward according to destination address with no awareness of the source or application protocol. The vRouter can forward according to source and destination data.

(Note that network flows can be more complex and include attributes like QoS and MPLS tags; however, that is outside the scope of this report.)

Types of network services

Now that we have outlined the nature of forwarding decision in a vRouter, we can consider which network services are possible and uncover some insight into how they work. The three services we’ll examine, in addition to the benefits of controller networking discussed previously, are:

  • Edge routing
  • Network security
  • Network load-balancing

Edge routing

The process of routing IP packets, which requires memory and data processing, leads to complex silicon designs that deliver high-bandwidth and throughput. The current practice is data center networks that perform IP routing in large core switches. Because the core switches handle large volumes of traffic and impact almost all services in a data center, one requirement is that low-latency and high-bandwidth routing be performed in custom silicon. The trade-off is that core-switch silicon is not able to provide routing services on other criteria, such as source and destination-routing. Maintaining even simpler tasks, such QoS handling, is complex and challenging on these platforms.

The use of overlay networking provides an opportunity for an edge vRouter to perform routing functions. Instead of routing onto a physical interface, the vRouter forwards a flow onto the tunnel that connects the endpoints. This is truly distributed routing on every server.

Integrating the vRouter with the WAN is also possible by establishing a tunnel directly from the vRouter to the WAN router (Figure 5), thus eliminating the need for a separate gateway, which has been a distinct disadvantage of early SDN overlay architectures.

A unique feature of Juniper’s JunosV Contrail is the use of MPLS over GRE tunnels to achieve direct MPLS integration from the vRouter to an MPLS WAN Network. Because most WAN routers can terminate GRE tunnels in hardware, this is a better choice for performance, low-latency, and interoperability with existing networking assets.

Figure 4. Edge routing within the data center and to WAN edge


Source: MyEtherealMind, GigaOM Research

The result is that the VM and agent are tightly integrated with the WAN service, they support programmatically configuration for new network services, and they provide richer end-to-end diagnostic capabilities for applications between data centers.

Network security

A flow table in the vRouter has “forward” actions to route into the tunnel; a drop action would be the equivalent of a network filter or access list. As before, the value of the controller changes this filtering from mildly useful to serious security progress. Configuring a network-access list to be useful requires a lot of effort. The network engineer must identify the source and destination and then manually configure it on the physical interface. In a virtual environment, building and implementing an access list on every physical interface is completely impractical.

The network controller overcomes these problems by having data on the servers assist in the creation of access lists by rules and validating them in the context of the entire network before applying the filter to the configuration of the virtual interface. If the virtual interface moves between vRouters, the access-list configuration will be applied in the new location by the controller. In order to avoid conflicting policies and to ensure compliance, these policies should be applied in context with broader network security practices.

Increasingly, security and network filtering is being performed at ingress to the network. This isn’t the same as stateful firewalling, but the control that is achieved at the network ingress provides extra controls that distribute the firewall function to the hosts.

Network load-balancing

vRouters can be configured to deliver two different modes of load-balancing—network load-balancing and server load-balancing. The client requesting a TCP connection initiates a session to a server, and the vRouter will add a new entry to its flow table for that specific flow with suitable actions to select the output port.

Network load-balancing is the use of the flow tables in the vRouter to distribute load across different paths in the overlay network. Each of those flow paths will traverse a different path in the physical network to share load across the available paths. This feature may be of limited use to the internal data center as LAN network enhancements, such as Juniper Networks’ QFabric and transparent interconnection of lots of links (TRILL), handle the load-balancing as a feature in ethernet. WAN services that connect data centers using multiple service providers have a number of possible deployment scenarios; Figure 7 shows an example in which multiple paths in the WAN are load-balanced between two data centers.

Figure 5. Network load-balancing over WAN by path


Source: MyEtherealMind, GigaOM Research

Server load-balancing is a broad term for a technology that distributes client connections between different servers. In this system, the network controller modifies the flow table to allocate the connections between servers by setting the actions. Consider a flow table with four flows to the same server at and two of each flow forwarded to and, respectively (Figure 8).

Demonstration flow table for server load-balancing

Screen shot 2013-07-24 at 5.08.25 PM

Source: MyEtherealMind, GigaOM Research

This capability is similar to that provided by expensive and complex application delivery controllers today and could be replaced by more distributed high-performance server load-balancing as part of an SDN solution.

7 Network functions virtualization

SDN networking is a strategic element in enabling network functions virtualization (NFV).

NFV is a statement from the European Telecommunications Standards Institute (ETSI) about the challenge for service providers around proprietary hardware appliances in the network. Service providers are looking to grow revenue with new services to customers, which leads to significant capital investment in hardware before the revenue is earned as well as project costs to deploy new hardware to the network. Proprietary hardware requires regular replacement with limited revenue upside. In fact, many custom appliances are x86 systems today.

Services like video-streaming, virus-scanning, and content-caching could be offered as software on a virtualization platform in the exchange. Processes that are similar to the data center would see standard server hardware added to provide more processing power. Storage also can be standardized.

This simplification of the appliance hardware would dramatically increase the flexibility of the exchange equipment. Using the methods we have described in this paper, customer data flows can be directly configured to consume services using controller networking. The controller would be accessible by a service delivery system to configure services for each customer according to customer account.

Also important is that services are easily composed from elements.

Figure 6. Network functions virtualization relationship with SDN

Screen shot 2013-07-24 at 5.12.53 PM

Source: ETSI network functions virtualization (NFV) whitepaper

8 Commoditization

Many believe that software-defined networking will lead to commoditization of the network hardware market and create significant disruption. In the section on overlay networks, we reviewed how overlay networks abstract the capabilities of the physical networks, and in the load-balancing section, we highlighted that ethernet has its own load-balancing capabilities.

Similarly, IP networks using technologies like MPLS or IP routing will provide features that enhance SDN. Overlay networking and network controllers add many new features to networking designs by creating abstraction for dynamic configuration and change. At the same time, this abstraction also creates opportunities for hardware vendors to offer new services in hardware. For example, increasing the size of the ternary content-addressable memory (TCAM) and binary content-addressable memory (BCAM) silicon in network hardware is vital to accommodate larger IP and MAC address tables for data networks that have many thousands of hosts. This silicon is expensive to design and manufacture and consumes a great deal of power. Overlay networking removes the need to grow TCAM/BCAM space since each tunnel is a single entry between hypervisors instead of an entry per TCP/IP flow, an order of magnitude reduction.

Instead hardware players can consider the long list of other features to be added to these devices, such as:

  • Better configuration API for automation
  • New methods of management, such as Puppet or Chef, to improve provisioning
  • Improved QoS
  • Protocol translations
  • Better sFlow and virtualization
  • Support for multilayer ethernet
  • Distributed system optimization

The list goes on. Of course, we can reasonably expect that new integrations between the physical and overlay networks will be developed to enhance the reliability and capability of overlay networks.

9 The final verdict

This paper highlights that SDN is a number of technologies that coalesce into a single solution. The importance of SDN is less about technology than about extracting more business value from networking.

The use of overlay networking creates a single network that is simpler to operate and manage by moving complexity to the edge of the network. This complexity is managed by a software network controller with software APIs at the system level that creates new opportunities for integration with service orchestration and operational managements systems.

SDN can improve network operations in more dynamic elastic cloud environments. Superior overlay approaches will provide significantly richer data sources, as well as visibility that is extended to include the server and applications, and because the network controller contains all the configuration data of each vRouter, details of the server network connection and real-time flow information are exposed in a unified interface.

Overlay networking has the flexibility and capability to change the forwarding path of the traffic flows dynamically and without taking any risks in the physical network. It also can optimize network service delivery. Finally, the vRouter can be used to inject native IP services into the network layer. The methods used for service injection have been around for some time, but the network controller provides programmatic control and visibility and combines with overlay networking to make these methods practical for widespread use.

This overview of SDN clarifies that the market momentum behind it signals widespread demand for the services and capabilities it can deliver.

SDN evolves an existing network by building on existing physical networks and adding an overlay network to enhance operational change processes and features. In order to scale and provide customer investment protection, it is critical that SDN systems ensure open interoperability across controllers and existing IP networks. While IT leadership needs to appreciate the significant technology benefits, the most significant benefit should be realized as dramatic improvements in business agility and the accelerated revenue growth realized by more rapid delivery of competitive services.

10 About Greg Ferro

Greg Ferro is a freelance network architect and engineer currently working in Great Britain for Fortune 100 companies on a wide range of enterprise networking and security technologies. He has more than 20 years of experience in networking with a recent focus on architecture and design for cloud networking. Security infrastructure is also a key part of his expertise.

He is the co-host of the Packet Pushers Podcast, a weekly podcast on data networking. Covering all areas of networking, Packet Pushers is a roundtable discussion of users, customers, and vendors who cover the networking industry. The discussion is focused on enterprise and cloud networking.

Ferro also writes regularly for Network Computing, for which he covers a range of networking topics, including product and strategy from the vendors, technology reviews, and “coalface” experiences, and he write a blog, Ethereal Mind.

11 About GigaOM Research

GigaOM Pro gives you insider access to expert industry insights on emerging markets. Focused on delivering highly relevant and timely research to the people who need it most, our analysis, reports, and original research come from the most respected voices in the industry. Whether you’re beginning to learn about a new market or are an industry insider, GigaOM Research addresses the need for relevant, illuminating insights into the industry’s most dynamic markets.

Visit us at:

12 Copyright

© Knowingly, Inc. 2013. "Evolving SDN: Tackling challenges for web-scale deployments" is a trademark of Knowingly, Inc. For permission to reproduce this report, please contact