Now who are the players? the Project Sponsor (CIO, Dir of IT, Sr VP of IT), Key application owners, Server architecture lead, LAN/WAN architecture leads, Data Center Mgr, and Security lead. Besides the above mentioned benefits of putting all these people in the room, it also cuts down the delay in Q&A. If someone has a question, there is no more of the “we have to ask the LAN guys”, they are right there in the room.

I’ve done many of these facilitated sessions and I can guarantee that when everyone walks out of the room after a few days they all have a vested interest in the final solution.

Carmelo Lisciotto

If a server “has a failure”, how can moving the VM be done without any downtime?

(of course, if all your customers are in one location - say, academia or a compute cluster somewhere - distributing load geographically may not make sense, but the concept still applies on the local level, I think.)

@darkuncle - the folks I met with were using VLANs, but probably not as you describe. I’ll dig into that more with 802.1q trunks on the servers.

@Daon - thanks for the comments and sharing your experiences.

We’ve had a stateless Grid for over 7 years, which allowed us not to migrate a live instance, but extremely rapidly move a shutdown instance to any server in the Grid. We use PXE to boot the servers from NFS volumes on NAS. When we only had 240 servers in the grid it wasn’t difficult to have them all in a single network space.

Then the Grid continued growing and we had added requirements for more than simple http access from the internet. Security and networks began to grow more and more concerned about the environment. About 4 years ago we realized we needed to go virtual to remain cost effective and be able to scale as the business continued growing, which finally opened the first shots in the war.

There are a number of vendors with solutions attempting to resolve it. As mentioned by stockandawe, Cisco VFrame is one. We were lucky enough to be part of a very early beta for this product. While Cisco has done a excellent job of seeing the nature of the problem and attempting to create a solution, it has one major draw back. VFrame needs complete control of the environment to work it and both Security and Networks were unable to overcome cultural and very real concerns about doing this.

Scalent (as noted by virtualman), Xsigo and a few others have taken a similar approach, but all face the same challenges. The cultural issues can’t be ignored. It’s ironic that as technologists we can fall into the same trap we so often accuse non-technologists: Fear of technology because it’s new and untried.

Many companies are going to be unwilling to be the first to adopt these new technologies because they not proven by wide acceptance. It’s often up to the smaller companies that aren’t afraid to take a risk and are capable of being more agile to adopt these technologies and prove they are reliable.

I think darkuncle hit the nail on the head, when he said everyone needs to be involved from the start. By involving networks security, apps and systems architects and engineers from the start you have the opportunity to support one another through large leaps of faith.

We haven’t really achieved the holy grail of a fully virtualized environment through all components and infrastructure yet, but we’re getting there. We’ve moved the entire Grid behind SSL-VPN and much like darkuncle, we’ve used VLANs to gain some flexibility, but that has limits as well. Our Grid has grown significantly and can host upto 6000 or so VMs, which is like a small datacenter unto itself. Having a single network space encompass so many systems adds complexity and risk.

Until more of the bigger vendors become more engaged in this space, it’s going to be a challenge finding solutions that make everyone happy.

In this particular case, the correct approach (well, one correct approach anyway) is to use VLANs, tag all the server VLANs that will be in use for any VMs to all ESX hosts, and firewall based on VLAN, rather than based on physical host. This approach tends to be more flexible and scalable for neteng in addition to the benefits for the systems and application folks (speaking from experience in building out just such an architecture).

From the website:

“Scalent…lets data centers change server software, network connectivity, and storage access in real-time”

and

“Scalent code ships in VMware ESX Server”.

Seems like a network-boot solution already exists to do the first-stage work you’re describing, Allan. (EG, power on a machine, set up the right network and storage connectivity, and boot the right hypervisor…)

Scalent also appears to be resold by HP, Unisys, and EMC. I thought I saw Scalent listed in the Nexus press release too (they are in the Google cached version), but then they’re missing on the Cisco site version…?

http://gigaom.com/2008/03/21/coming-soon-the-cisco-blade-server/
http://advice.cio.com/allan_leinwand/who_will_control_the_next_enterprise_data_center

Check out this talk on Cisco Nexus 5000 (which is a network entity) and how it it designed (theoretically) to enable VMWare (which is a server-virtualization entity).
http://www.youtube.com/watch?v=ZOFRjWSKPvI

Disclaimer: I am not a systems/network/virtualization engineer, but in a past life did work at Cisco.