A couple of recent posts from VMWare and Citrix seemed to have an interesting correlation. Both of these posts have a very similiar context - dealing with their hypervisor performance with 10 Gig Ethernet Adapters.
The final analysis from the VMWare camp:
Two virtual machines can easily saturate a 10Gbps link (the practical limit is 9.3 Gbps for packets that use the standard MTU size because of protocol overheads), and the thoughput remains constant as we add more virtual machines.
Using Jumbo frames, a single virtual machine can saturate a 10Gbps link on the transmit path and can recieve netwrk traffic at rates up to 5.7 Gbps.
Citrix also announced very similiar results:
we have basically maxxed out bidirectional I/O on a single 10Gb/s link, with only 4 guests. The exact figures are 18.50 Gbps (bidirectional) using 4 guest machines.
I can read the question in your mind: How can the simple Linux Networking stack in XenServer match the virtualization optimized I/O Stack of the ESX VMKernel? Well you got a very valid question and I would like to point you to the dates when the two performance tests were done. ESX performance was done in Nov 2008 and the XenServer results have been declared in April 2009. So what is it that has drastically changed in the 5 months?
The answer is the new Nehelam processors from Intel. The Citrix tests were conducted using the Nehelam processors and IOV enhanced 10Gbps NICs with Solarflare IO acceleration. This offers a powerful direct hardware-to-guest acceleration path that avoids the necessity for the hypervisor to process I/O on behalf of the guests. When the same tests were conducted without the IO acceleration, the XenServer hypervisor CPU got maxxed out at 8.45 Gbps (bidirectional)!!!.
This result leads to an interesting conclusion: Though the optimized VMKernel is far superior to the Xen Kernel, a lot of these features are being offset by the hardware virtualization capabilities built into the processors and NIC adapters by the hardware vendors. This has helped XenServer match or sometimes outperform ESX in terms of hyppervisor performance with both CPU and IO intensive workloads. Believe the IOV Ethernet Adapters should also help XenServer overcome the 6 supported physical NICs limitation.
For a customer, I believe it is the final numbers that matter more than how the performance is achieved. I would rather go with a hardware optimized platform than rely on the s/w stacks to drive performance. What do you say?
Sunday, April 19, 2009
Tuesday, April 14, 2009
Evolution of Virtualization
Virtualization in the x86 market has dramatically evolved over the past decade with significant contributions from different players in different technologies like Hardware manufacturers (Intel, AMD), storage providers (Netapp, EMC), Server manufacturers (HP, Cisco), hypervisor software vendors (VMWare, Citrix, MS) and Management app vendors (all big players and countless startups).
In this post, I will describe the major stages in the whole evolutionary process of virtualization.
First Stage - Binary Translation:
It all started in the research labs of VMWare where Rosenbalm and his team were working on a piece of software called the hypervisor that gave the ability to run multiple OS environments on the same piece of hardware.
The traditional Operating Systems ran on Ring 0 and the Applications would run on Ring 3. But with virtualization, the hypervisor ran on Ring 0 and that forced the OS to run on Ring 1. This meant that certain previleged CPU instructions generated by the OS would fail since it was running on Ring 1. Rosenbalm and his team invented the technique of binary translation to trap these system calls and run them on behalf of the OS. Over the years, VMWare has effeciently optimized the binary translation technique and has brought down the hypervisor overhead to single digits.
While VMWare was busy perfecting its binary translation code, Intel and AMD were working on getting the virtualization capabilities into the processor itself that led to the next stage in virtualization.
Second Stage - Hardware Virtualization
To overcome the problem of running OS environments on Ring 1, chip vendors developed virtualization enabled processors (Intel VT and AMD Pacifica). These technologies allowed the hypervisor to run on Ring -1 and share the hardware resources between the different OS environments (VMs) running on Ring 0.
Though the first generation of hardware virtualization did not achieve any significant performance gain over VMWare's binary translation technique, it opened the gates for vendors to quickly develop their own hypervisors. Xen was one of the first few to leverage hardware virtualization and develop an enterprise class hypervisor.
Third Stage - Memory and IO Virtualization:
Till the recent release of the Intel Nehelam processors, the hypervisor had to manage the mapping from the physical memory pages to the virtual pages allotted to the VMs. They used shadow page tables to manage this. With Nehelam, Intel has built Virtualized Memory management into the processor itself. This should free up some more CPU cycles utilized by the hypervisor.
Also the hypervisor network stack was used to process the guest network traffic. This has been a serious bottleneck in scaling the IO performance of the hypervisors. This has been overcome with Intel shipping virtualized networking adapters. This has provided the ability to directly present the physical network adapter to the VM. Refer to the post from Simon Crosby that indicates the performance enhancements when using the SR-IOV 10G adapters with Citrix XenServer.
+.JPG)
With all these improvements, a large portion of the hypervisor functionalities has been sunk into the hardware itself. This has enabled the hypervisors to achieve near native performance and pushed the vendors to build more stable and scalable management apps and advance into the cloud computing paradigm
In this post, I will describe the major stages in the whole evolutionary process of virtualization.
First Stage - Binary Translation:
It all started in the research labs of VMWare where Rosenbalm and his team were working on a piece of software called the hypervisor that gave the ability to run multiple OS environments on the same piece of hardware.
The traditional Operating Systems ran on Ring 0 and the Applications would run on Ring 3. But with virtualization, the hypervisor ran on Ring 0 and that forced the OS to run on Ring 1. This meant that certain previleged CPU instructions generated by the OS would fail since it was running on Ring 1. Rosenbalm and his team invented the technique of binary translation to trap these system calls and run them on behalf of the OS. Over the years, VMWare has effeciently optimized the binary translation technique and has brought down the hypervisor overhead to single digits.
While VMWare was busy perfecting its binary translation code, Intel and AMD were working on getting the virtualization capabilities into the processor itself that led to the next stage in virtualization.
Second Stage - Hardware Virtualization
To overcome the problem of running OS environments on Ring 1, chip vendors developed virtualization enabled processors (Intel VT and AMD Pacifica). These technologies allowed the hypervisor to run on Ring -1 and share the hardware resources between the different OS environments (VMs) running on Ring 0.
Though the first generation of hardware virtualization did not achieve any significant performance gain over VMWare's binary translation technique, it opened the gates for vendors to quickly develop their own hypervisors. Xen was one of the first few to leverage hardware virtualization and develop an enterprise class hypervisor.
Third Stage - Memory and IO Virtualization:
Till the recent release of the Intel Nehelam processors, the hypervisor had to manage the mapping from the physical memory pages to the virtual pages allotted to the VMs. They used shadow page tables to manage this. With Nehelam, Intel has built Virtualized Memory management into the processor itself. This should free up some more CPU cycles utilized by the hypervisor.
Also the hypervisor network stack was used to process the guest network traffic. This has been a serious bottleneck in scaling the IO performance of the hypervisors. This has been overcome with Intel shipping virtualized networking adapters. This has provided the ability to directly present the physical network adapter to the VM. Refer to the post from Simon Crosby that indicates the performance enhancements when using the SR-IOV 10G adapters with Citrix XenServer.
+.JPG)
With all these improvements, a large portion of the hypervisor functionalities has been sunk into the hardware itself. This has enabled the hypervisors to achieve near native performance and pushed the vendors to build more stable and scalable management apps and advance into the cloud computing paradigm
Subscribe to:
Posts (Atom)