Evolution of Virtualization

Virtualization in the x86 market has dramatically evolved over the past decade with significant contributions from different players in different technologies like Hardware manufacturers (Intel, AMD), storage providers (Netapp, EMC), Server manufacturers (HP, Cisco), hypervisor software vendors (VMWare, Citrix, MS) and Management app vendors (all big players and countless startups).

In this post, I will describe the major stages in the whole evolutionary process of virtualization.

First Stage - Binary Translation:

It all started in the research labs of VMWare where Rosenbalm and his team were working on a piece of software called the hypervisor that gave the ability to run multiple OS environments on the same piece of hardware.

The traditional Operating Systems ran on Ring 0 and the Applications would run on Ring 3. But with virtualization, the hypervisor ran on Ring 0 and that forced the OS to run on Ring 1. This meant that certain previleged CPU instructions generated by the OS would fail since it was running on Ring 1. Rosenbalm and his team invented the technique of binary translation to trap these system calls and run them on behalf of the OS. Over the years, VMWare has effeciently optimized the binary translation technique and has brought down the hypervisor overhead to single digits.


While VMWare was busy perfecting its binary translation code, Intel and AMD were working on getting the virtualization capabilities into the processor itself that led to the next stage in virtualization.

Second Stage - Hardware Virtualization

To overcome the problem of running OS environments on Ring 1, chip vendors developed virtualization enabled processors (Intel VT and AMD Pacifica). These technologies allowed the hypervisor to run on Ring -1 and share the hardware resources between the different OS environments (VMs) running on Ring 0.


Though the first generation of hardware virtualization did not achieve any significant performance gain over VMWare's binary translation technique, it opened the gates for vendors to quickly develop their own hypervisors. Xen was one of the first few to leverage hardware virtualization and develop an enterprise class hypervisor.

Third Stage - Memory and IO Virtualization:

Till the recent release of the Intel Nehelam processors, the hypervisor had to manage the mapping from the physical memory pages to the virtual pages allotted to the VMs. They used shadow page tables to manage this. With Nehelam, Intel has built Virtualized Memory management into the processor itself. This should free up some more CPU cycles utilized by the hypervisor.

Also the hypervisor network stack was used to process the guest network traffic. This has been a serious bottleneck in scaling the IO performance of the hypervisors. This has been overcome with Intel shipping virtualized networking adapters. This has provided the ability to directly present the physical network adapter to the VM. Refer to the post from Simon Crosby that indicates the performance enhancements when using the SR-IOV 10G adapters with Citrix XenServer.


With all these improvements, a large portion of the hypervisor functionalities has been sunk into the hardware itself. This has enabled the hypervisors to achieve near native performance and pushed the vendors to build more stable and scalable management apps and advance into the cloud computing paradigm