x86 virtualization is the use of hardware-assisted virtualization capabilities on an x86/x86-64 CPU.
In the late 1990s x86 virtualization was achieved by complex software techniques, necessary to compensate for the processor's lack of hardware-assisted virtualization capabilities while attaining reasonable performance. In 2005 and 2006, both Intel ( VT-x) and AMD (
AMD-V
x86 virtualization is the use of hardware-assisted virtualization capabilities on an x86/x86-64 CPU.
In the late 1990s x86 virtualization was achieved by complex software techniques, necessary to compensate for the processor's lack of hardware-as ...
) introduced limited hardware virtualization support that allowed simpler virtualization software but offered very few speed benefits. Greater hardware support, which allowed substantial speed improvements, came with later processor models.
Software-based virtualization
The following discussion focuses only on virtualization of the x86 architecture
protected mode
In computing, protected mode, also called protected virtual address mode, is an operational mode of x86-compatible central processing units (CPUs). It allows system software to use features such as virtual memory, paging and safe multi-tasking d ...
.
In protected mode the operating system kernel runs at a higher privilege such as ring 0, and applications at a lower privilege such as ring 3. In software-based virtualization, a host OS has direct access to hardware while the guest OSs have limited access to hardware, just like any other application of the host OS. One approach used in x86 software-based virtualization to overcome this limitation is called ''ring deprivileging'', which involves running the guest OS at a ring higher (lesser privileged) than 0.
Three techniques made virtualization of protected mode possible:
* Binary translation is used to rewrite in terms of ring 3 instructions certain ring 0 instructions, such as POPF, that would otherwise fail silently or behave differently when executed above ring 0, making the classic trap-and-emulate virtualization impossible. To improve performance, the translated basic blocks need to be cached in a coherent way that detects code patching (used in VxDs for instance), the reuse of pages by the guest OS, or even self-modifying code.
* A number of key data structures used by a processor need to be shadowed. Because most operating systems use paged virtual memory, and granting the guest OS direct access to the MMU would mean loss of control by the virtualization manager, some of the work of the x86 MMU needs to be duplicated in software for the guest OS using a technique known as ''shadow page tables''. This involves denying the guest OS any access to the actual page table entries by trapping access attempts and emulating them instead in software. The x86 architecture uses hidden state to store segment descriptors in the processor, so once the segment descriptors have been loaded into the processor, the memory from which they have been loaded may be overwritten and there is no way to get the descriptors back from the processor. ''Shadow descriptor tables'' must therefore be used to track changes made to the descriptor tables by the guest OS.
* I/O device emulation: Unsupported devices on the guest OS must be emulated by a device emulator that runs in the host OS.
These techniques incur some performance overhead due to lack of MMU virtualization support, as compared to a VM running on a natively virtualizable architecture such as the IBM System/370.
On traditional mainframes, the classic type 1 hypervisor was self-standing and did not depend on any operating system or run any user applications itself. In contrast, the first x86 virtualization products were aimed at workstation computers, and ran a guest OS inside a host OS by embedding the hypervisor in a kernel module that ran under the host OS (type 2 hypervisor).
There has been some controversy whether the x86 architecture with no hardware assistance is virtualizable as described by Popek and Goldberg. VMware researchers pointed out in a 2006 ASPLOS paper that the above techniques made the x86 platform virtualizable in the sense of meeting the three criteria of Popek and Goldberg, albeit not by the classic trap-and-emulate technique.
A different route was taken by other systems like Denali, L4, and
Xen
Xen (pronounced ) is a type-1 hypervisor, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. It was
originally developed by the University of Cambridge Computer Laboratory an ...
, known as paravirtualization, which involves porting operating systems to run on the resulting virtual machine, which does not implement the parts of the actual x86 instruction set that are hard to virtualize. The paravirtualized I/O has significant performance benefits as demonstrated in the original SOSP'03 Xen paper.
The initial version of x86-64 ( AMD64) did not allow for a software-only full virtualization due to the lack of segmentation support in
long mode
In the x86-64 computer architecture, long mode is the mode where a 64-bit operating system can access 64-bit instructions and registers. 64-bit programs are run in a sub-mode called 64-bit mode, while 32-bit programs and 16-bit protected mode pr ...
, which made the protection of the hypervisor's memory impossible, in particular, the protection of the trap handler that runs in the guest kernel address space. Revision D and later 64-bit AMD processors (as a rule of thumb, those manufactured in 90 nm or less) added basic support for segmentation in long mode, making it possible to run 64-bit guests in 64-bit hosts via binary translation. Intel did not add segmentation support to its x86-64 implementation ( Intel 64), making 64-bit software-only virtualization impossible on Intel CPUs, but Intel VT-x support makes 64-bit hardware assisted virtualization possible on the Intel platform.
On some platforms, it is possible to run a 64-bit guest on a 32-bit host OS if the underlying processor is 64-bit and supports the necessary virtualization extensions.
Hardware-assisted virtualization
In 2005 and 2006, Intel and AMD (working independently) created new processor extensions to the x86 architecture. The first generation of x86 hardware virtualization addressed the issue of privileged instructions. The issue of low performance of virtualized system memory was addressed with MMU virtualization that was added to the chipset later.
Central processing unit
Virtual 8086 mode
Based on painful experiences with the 80286 protected mode, which by itself was not suitable enough to run concurrent DOS applications well, Intel introduced the virtual 8086 mode in their
Athlon 64
The Athlon 64 is a ninth-generation, AMD64-architecture microprocessor produced by Advanced Micro Devices (AMD), released on September 23, 2003. It is the third processor to bear the name ''Athlon'', and the immediate successor to the Athlon XP. T ...
and Athlon 64 X2 family of processors with revisions "F" or "G" on
socket AM2
The Socket AM2, renamed from Socket M2 (to prevent using the same name as Cyrix MII processors), is a CPU socket designed by AMD for desktop processors, including the performance, mainstream and value segments. It was released on May 23, 2006, as ...
, Turion 64 X2, and Opteron 2nd generation and third-generation, Phenom and Phenom II processors. The APU Fusion processors support AMD-V. AMD-V is not supported by any Socket 939 processors. The only Sempron processors which support it are APUs and Huron,
Regor
Gamma Velorum is a quadruple star system in the constellation Vela. This name is the Bayer designation for the star, which is Latinised from γ Velorum and abbreviated γ Vel. At a combined magnitude of +1.7, it is one of t ...
, Sargas desktop CPUs.
AMD Opteron CPUs beginning with the Family 0x10 Barcelona line, and Phenom II CPUs, support a second generation hardware virtualization technology called Rapid Virtualization Indexing (formerly known as Nested Page Tables during its development), later adopted by Intel as Extended Page Tables (EPT).
As of 2019, all Zen-based AMD processors support AMD-V.
The CPU flag for AMD-V is "svm". This may be checked in BSD derivatives via dmesg or sysctl and in Linux via /proc/ cpuinfo. Instructions in AMD-V include VMRUN, VMLOAD, VMSAVE, CLGI, VMMCALL, INVLPGA, SKINIT, and STGI.
With some
motherboard
A motherboard (also called mainboard, main circuit board, mb, mboard, backplane board, base board, system board, logic board (only in Apple computers) or mobo) is the main printed circuit board (PCB) in general-purpose computers and other expand ...
s, users must enable AMD SVM feature in the
BIOS
In computing, BIOS (, ; Basic Input/Output System, also known as the System BIOS, ROM BIOS, BIOS ROM or PC BIOS) is firmware used to provide runtime services for operating systems and programs and to perform hardware initialization during the ...
setup before applications can make use of it.
Intel virtualization (VT-x)
Previously codenamed "Vanderpool", VT-x represents Intel's technology for virtualization on the x86 platform. On November 13, 2005, Intel released two models of Pentium 4 (Model 662 and 672) as the first Intel processors to support VT-x. The CPU flag for VT-x capability is "vmx"; in Linux, this can be checked via /proc/cpuinfo, or in macOS via sysctl machdep.cpu.features.To see if your processor supports hardware virtualization Intel 2012.
"VMX" stands for Virtual Machine Extensions, which adds 13 new instructions: VMPTRLD, VMPTRST, VMCLEAR, VMREAD, VMWRITE, VMCALL, VMLAUNCH, VMRESUME, VMXOFF, VMXON, INVEPT, INVVPID, and VMFUNC. These instructions permit entering and exiting a virtual execution mode where the guest OS perceives itself as running with full privilege (ring 0), but the host OS remains protected.
, almost all newer server, desktop and mobile Intel processors support VT-x, with some of the Intel Atom processors as the primary exception. With some
motherboard
A motherboard (also called mainboard, main circuit board, mb, mboard, backplane board, base board, system board, logic board (only in Apple computers) or mobo) is the main printed circuit board (PCB) in general-purpose computers and other expand ...
s, users must enable Intel's VT-x feature in the
BIOS
In computing, BIOS (, ; Basic Input/Output System, also known as the System BIOS, ROM BIOS, BIOS ROM or PC BIOS) is firmware used to provide runtime services for operating systems and programs and to perform hardware initialization during the ...
setup before applications can make use of it.
Intel started to include Extended Page Tables (EPT), a technology for page-table virtualization, since the Nehalem architecture, released in 2008. In 2010, Westmere added support for launching the logical processor directly in
real mode
Real mode, also called real address mode, is an operating mode of all x86-compatible CPUs. The mode gets its name from the fact that addresses in real mode always correspond to real locations in memory. Real mode is characterized by a 20-bit seg ...
a feature called "unrestricted guest", which requires EPT to work.
Since the Haswell microarchitecture (announced in 2013), Intel started to include ''VMCS shadowing'' as a technology that accelerates nested virtualization of VMMs.
The ''virtual machine control structure'' (VMCS) is a
data structure
In computer science, a data structure is a data organization, management, and storage format that is usually chosen for efficient access to data. More precisely, a data structure is a collection of data values, the relationships among them, a ...
in memory that exists exactly once per VM, while it is managed by the VMM. With every change of the execution context between different VMs, the VMCS is restored for the current VM, defining the state of the VM's virtual processor. As soon as more than one VMM or nested VMMs are used, a problem appears in a way similar to what required shadow page table management to be invented, as described above. In such cases, VMCS needs to be shadowed multiple times (in case of nesting) and partially implemented in software in case there is no hardware support by the processor. To make shadow VMCS handling more efficient, Intel implemented hardware support for VMCS shadowing.
VIA virtualization (VIA VT)
VIA Nano 3000 Series Processors and higher support VIA VT virtualization technology compatible with Intel VT-x. EPT is present in Zhaoxin ZX-C, a descendant of VIA QuadCore-E & Eden X4 similar to Nano C4350AL.
Interrupt virtualization (AMD AVIC and Intel APICv)
In 2012, AMD announced their ''Advanced Virtual Interrupt Controller'' (''AVIC'') targeting interrupt overhead reduction in virtualization environments. This technology, as announced, does not support x2APIC.
In 2016, AVIC is available on the AMD family 15h models 6Xh
(Carrizo) processors and newer.
Also in 2012, Intel announced a similar technology for interrupt and APIC virtualization, which did not have a brand name at its announcement time.
Later, it was branded as ''APIC virtualization'' (''APICv'')
and it became commercially available in the Ivy Bridge EP series of Intel CPUs, which is sold as Xeon E5-26xx v2 (launched in late 2013) and as Xeon E5-46xx v2 (launched in early 2014).
Graphics processing unit
Graphics virtualization is not part of the x86 architecture. Intel
Graphics Virtualization Technology
Intel Graphics Technology (GT) is the collective name for a series of integrated graphics processors (IGPs) produced by Intel that are manufactured on the same package or die as the central processing unit (CPU). It was first introduced in 2 ...
(GVT) provides graphics virtualization as part of more recent Gen graphics architectures. Although AMD APUs implement the x86-64 instruction set, they implement AMD's own graphics architectures ( TeraScale, GCN and RDNA) which do not support graphics virtualization. Larrabee was the only graphics
microarchitecture
In computer engineering, microarchitecture, also called computer organization and sometimes abbreviated as µarch or uarch, is the way a given instruction set architecture (ISA) is implemented in a particular processor. A given ISA may be impl ...
based on x86, but it likely did not include support for graphics virtualization.
Chipset
Memory and I/O virtualization is performed by the chipset. Typically these features must be enabled by the BIOS, which must be able to support them and also be set to use them.
I/O MMU virtualization (AMD-Vi and Intel VT-d)
An input/output memory management unit (IOMMU) allows guest virtual machines to directly use
peripheral
A peripheral or peripheral device is an auxiliary device used to put information into and get information out of a computer. The term ''peripheral device'' refers to all hardware components that are attached to a computer and are controlled by the ...
devices, such as Ethernet, accelerated graphics cards, and hard-drive controllers, through
DMA
DMA may refer to:
Arts
* ''DMA'' (magazine), a defunct dance music magazine
* Dallas Museum of Art, an art museum in Texas, US
* Danish Music Awards, an award show held in Denmark
* BT Digital Music Awards, an annual event in the UK
* Doctor of M ...
and interrupt remapping. This is sometimes called ''PCI passthrough''.
An IOMMU also allows operating systems to eliminate bounce buffers needed to allow themselves to communicate with peripheral devices whose memory address spaces are smaller than the operating system's memory address space, by using memory address translation. At the same time, an IOMMU also allows operating systems and hypervisors to prevent buggy or malicious hardware from compromising memory security.
Both AMD and Intel have released their IOMMU specifications:
* AMD's I/O Virtualization Technology, "AMD-Vi", originally called "IOMMU"
* Intel's "Virtualization Technology for Directed I/O" (VT-d), included in most high-end (but not all) newer Intel processors since the Core 2 architecture.
In addition to the CPU support, both
motherboard
A motherboard (also called mainboard, main circuit board, mb, mboard, backplane board, base board, system board, logic board (only in Apple computers) or mobo) is the main printed circuit board (PCB) in general-purpose computers and other expand ...
BIOS
In computing, BIOS (, ; Basic Input/Output System, also known as the System BIOS, ROM BIOS, BIOS ROM or PC BIOS) is firmware used to provide runtime services for operating systems and programs and to perform hardware initialization during the ...
or UEFI) need to fully support the IOMMU I/O virtualization functionality for it to be usable. Only the
PCI
PCI may refer to:
Business and economics
* Payment card industry, businesses associated with debit, credit, and other payment cards
** Payment Card Industry Data Security Standard, a set of security requirements for credit card processors
* Pro ...
or
PCI Express
PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe or PCI-e, is a high-speed serial computer expansion bus standard, designed to replace the older PCI, PCI-X and AGP bus standards. It is the common ...
devices supporting ''function level reset'' (FLR) can be virtualized this way, as it is required for reassigning various device functions between virtual machines. If a device to be assigned does not support
Message Signaled Interrupts
Message Signalled Interrupts (MSI) are an alternative in-band method of signalling an interrupt, using special in-band messages to replace traditional out-of-band assertion of dedicated interrupt lines. While more complex to implement in a devic ...
(MSI), it must not share interrupt lines with other devices for the assignment to be possible.
All conventional PCI devices routed behind a PCI/ PCI-X-to-PCI Express bridge can be assigned to a guest virtual machine only all at once; PCI Express devices have no such restriction.
Network virtualization (VT-c)
* Intel's "Virtualization Technology for Connectivity" (VT-c).
= PCI-SIG Single Root I/O Virtualization (SR-IOV)
=
''PCI-SIG Single Root I/O Virtualization (SR-IOV)'' provides a set of general (non-x86 specific) I/O virtualization methods based on
PCI Express
PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe or PCI-e, is a high-speed serial computer expansion bus standard, designed to replace the older PCI, PCI-X and AGP bus standards. It is the common ...
(PCIe) native hardware, as standardized by PCI-SIG:
* ''Address translation services (ATS)'' supports native IOV across PCI Express via address translation. It requires support for new transactions to configure such translations.
* ''
Single-root IOV
In virtualization, single root input/output virtualization (SR-IOV) is a specification that allows the isolation of PCI Express resources for manageability and performance reasons.
Details
A single physical PCI Express bus can be shared in a vir ...
(SR-IOV or SRIOV)'' supports native IOV in existing single-root complex PCI Express topologies. It requires support for new device capabilities to configure multiple virtualized configuration spaces.
* ''Multi-root IOV (MR-IOV)'' supports native IOV in new topologies (for example, blade servers) by building on SR-IOV to provide multiple root complexes which share a common PCI Express hierarchy.
In SR-IOV, the most common of these, a host VMM configures supported devices to create and allocate virtual "shadows" of their configuration spaces so that virtual machine guests can directly configure and access such "shadow" device resources. With SR-IOV enabled, virtualized network interfaces are directly accessible to the guests,
avoiding involvement of the VMM and resulting in high overall performance; for example, SR-IOV achieves over 95% of the bare metal network bandwidth in NASA's virtualized datacenter and in the Amazon Public Cloud.
See also
*
Comparison of application virtualization software
Application virtualization software refers to both application virtual machines and software responsible for implementing them. Application virtual machines are typically used to allow application bytecode to run portably on many different comput ...
Message Signaled Interrupts
Message Signalled Interrupts (MSI) are an alternative in-band method of signalling an interrupt, using special in-band messages to replace traditional out-of-band assertion of dedicated interrupt lines. While more complex to implement in a devic ...