NVM Express (NVMe) or Non-Volatile Memory Host Controller Interface Specification (NVMHCIS) is an open, logical-device interface

specification A specification often refers to a set of documented requirements to be satisfied by a material, design, product, or service. A specification is often a type of technical standard. There are different types of technical or engineering specificati ...

for accessing a computer's

non-volatile storage Non-volatile memory (NVM) or non-volatile storage is a type of computer memory that can retain stored information even after power is removed. In contrast, volatile memory needs constant power in order to retain data. Non-volatile memory typi ...

media usually attached via PCI Express (PCIe) bus. The initialism ''NVM'' stands for ''

non-volatile memory Non-volatile memory (NVM) or non-volatile storage is a type of computer memory that can retain stored information even after power is removed. In contrast, volatile memory needs constant power in order to retain data. Non-volatile memory typi ...

'', which is often NAND flash memory that comes in several physical form factors, including

solid-state drive A solid-state drive (SSD) is a solid-state storage device that uses integrated circuit assemblies to store data persistently, typically using flash memory, and functioning as secondary storage in the hierarchy of computer storage. It is a ...

s (SSDs), PCIe add-in cards, and

M.2 M.2, pronounced ''m dot two'' and formerly known as the Next Generation Form Factor (NGFF), is a specification for internally mounted computer expansion cards and associated connectors. M.2 replaces the mSATA standard, which uses the PCI Ex ...

cards, the successor to mSATA cards. NVM Express, as a logical-device interface, has been designed to capitalize on the low latency and internal parallelism of solid-state storage devices. Architecturally, the logic for NVMe is physically stored within and executed by the NVMe controller chip that is physically co-located with the storage media, usually an SSD. Version changes for NVMe, e.g., 1.3 to 1.4, are incorporated within the storage media, and do not affect PCIe-compatible components such as motherboards and CPUs. By its design, NVM Express allows host hardware and software to fully exploit the levels of parallelism possible in modern SSDs. As a result, NVM Express reduces I/O overhead and brings various performance improvements relative to previous logical-device interfaces, including multiple long command queues, and reduced latency. The previous interface protocols like

AHCI The Advanced Host Controller Interface (AHCI) is a technical standard defined by Intel that specifies the register-level interface of Serial ATA (SATA) host controllers in a non-implementation-specific manner in its motherboard chipsets. The ...

were developed for use with far slower

hard disk drive A hard disk drive (HDD), hard disk, hard drive, or fixed disk is an electro-mechanical data storage device that stores and retrieves digital data using magnetic storage with one or more rigid rapidly rotating platters coated with magne ...

s (HDD) where a very lengthy delay (relative to CPU operations) exists between a request and data transfer, where data speeds are much slower than RAM speeds, and where disk rotation and seek time give rise to further optimization requirements. NVM Express devices are chiefly available in the form of standard-sized PCI Express

expansion card In computing, an expansion card (also called an expansion board, adapter card, peripheral card or accessory card) is a printed circuit board that can be inserted into an electrical connector, or expansion slot (also referred to as a bus sl ...

s and as 2.5-inch form-factor devices that provide a four-lane PCI Express interface through the U.2 connector (formerly known as SFF-8639). Storage devices using

SATA Express SATA Express (sometimes unofficially shortened to SATAe) is a computer bus interface that supports both Serial ATA (SATA) and PCI Express (PCIe) storage devices, initially standardized in the SATA 3.2 specification. The SATA Express con ...

and the M.2 specification which support NVM Express as the logical-device interface are a popular use-case for NVMe and have become the dominant form of solid-state storage for servers, desktops, and laptops alike.

Specifications

Specifications for NVMe released to date include: * 1.0e (January 2013) * 1.1b (July 2014) * 1.2 (November 2014) ** 1.2a (October 2015) ** 1.2b (June 2016) ** 1.2.1 (June 2016) * 1.3 (May 2017) ** 1.3a (October 2017) ** 1.3b (May 2018) ** 1.3c (May 2018) ** 1.3d (March 2019) * 1.4 (June 2019) ** 1.4a (March 2020) **1.4b (September 2020) * 2.0 (May 2021) ** 2.0a (July 2021) ** 2.0b (January 2022) ** 2.0c (October 2022)

Background

Historically, most SSDs used buses such as

SATA SATA (Serial AT Attachment) is a computer bus interface that connects host bus adapters to mass storage devices such as hard disk drives, optical drives, and solid-state drives. Serial ATA succeeded the earlier Parallel ATA (PATA) standard t ...

, SAS or Fibre Channel for interfacing with the rest of a computer system. Since SSDs became available in mass markets, SATA has become the most typical way for connecting SSDs in

personal computer A personal computer (PC) is a multi-purpose microcomputer whose size, capabilities, and price make it feasible for individual use. Personal computers are intended to be operated directly by an end user, rather than by a computer expert or tec ...

s; however, SATA was designed primarily for interfacing with mechanical

s (HDDs), and it became increasingly inadequate for SSDs, which improved in speed over time. For example, within about five years of mass market mainstream adoption (2005–2010) many SSDs were already held back by the comparatively slow data rates available for hard drives—unlike hard disk drives, some SSDs are limited by the maximum

throughput Network throughput (or just throughput, when in context) refers to the rate of message delivery over a communication channel, such as Ethernet or packet radio, in a communication network. The data that these messages contain may be delivered ove ...

of SATA. High-end SSDs had been made using the PCI Express bus before NVMe, but using non-standard specification interfaces. By standardizing the interface of SSDs,

operating system An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs. Time-sharing operating systems schedule tasks for efficient use of the system and may also i ...

s only need one common device driver to work with all SSDs adhering to the specification. It also means that each SSD manufacturer does not have to design specific interface drivers. This is similar to how USB mass storage devices are built to follow the USB mass-storage device class specification and work with all computers, with no per-device drivers needed. NVM Express devices are also used as the building block of the burst buffer storage in many leading supercomputers, such as Fugaku Supercomputer, Summit Supercomputer and Sierra Supercomputer, etc.

History

The first details of a new standard for accessing non-volatile memory emerged at the Intel Developer Forum 2007, when NVMHCI was shown as the host-side protocol of a proposed architectural design that had Open NAND Flash Interface Working Group (ONFI) on the memory (flash) chips side. A NVMHCI working group led by Intel was formed that year. The NVMHCI 1.0 specification was completed in April 2008 and released on Intel's web site. Technical work on NVMe began in the second half of 2009.http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130813_A12_Onufryk.pdf The NVMe specifications were developed by the NVM Express Workgroup, which consists of more than 90 companies; Amber Huffman of

Intel Intel Corporation is an American multinational corporation and technology company headquartered in Santa Clara, California. It is the world's largest semiconductor chip manufacturer by revenue, and is one of the developers of the x86 seri ...

was the working group's chair. Version 1.0 of the specification was released on 1 March 2011, while version 1.1 of the specification was released on 11 October 2012. Major features added in version 1.1 are multi-path I/O (with namespace sharing) and arbitrary-length scatter-gather I/O. It is expected that future revisions will significantly enhance namespace management. Because of its feature focus, NVMe 1.1 was initially called "Enterprise NVMHCI". An update for the base NVMe specification, called version 1.0e, was released in January 2013. In June 2011, a Promoter Group led by seven companies was formed. The first commercially available NVMe chipsets were released by

Integrated Device Technology Integrated Device Technology, Inc., is an American corporation headquartered in San Jose, California, that designs, manufactures, and markets low-power, high-performance mixed-signal semiconductor solutions for the advanced communications, com ...

(89HF16P04AG3 and 89HF32P08AG3) in August 2012. The first NVMe drive,

Samsung The Samsung Group (or simply Samsung) ( ko, 삼성 ) is a South Korean multinational manufacturing conglomerate headquartered in Samsung Town, Seoul, South Korea. It comprises numerous affiliated businesses, most of them united under the ...

's XS1715 enterprise drive, was announced in July 2013; according to Samsung, this drive supported 3 GB/s read speeds, six times faster than their previous enterprise offerings. The LSI SandForce SF3700 controller family, released in November 2013, also supports NVMe. A Kingston HyperX " prosumer" product using this controller was showcased at the Consumer Electronics Show 2014 and promised similar performance. In June 2014, Intel announced their first NVM Express products, the Intel SSD data center family that interfaces with the host through PCI Express bus, which includes the DC P3700 series, the DC P3600 series, and the DC P3500 series. , NVMe drives are commercially available. In March 2014, the group incorporated to become NVM Express, Inc., which consists of more than 65 companies from across the industry. NVM Express specifications are owned and maintained by NVM Express, Inc., which also promotes industry awareness of NVM Express as an industry-wide standard. NVM Express, Inc. is directed by a thirteen-member board of directors selected from the Promoter Group, which includes Cisco, Dell, EMC, HGST, Intel, Micron, Microsoft, NetApp, Oracle, PMC, Samsung, SanDisk and Seagate. In September 2016, the CompactFlash Association announced that it would be releasing a new memory card specification,

CFexpress CFexpress is a standard for removable media cards proposed by the CompactFlash Association (CFA). The standard uses PCIe 3.0 interface with 1 to 4 lanes where 1 GB/s data can be provided per lane. NVM Express is also supported to provide low overh ...

, which uses NVMe. NVMe Host Memory Buffer (HMB) added in version 1.2 of the NVMe specification.The Toshiba RC100 SSD Review: Tiny Drive In A Big Market
/ref> HMB allows SSDs to utilize the host's DRAM, which can improve the I/O performance for DRAM-less SSDs.

Form factors

There are many form factors of NVMe solid-state drive, such as AIC, U.2, U.3, M.2 etc.

AIC (add-in card)

Almost all early NVMe solid-state drives are HHHL (half height, half length) or FHHL (full height, half length) AIC, with a

PCIe PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe or PCI-e, is a high-speed serial computer expansion bus standard, designed to replace the older PCI, PCI-X and AGP bus standards. It is the common ...

2.0 or 3.0 interface. A HHHL NVMe solid-state drive card is easy to insert into a PCIe slot of a server.

U.2 (SFF-8639)

U.2, formerly known as SFF-8639, is a computer interface for connecting solid-state drives to a computer. It uses up to four PCI Express lanes. Available servers can combine up to 48 U.2 NVMe solid-state drives.

U.3 (SFF-8639 or SFF-TA-1001)

U.3 is built on the U.2 spec and uses the same SFF-8639 connector. It is a 'tri-mode' standard, combining SAS, SATA and NVMe support into a single controller. U.3 can also support hot-swap between the different drives where firmware support is available. U.3 drives are still backward compatible with U.2, but U.2 drives are not compatible with U.3 hosts.

M.2

, formerly known as the Next Generation Form Factor (NGFF), uses a M.2 NVMe solid-state drive computer bus. Interfaces provided through the M.2 connector are PCI Express 3.0 or PCI Express 4.0 (up to four lanes).

EDSFF

NVMe-oF

NVM Express over Fabrics (NVMe-oF) is the concept of using a transport protocol over a network to connect remote NVMe devices, contrary to regular NVMe where physical NVMe devices are connected to a PCIe bus either directly or over a PCIe switch to a PCIe bus. In August 2017, a standard for using NVMe over Fibre Channel (FC) was submitted by the standards organization International Committee for Information Technology Standards (ICITS), and this combination is often referred to as FC-NVMe or sometimes NVMe/FC. As of May 2021, supported NVMe transport protocols are: * FC, FC-NVMe * TCP, NVMe/TCP *

Ethernet Ethernet () is a family of wired computer networking technologies commonly used in local area networks (LAN), metropolitan area networks (MAN) and wide area networks (WAN). It was commercially introduced in 1980 and first standardized in 1 ...

, RoCE v1/v2 ( RDMA over converged Ethernet) *

InfiniBand InfiniBand (IB) is a computer networking communications standard used in high-performance computing that features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also use ...

, NVMe over InfiniBand or NVMe/IB The standard for NVMe over Fabrics was published by NVM Express, Inc. in 2016. The following software implements the NVMe-oF protocol: * Linux NVMe-oF initiator and target. RoCE transport was supported initially, and with Linux kernel 5.x, native support for TCP was added. * Storage Performance Development Kit (SPDK) NVMe-oF initiator and target drivers. Both RoCE and TCP transports are supported. * Starwind NVMe-oF initiator and target for Microsoft Windows, supporting both RoCE and TCP transports.

Comparison with AHCI

The Advanced Host Controller Interface (AHCI) has the benefit of wide software compatibility, but has the downside of not delivering optimal performance when used with

SSDs A solid-state drive (SSD) is a solid-state storage device that uses integrated circuit assemblies to store data persistently, typically using flash memory, and functioning as secondary storage in the hierarchy of computer storage. It is a ...

connected via the PCI Express bus. As a logical-device interface, AHCI was developed when the purpose of a

host bus adapter In computer hardware, a host controller, host adapter, or host bus adapter (HBA), connects a computer system bus, which acts as the host system, to other network and storage devices. The terms are primarily used to refer to devices for conne ...

(HBA) in a system was to connect the CPU/memory subsystem with a much slower storage subsystem based on rotating

magnetic media Magnetic storage or magnetic recording is the storage of data on a magnetized medium. Magnetic storage uses different patterns of magnetisation in a magnetizable material to store data and is a form of non-volatile memory. The information is ac ...

. As a result, AHCI introduces certain inefficiencies when used with SSD devices, which behave much more like

RAM Ram, ram, or RAM may refer to: Animals * A male sheep * Ram cichlid, a freshwater tropical fish People * Ram (given name) * Ram (surname) * Ram (director) (Ramsubramaniam), an Indian Tamil film director * RAM (musician) (born 1974), Dutch * ...

than like spinning media. The NVMe device interface has been designed from the ground up, capitalizing on the lower latency and parallelism of PCI Express SSDs, and complementing the parallelism of contemporary CPUs, platforms and applications. At a high level, the basic advantages of NVMe over AHCI relate to its ability to exploit parallelism in host hardware and software, manifested by the differences in command queue depths, efficiency of

interrupt In digital computers, an interrupt (sometimes referred to as a trap) is a request for the processor to ''interrupt'' currently executing code (when permitted), so that the event can be processed in a timely manner. If the request is accepted, ...

processing, the number of uncacheable register accesses, etc., resulting in various performance improvements. The table below summarizes high-level differences between the NVMe and AHCI logical-device interfaces.

Operating system support

;

ChromeOS ChromeOS, sometimes stylized as chromeOS and formerly styled as Chrome OS, is a Linux-based operating system designed by Google. It is derived from the open-source ChromiumOS and uses the Google Chrome web browser as its principal user interfa ...

: On February 24, 2015, support for booting from NVM Express devices was added to

. ;

DragonFly BSD DragonFly BSD is a free and open-source Unix-like operating system forked from FreeBSD 4.8. Matthew Dillon, an Amiga developer in the late 1980s and early 1990s and FreeBSD developer between 1994 and 2003, began working on DragonFly BSD in ...

: The first release of

with NVMe support is version 4.6. ; FreeBSD : Intel sponsored a NVM Express driver for FreeBSD's head and stable/9 branches. The nvd(4) and nvme(4) drivers are included in the GENERIC kernel configuration by default since FreeBSD version 10.2 in 2015. ; Genode : Support for consumer-grade NVMe was added to the Genode framework as part of the 18.05 release. ;

Haiku is a type of short form poetry originally from Japan. Traditional Japanese haiku consist of three phrases that contain a ''kireji'', or "cutting word", 17 '' on'' (phonetic units similar to syllables) in a 5, 7, 5 pattern, and a ''kigo'', or s ...

gained support for NVMe on April 18, 2019. ;

illumos Illumos (stylized as illumos) is a partly free and open-source Unix operating system. It is based on OpenSolaris, which was based on System V Release 4 (SVR4) and the Berkeley Software Distribution (BSD). Illumos comprises a kernel, device d ...

received support for NVMe on October 15, 2014. ;

iOS iOS (formerly iPhone OS) is a mobile operating system created and developed by Apple Inc. exclusively for its hardware. It is the operating system that powers many of the company's mobile devices, including the iPhone; the term also include ...

: With the release of the iPhone 6S and 6S Plus,

Apple An apple is an edible fruit produced by an apple tree (''Malus domestica''). Apple trees are cultivated worldwide and are the most widely grown species in the genus ''Malus''. The tree originated in Central Asia, where its wild ancestor, ' ...

introduced the first mobile deployment of NVMe over

in smartphones. Apple followed these releases with the release of the first-generation

iPad Pro The iPad Pro is a premium model of Apple's iPad tablet computer. It runs iPadOS, a tablet-optimized version of the iOS operating system. The original iPad Pro was introduced in September 2015, and ran iOS 9. The second-generation iPad P ...

and first-generation iPhone SE that also use NVMe over PCIe. ;

Linux Linux ( or ) is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged as a Linux distribution, w ...

: Intel published an NVM Express driver for

on 3 March 2011, which was merged into the Linux kernel mainline on 18 January 2012 and released as part of version 3.3 of the Linux kernel on 19 March 2012. Linux supports NVMe Host Memory Buffer from version 4.13.1 with default maximum size 128 MB. ;

macOS macOS (; previously OS X and originally Mac OS X) is a Unix operating system developed and marketed by Apple Inc. since 2001. It is the primary operating system for Apple's Mac computers. Within the market of desktop and lapt ...

: Apple introduced software support for NVM Express in Yosemite 10.10.3. The NVMe hardware interface was introduced in the 2016 MacBook and MacBook Pro. ; NetBSD : NetBSD added support for NVMe in NetBSD 8.0. The implementation is derived from OpenBSD 6.0. ; OpenBSD : Development work required to support NVMe in OpenBSD has been started in April 2014 by a senior developer formerly responsible for USB 2.0 and

support. Support for NVMe has been enabled in the OpenBSD 6.0 release. ;

OS/2 OS/2 (Operating System/2) is a series of computer operating systems, initially created by Microsoft and IBM under the leadership of IBM software designer Ed Iacobucci. As a result of a feud between the two companies over how to position OS/2 r ...

: Arca Noae provides an NVMe driver for ArcaOS, as of April, 2021. The driver requires advanced interrupts as provided by the ACPI PSD running in advanced interrupt mode (mode 2), thus requiring the SMP kernel, as well. ; Solaris : Solaris received support for NVMe in Oracle Solaris 11.2. ;

VMware VMware, Inc. is an American cloud computing and virtualization technology company with headquarters in Palo Alto, California. VMware was the first commercially successful company to virtualize the x86 architecture. VMware's desktop software ru ...

: Intel has provided an NVMe driver for

, which is included in vSphere 6.0 and later builds, supporting various NVMe devices. As of vSphere 6 update 1, VMware's VSAN software-defined storage subsystem also supports NVMe devices. ;

Windows Windows is a group of several proprietary graphical operating system families developed and marketed by Microsoft. Each family caters to a certain sector of the computing industry. For example, Windows NT for consumers, Windows Server for ser ...

: Microsoft added native support for NVMe to Windows 8.1 and

Windows Server 2012 R2 Windows Server 2012 R2, codenamed "Windows Server 8.1" or "Windows Server Blue", is the seventh version of the Windows Server operating system by Microsoft, as part of the Windows NT family of operating systems. It was unveiled on June 3, 2013 a ...

. Native drivers for

Windows 7 Windows 7 is a major release of the Windows NT operating system developed by Microsoft. It was released to manufacturing on July 22, 2009, and became generally available on October 22, 2009. It is the successor to Windows Vista, released nearly ...

and

Windows Server 2008 R2 Windows Server 2008 R2 is the fifth version of the Windows Server operating system produced by Microsoft and released as part of the Windows NT family of operating systems. It was released to manufacturing on July 22, 2009, and became generall ...

have been added in updates. Many vendors have released their own Windows drivers for their devices as well. There are also manually customized installer files available to install a specific vendor's driver to any NVMe card, such as using a Samsung NVMe driver with a non-Samsung NVMe device, which may be needed for additional features, performance, and stability. Support for NVMe HMB was added in Windows 10 Anniversary Update (Version 1607) in 2016. : The

OpenFabrics Alliance The OpenFabrics Alliance is a non-profit organization that promotes remote direct memory access (RDMA) switched fabric technologies for server and storage connectivity. These high-speed data-transport technologies are used in high-performance ...

maintains an open-source NVMe Windows Driver for Windows 7/8/8.1 and Windows Server 2008R2/2012/2012R2, developed from the baseline code submitted by several promoter companies in the NVMe workgroup, specifically IDT, Intel, and LSI. The current release is 1.5 from December 2016.

Software support

; QEMU : NVMe is supported by

QEMU QEMU is a free and open-source emulator (Quick EMUlator). It emulates the machine's central processing unit, processor through dynamic binary translation and provides a set of different hardware and device models for the machine, enabling it t ...

since version 1.6 released on August 15, 2013. NVMe devices presented to QEMU guests can be either real or emulated. ; UEFI : An open source NVMe driver for UEFI is available on SourceForge.

Management tools

nvmecontrol

The nvmecontrol tool is used to control an NVMe disk from the command line on FreeBSD. It was added in FreeBSD 9.2.

nvme-cli

NVM-Express user space tooling for Linux.

References

External links

*
NVMe info

CompactFlash Association

LFCS: Preparing Linux for nonvolatile memory devices

LWN.net LWN.net is a computing webzine with an emphasis on free software and software for Linux and other Unix-like operating systems. It consists of a weekly issue, separate stories which are published most days, and threaded discussion attached to ...

, April 19, 2013, by Jonathan Corbet
Multipathing PCI Express Storage
Linux Foundation, March 12, 2015, by Keith Busch
NVMe, NVMe-oF and RDMA for network engineers
August 2020, by Jerome Tissieres {{Computer bus Computer storage buses Solid-state computer storage Computer hardware standards