InfiniBand (IB) is a computer networking communications standard used in

high-performance computing High-performance computing (HPC) is the use of supercomputers and computer clusters to solve advanced computation problems. Overview HPC integrates systems administration (including network and security knowledge) and parallel programming into ...

that features very high

throughput Network throughput (or just throughput, when in context) refers to the rate of message delivery over a communication channel in a communication network, such as Ethernet or packet radio. The data that these messages contain may be delivered ov ...

and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also used as either a direct or switched interconnect between servers and storage systems, as well as an interconnect between storage systems. It is designed to be

scalable Scalability is the property of a system to handle a growing amount of work. One definition for software systems specifies that this may be done by adding resources to the system. In an economic context, a scalable business model implies that ...

and uses a

switched fabric Switched fabric or switching fabric is a network topology in which network nodes interconnect via one or more network switches (particularly crossbar switches). Because a switched fabric network spreads network traffic across multiple physical l ...

network topology Network topology is the arrangement of the elements (Data link, links, Node (networking), nodes, etc.) of a communication network. Network topology can be used to define or describe the arrangement of various types of telecommunication networks, ...

. Between 2014 and June 2016, it was the most commonly used interconnect in the

TOP500 The TOP500 project ranks and details the 500 most powerful non-distributed computing, distributed computer systems in the world. The project was started in 1993 and publishes an updated list of the supercomputers twice a year. The first of these ...

list of supercomputers.

Mellanox Mellanox Technologies Ltd. () was an Israeli-American multinational supplier of computer networking products based on InfiniBand and Ethernet technology. Mellanox offered adapters, switches, software, cables and silicon for markets including high ...

(acquired by

Nvidia Nvidia Corporation ( ) is an American multinational corporation and technology company headquartered in Santa Clara, California, and incorporated in Delaware. Founded in 1993 by Jensen Huang (president and CEO), Chris Malachowsky, and Curti ...

) manufactures InfiniBand

host bus adapter In computer hardware a host controller, host adapter or host bus adapter (HBA) connects a computer system bus which acts as the host system to other network and storage devices. The terms are primarily used to refer to devices for connecting ...

s and

network switch A network switch (also called switching hub, bridging hub, Ethernet switch, and, by the IEEE, MAC bridge) is networking hardware that connects devices on a computer network by using packet switching to receive and forward data to the destinat ...

es, which are used by large computer system and database vendors in their product lines. As a computer cluster interconnect, IB competes with

Ethernet Ethernet ( ) is a family of wired computer networking technologies commonly used in local area networks (LAN), metropolitan area networks (MAN) and wide area networks (WAN). It was commercially introduced in 1980 and first standardized in 198 ...

Fibre Channel Fibre Channel (FC) is a high-speed data transfer protocol providing in-order, lossless delivery of raw block data. Fibre Channel is primarily used to connect computer data storage to Server (computing), servers in storage area networks (SAN) in ...

, and Intel

Omni-Path Omni-Path Architecture (OPA) is a high-performance communication architecture developed by Intel. It aims for low communication latency, low power consumption and a high throughput. It directly competes with InfiniBand. Intel planned to develop te ...

. The technology is promoted by the InfiniBand Trade Association.

History

InfiniBand originated in 1999 from the merger of two competing designs: Future I/O and Next Generation I/O (NGIO). NGIO was led by

Intel Intel Corporation is an American multinational corporation and technology company headquartered in Santa Clara, California, and Delaware General Corporation Law, incorporated in Delaware. Intel designs, manufactures, and sells computer compo ...

, with a specification released in 1998, and joined by

Sun Microsystems Sun Microsystems, Inc., often known as Sun for short, was an American technology company that existed from 1982 to 2010 which developed and sold computers, computer components, software, and information technology services. Sun contributed sig ...

and

Dell Dell Inc. is an American technology company that develops, sells, repairs, and supports personal computers (PCs), Server (computing), servers, data storage devices, network switches, software, computer peripherals including printers and webcam ...

. Future I/O was backed by

Compaq Compaq Computer Corporation was an American information technology, information technology company founded in 1982 that developed, sold, and supported computers and related products and services. Compaq produced some of the first IBM PC compati ...

IBM International Business Machines Corporation (using the trademark IBM), nicknamed Big Blue, is an American Multinational corporation, multinational technology company headquartered in Armonk, New York, and present in over 175 countries. It is ...

, and

Hewlett-Packard The Hewlett-Packard Company, commonly shortened to Hewlett-Packard ( ) or HP, was an American multinational information technology company. It was founded by Bill Hewlett and David Packard in 1939 in a one-car garage in Palo Alto, California ...

. This led to the formation of the InfiniBand Trade Association (IBTA), which included both sets of hardware vendors as well as software vendors such as

Microsoft Microsoft Corporation is an American multinational corporation and technology company, technology conglomerate headquartered in Redmond, Washington. Founded in 1975, the company became influential in the History of personal computers#The ear ...

. At the time it was thought some of the more powerful computers were approaching the

interconnect bottleneck The interconnect bottleneck comprises limits on integrated circuit (IC) performance due to limits on the speed of connections between components, versus the internal speed of components. In 2006 it was predicted to be a "looming crisis" by 2010. ...

of the

PCI PCI may refer to: Business and economics * Payment card industry, businesses associated with debit, credit, and other payment cards ** Payment Card Industry Data Security Standard, a set of security requirements for credit card processors * Prov ...

bus, in spite of upgrades like

PCI-X PCI-X, short for Peripheral Component Interconnect eXtended, is a computer bus and expansion card standard that enhances the 32-bit Conventional PCI, PCI local bus for higher Bandwidth (computing), bandwidth demanded mostly by Server (computing ...

. Version 1.0 of the InfiniBand Architecture Specification was released in 2000. Initially the IBTA vision for IB was simultaneously a replacement for PCI in I/O, Ethernet in the machine room,

cluster may refer to: Science and technology Astronomy * Cluster (spacecraft), constellation of four European Space Agency spacecraft * Cluster II (spacecraft), a European Space Agency mission to study the magnetosphere * Asteroid cluster, a small ...

interconnect and

. IBTA also envisaged decomposing server hardware on an IB

fabric Textile is an umbrella term that includes various fiber-based materials, including fibers, yarns, filaments, threads, and different types of fabric. At first, the word "textiles" only referred to woven fabrics. However, weaving is no ...

had been founded in 1999 to develop NGIO technology, but by 2001 shipped an InfiniBand product line called InfiniBridge at 10 Gbit/second speeds. Following the burst of the

dot-com bubble The dot-com bubble (or dot-com boom) was a stock market bubble that ballooned during the late-1990s and peaked on Friday, March 10, 2000. This period of market growth coincided with the widespread adoption of the World Wide Web and the Interne ...

there was hesitation in the industry to invest in such a far-reaching technology jump. By 2002, Intel announced that instead of shipping IB integrated circuits ("chips"), it would focus on developing

PCI Express PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe, is a high-speed standard used to connect hardware components inside computers. It is designed to replace older expansion bus standards such as Peripher ...

, and Microsoft discontinued IB development in favor of extending Ethernet.

and

Hitachi () is a Japanese Multinational corporation, multinational Conglomerate (company), conglomerate founded in 1910 and headquartered in Chiyoda, Tokyo. The company is active in various industries, including digital systems, power and renewable ener ...

continued to support IB. In 2003, the System X supercomputer built at

Virginia Tech The Virginia Polytechnic Institute and State University, commonly referred to as Virginia Tech (VT), is a Public university, public Land-grant college, land-grant research university with its main campus in Blacksburg, Virginia, United States ...

used InfiniBand in what was estimated to be the third largest computer in the world at the time. The OpenIB Alliance (later renamed OpenFabrics Alliance) was founded in 2004 to develop an open set of software for the

Linux Linux ( ) is a family of open source Unix-like operating systems based on the Linux kernel, an kernel (operating system), operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically package manager, pac ...

kernel. By February, 2005, the support was accepted into the 2.6.11 Linux kernel. In November 2005 storage devices finally were released using InfiniBand from vendors such as Engenio. Cisco, desiring to keep technology superior to Ethernet off the market, adopted a "buy to kill" strategy. Cisco successfully killed InfiniBand switching companies such as Topspin via acquisition. Of the top 500 supercomputers in 2009,

Gigabit Ethernet In computer networking, Gigabit Ethernet (GbE or 1 GigE) is the term applied to transmitting Ethernet frames at a rate of a gigabit per second. The most popular variant, 1000BASE-T, is defined by the IEEE 802.3ab standard. It came into use in ...

was the internal interconnect technology in 259 installations, compared with 181 using InfiniBand. In 2010, market leaders Mellanox and Voltaire merged, leaving just one other IB vendor,

QLogic QLogic Corporation was an American manufacturer of networking server and storage networking connectivity and application acceleration products, based in Aliso Viejo, California through 2016. QLogic's products include Fibre Channel adapters, c ...

, primarily a

vendor. At the 2011

International Supercomputing Conference The ISC High Performance, formerly known as the International Supercomputing Conference, is a yearly conference on supercomputing which has been held in Europe since 1986. It stands as the oldest supercomputing conference in the world. History ...

, links running at about 56 gigabits per second (known as FDR, see below), were announced and demonstrated by connecting booths in the trade show. In 2012, Intel acquired QLogic's InfiniBand technology, leaving only one independent supplier. By 2014, InfiniBand was the most popular internal connection technology for supercomputers, although within two years,

10 Gigabit Ethernet 10 Gigabit Ethernet (abbreviated 10GE, 10GbE, or 10 GigE) is a group of computer networking technologies for transmitting Ethernet frames at a rate of 10 gigabits per second. It was first defined by the IEEE 802.3ae-2002 standard. Unlik ...

started displacing it. In 2016, it was reported that

Oracle Corporation Oracle Corporation is an American Multinational corporation, multinational computer technology company headquartered in Austin, Texas. Co-founded in 1977 in Santa Clara, California, by Larry Ellison, who remains executive chairman, Oracle was ...

(an investor in Mellanox) might engineer its own InfiniBand hardware. In 2019

acquired Mellanox, the last independent supplier of InfiniBand products.

Specification

Specifications are published by the InfiniBand trade association.

Performance

Original names for speeds were single-data rate (SDR), double-data rate (DDR) and quad-data rate (QDR) as given below. Subsequently, other three-letter acronyms were added for even higher data rates. ; Notes Each link is duplex. Links can be aggregated: most systems use a 4 link/lane connector (QSFP). HDR often makes use of 2x links (aka HDR100, 100 Gb link using 2 lanes of HDR, while still using a QSFP connector). 8x is called for with NDR switch ports using OSFP (Octal Small Form Factor Pluggable) connectors InfiniBand provides

remote direct memory access In computing, remote direct memory access (RDMA) is a direct memory access from the memory of one computer into that of another without involving either one's operating system. This permits high-throughput, low- latency networking, which is especia ...

(RDMA) capabilities for low CPU overhead.

Topology

InfiniBand uses a

topology, as opposed to early shared medium

. All transmissions begin or end at a channel adapter. Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or

quality of service Quality of service (QoS) is the description or measurement of the overall performance of a service, such as a telephony or computer network, or a cloud computing service, particularly the performance seen by the users of the network. To quantitat ...

(QoS).

Messages

InfiniBand transmits data in packets of up to 4 KB that are taken together to form a message. A message can be: * a remote direct memory access read or write * a

channel Channel, channels, channeling, etc., may refer to: Geography * Channel (geography), a landform consisting of the outline (banks) of the path of a narrow body of water. Australia * Channel Country, region of outback Australia in Queensland and pa ...

send or receive * a transaction-based operation (that can be reversed) * a

multicast In computer networking, multicast is a type of group communication where data transmission is addressed to a group of destination computers simultaneously. Multicast can be one-to-many or many-to-many distribution. Multicast differs from ph ...

transmission * an

atomic operation In concurrent programming, an operation (or set of operations) is linearizable if it consists of an ordered list of Execution (computing), invocation and response Event (computing), events, that may be extended by adding response events such tha ...

Physical interconnection

In addition to a board form factor connection, it can use both active and passive copper (up to 10 meters) and

optical fiber cable A fiber-optic cable, also known as an optical-fiber cable, is an assembly similar to an electrical cable but containing one or more optical fibers that are used to carry light. The optical fiber elements are typically individually coated with p ...

(up to 10 km).

QSFP file:SFP board 2.jpg, Small Form-factor Pluggable connected to a pair of fiber-optic cables Small Form-factor Pluggable (SFP) is a compact, hot-pluggable network interface module format used for both telecommunication and data communications a ...

connectors are used. The InfiniBand Association also specified the CXP connector system for speeds up to 120 Gbit/s over copper, active optical cables, and optical transceivers using parallel multi-mode fiber cables with 24-fiber MPO connectors.

Software interfaces

Mellanox operating system support is available for

Solaris Solaris is the Latin word for sun. It may refer to: Arts and entertainment Literature, television and film * ''Solaris'' (novel), a 1961 science fiction novel by Stanisław Lem ** ''Solaris'' (1968 film), directed by Boris Nirenburg ** ''Sol ...

FreeBSD FreeBSD is a free-software Unix-like operating system descended from the Berkeley Software Distribution (BSD). The first version was released in 1993 developed from 386BSD, one of the first fully functional and free Unix clones on affordable ...

Red Hat Enterprise Linux Red Hat Enterprise Linux (RHEL) is a commercial Linux distribution developed by Red Hat. Red Hat Enterprise Linux is released in server versions for x86-64, Power ISA, ARM64, and IBM Z and a desktop version for x86-64. Fedora Linux and ...

SUSE Linux Enterprise Server SUSE Linux Enterprise (SLE) is a Linux-based operating system developed by SUSE. It is available in two editions, suffixed with Server (SLES) for servers and mainframes, and Desktop (SLED) for workstations and desktop computers. Its major ve ...

(SLES),

Windows Windows is a Product lining, product line of Proprietary software, proprietary graphical user interface, graphical operating systems developed and marketed by Microsoft. It is grouped into families and subfamilies that cater to particular sec ...

HP-UX HP-UX (from "Hewlett Packard Unix") is a proprietary software, proprietary implementation of the Unix operating system developed by Hewlett Packard Enterprise; current versions support HPE Integrity Servers, based on Intel's Itanium architect ...

VMware ESX VMware ESXi (formerly ESX) is an enterprise-class, type-1 hypervisor developed by VMware, a subsidiary of Broadcom, for deploying and serving virtual computers. As a type-1 hypervisor, ESXi is not a software application that is installed on ...

, and

AIX Aix or AIX may refer to: Computing * AIX, a line of IBM computer operating systems *Alternate index, for an IBM Virtual Storage Access Method key-sequenced data set * Athens Internet Exchange, a European Internet exchange point Places Belg ...

. InfiniBand has no specific standard

application programming interface An application programming interface (API) is a connection between computers or between computer programs. It is a type of software Interface (computing), interface, offering a service to other pieces of software. A document or standard that des ...

(API). The standard only lists a set of verbs such as ibv_open_device or ibv_post_send, which are abstract representations of functions or methods that must exist. The syntax of these functions is left to the vendors. Sometimes for reference this is called the ''verbs'' API. The

de facto standard A ''de facto'' standard is a custom or convention that is commonly used even though its use is not required. is a Latin phrase (literally " of fact"), here meaning "in practice but not necessarily ordained by law" or "in practice or actuality, ...

software is developed by OpenFabrics Alliance and called the Open Fabrics Enterprise Distribution (OFED). It is released under two licenses GPL2 or

BSD license BSD licenses are a family of permissive free software licenses, imposing minimal restrictions on the use and distribution of covered software. This is in contrast to copyleft licenses, which have share-alike requirements. The original BSD lic ...

for Linux and FreeBSD, and as Mellanox OFED for Windows (product names: WinOF / WinOF-2; attributed as host controller driver for matching specific ConnectX 3 to 5 devices) under a choice of BSD license for Windows. It has been adopted by most of the InfiniBand vendors, for

, and

Microsoft Windows Windows is a Product lining, product line of Proprietary software, proprietary graphical user interface, graphical operating systems developed and marketed by Microsoft. It is grouped into families and subfamilies that cater to particular sec ...

refers to a software library called libibverbs, for its

operating system, as well as "AIX InfiniBand verbs". The Linux kernel support was integrated in 2005 into the kernel version 2.6.11.

Ethernet over InfiniBand

Ethernet over InfiniBand, abbreviated to EoIB, is an Ethernet implementation over the InfiniBand protocol and connector technology. EoIB enables multiple Ethernet bandwidths varying on the InfiniBand (IB) version. Ethernet's implementation of the

Internet Protocol Suite The Internet protocol suite, commonly known as TCP/IP, is a framework for organizing the communication protocols used in the Internet and similar computer networks according to functional criteria. The foundational protocols in the suite are ...

, usually referred to as TCP/IP, is different in some details compared to the direct InfiniBand protocol in IP over IB (IPoIB).

History

Specification

Performance

Topology

Messages

Physical interconnection

Software interfaces

Ethernet over InfiniBand

See also