Comparing Fabric Technologies: InfiniBand Architecture, Gigabit Ethernet, and Fibre Channel

Abstract

The InfiniBand Architecture defines one of the newest networking technologies and naturally raises the question of when/where/how will it replace existing technologies. A careful examination of the leading network technologies demonstrates that there is plenty of room for incumbent networks and the newer technologies. Three of the most popular networking technologies that exceed 1 gigabit per second signaling rates are Gigabit Ethernet, Fibre Channel, and InfiniBand.

Switched Networks and Fabrics

A network is a set of connected nodes that can communicate with each other. To implement a network, definitions for the connections, a communication protocol, and an addressing scheme are required. One way to build cost effective networks is to connect several nodes to the same physical medium to build a network segment

The first popular computer networks such as Ethernet and Token Ring were constructed using a shared media, connected from node to node in a daisy chain fashion. Repeaters and bridges were used to connect multiple segments together. Ethernet and Token Ring use two fundamentally different approaches to share the medium.

Stringing high-bandwidth cables from node to node is a wiring nightmare. An early innovation was to use multi-port repeaters or hubs so that each node could be wired to a central point. The hub internally connected all the nodes to build the network segment. Once all the nodes were connected to the same box, the next innovation was to replace the hub with a switch. A switch allows each pair of communicating nodes to have a full bandwidth segment that is isolated from other nodes as illustrated in Figure 2.

Figure 2. Switched network segments.

The top node thinks it is on a single network segment, but the switch dynamically changes which nodes are connected to that segment.

As early network protocols evolved to switched network segments, newer protocols such as ATM and Fibre Channel were defined that only used a switched infrastructure, i.e. a fabric. Nodes understand they are connected to a fabric and interact with a fabric management entity. From a user perspective, there is little if any difference between switched network segments and fabrics. However, at the lowest levels of the protocols there can be some differences in behavior that show up during network management or failure modes.

Gigabit Ethernet switching technology is based on proprietary switches that dynamically segment switch ports. The management model that is defined is based on a set of counters for each switch port. Fibre Channel defines the behavior of switches in the fabric as well as a model to manage switches and end nodes. The aggregation of multiple switches to build large Fibre Channel fabrics is done in a proprietary manner. InfiniBand defines a comprehensive model for switches and management such that a large subnet can be built using standards-based switches and management applications. Ethernet installations build switched networks, Fibre Channel installations build fabrics, and InfiniBand installations build subnets. The vocabulary is different, but they all describe the same infrastructure.

Technology Comparison

The OSI reference model and its layered architecture serves as a framework to compare Gigabit Ethernet, Fibre Channel, and InfiniBand as shown in Figure 4.

Figure 4. Comparison of protocols based on lower layers of OSI reference model.

The Upper Level Protocols represent the combined application, presentation, and session layers of the OSI model. The shaded boxes represent layers not included in the protocol standards. Notice that the Ethernet standards implement lower levels of the OSI model and are generally associated with the TCP transport layer and IP network layer.

At the physical layer, the protocol architectures are similar. Bit encoding for all three is based on 8B/10B encoding. Both copper and fiber optic cables are defined. InfiniBand defines its medium in such a way it can be implemented as printed circuit board traces. Ethernet can be implemented using shared media through a hub, and Fibre Channel has an arbitrated loop option that allows the media to be shared. Gigabit Ethernet has a variety of physical media defined such as 1000BaseT (4 pairs of unshielded, twisted pair wires), 1000Base-CX (twin axial shielded twisted pairs), or 1000Base-LX (long haul single-mode fiber optics). Fibre Channel is also defined to run on a variety of copper cabling that use different connectors.

The link layer defines how bits are formatted into frames or packets. Gigabit Ethernet packet formats are defined by the IEEE 802.3 standards, which are the same across all the Ethernet speeds (10 Mbps, 100Mbps, and Gigabit). Ethernet link communications implement flow control based on the carrier sense with collision detection or unrestrained full-duplex communication links. Fibre Channel links use flow control based on link-link credit for datagram service or end-end flow control for connection service. InfiniBand uses link-link credit flow control with optional end-end context flow control.

The network layer provides a consistent interface to address all nodes on the network. The Internet Protocol provides a mechanism for global addressing across an Internet. The network layer of Fibre Channel provides a 24-bit addressing space for a fabric. InfiniBand provides a local subnet addressing space of 15 bits, and a global addressing space based on Internet Protocol version 6 addresses. It should be noted that the Internet Protocol used by Ethernet could be mapped to either of the other two transports.

The transport level provides a way for applications to exchange data streams and messages. Ethernet does not provide any support for transport level protocols. InfiniBand relies a suite of sub specifications to define a variety of transports. Mappings for IP, HIPPI, VI, SCSI, ESCON, and other transports are defined. The connection oriented and datagram oriented services of Fibre Channel support these transports as needed. InfiniBand provides a rich set of datagram and connection oriented communication channels, it supports reliable or unreliable delivery, it has memory semantic operations for delivering data to and from a memory buffer on a remote node, and it supports atomic operations useful for interprocess communication protocols. InfiniBand goes beyond the other standards by defining an abstract set of semantics that applications can use to with InfiniBand connections. In fact, the InfiniBand Architecture specification implies the existence of management applications that use the verb interface to configure and manage the subnet.

A summary of the network technologies is shown in Table 1.

Technology	Standards Body	Signaling Speed	First Standard	Maximum frame size	Primary Application
Gigabit Ethernet	IEEE	1.25 Gbps	1999	1.5K	Local Area Network (LAN)
Fibre Channel	ANSI	1 Gbps	1988	2K	Storage Area Network (SAN)
InfiniBand Architecture	InfiniBand Trade Association	2.5 Gbps	2001	4K	I/O Area Network (IAN)

Table 1. Comparison of networking technologies.

The primary application for Gigabit Ethernet is local area networks (LAN). The rapidly evolving field of storage area networks (SAN) applies networking concepts based on Fibre Channel to provide connectivity between server systems and storage devices. The primary application for InfiniBand is provide connectivity for an I/O Area Network (IAN ). An IAN is introduced to identify the architecture used to connect CPU-memory complexes to I/O Controllers, which in turn provide connectivity to a LAN, SAN, or any other I/O media. A bus, e.g. PCI, ISA, or Micro Channel, typically provides this connectivity. A representative architecture of an IAN is implemented with a PCI bus inside a server, as illustrated in Figure 5.

Figure 5. An example of an I/O Area Network (IAN): the PCI bus.

Though not shown in the diagram, the I/O Controller is generally packaged as an adapter and labeled based on the type of I/O medium to which it provides connectivity. An adapter that provides a general network connection is a Network Interface Card (NIC). An adapter that provides a connection to storage media is called a Host Bus Adapter (HBA).

In part, the InfiniBand Architecture was developed to enable a fabric based I/O Area Network.

Application of the Technologies

Using the definition of the networking technologies from the previous section, Figure 6 illustrates how these technologies are used to the problem of server I/O.

A server is a system optimized to process transactions on behalf of clients. Transactions may be a simple as a single database update or as complex as an interactive session between a user and the server application.

To be useful, the server needs to be connected to clients and mass storage. A local area network (LAN) provides connections between peer nodes to the distance of hundreds of meters, enabling many clients to be attached to the server. For a long distance connection, Wide Area Network (WAN) technology is applied to connect a LAN or server. Mass storage is attached to the server using Storage Area Network (SAN) technology. Routers are used to connect the various network technology outside of the server.

The illustration in Figure 6 emphasizes the architecture of the server and how the connection to the client network and mass storage is implemented using an I/O Area Network (IAN). The server on the left shows an IAN that is contained within the server, but a connection to external IAN nodes is still possible. The server cluster on the right illustrates a new server model: one in which the I/O adapters are separated from the CPU-memory complex. The infrastructure to scale the I/O adapters is decoupled from the server. Independent of the server, I/O adapters can be added, greater bandwidth connections can be implemented, and redundant configurations can be built.

In general, Ethernet technologies implement LANs and Fibre Channel technology implements SANs. InfiniBand is the new technology, and it will initially be applied to implement IANs using fabric technology.

Conclusions

There are a variety of switched fabrics and networks to apply to the problem of server I/O. The selection of the "best" technology depends on particular situations, legacy installations, and the desired application. The Ethernet technologies are well suited for client-server connections in a LAN environment, and Fibre Channel technology is the well suited for storage applications in a SAN. InfiniBand is a new technology well suited for interconnection of CPU-memory complexes to I/O in an IAN. A fabric-based IAN enables clustering as well as connections to I/O controllers.

The comparison of these three, layered technologies demonstrated a striking similarity, yet there are subtle differences evident in specific installations. The management model of these three technologies is different, each one supporting the needs of the application. Ethernet network management is loosely defined, and is tolerant of peer nodes joining and leaving segments. There is no need for a node to communicate with a central management entity. This pure peer-to-peer network has few dependencies on the existence of particular nodes, which can be good when a network spans hundreds of offices.

Management of a fabric is better defined out of necessity. The fabric manager must configure each node as it joins the fabric. It implements partitioning and provides a central directory to discover attached nodes. Fibre Channel defines a fabric manager to reside in the fabric, and relies upon proprietary implementations to expand management across many switches. InfiniBand takes this one step further and defines the management entity such that it can reside on any node of the network. It configures and manages the subnet in a manner defined by the InfiniBand Architecture Specification.

Application protocols traditionally favor particular technologies. Most client-server communication is based on TCP/IP or similar protocols. Storage protocols are based on SCSI-3 in the enterprise environment. The protocols used between CPU-memory complexes and adapters have been a mix of bus protocols and custom control messages, and InfiniBand Architecture will standardize these connections. InfiniBand also enables a new level of clustering by allowing CPU-memory complexes to directly communicate using the IAN.

Gigabit Ethernet and its slower speed cousins are the networking technology of choice for client-server networks, Fibre Channel is the fabric technology for storage applications, and InfiniBand Architecture is introduced as a fabric technology for CPU-memory complex to adapter connections.

Jeff Russell, Server I/O Architect, Crossroads Systems, Inc.