Migration to RDMA-Core

Overview

As a leading RDMA-based interconnect provider, NVIDIA strives to bring all its software drivers and features to the Linux Upstream in order to guarantee best out-of-the box experience.

The NVIDIA Linux driver development methodology is “Upstream first”. Therefore, NVIDIA contributes all its developed features to the RDMA subsystem before porting them to its own Linux driver, that is, MLNX_OFED.

Each generation of the ConnectX® network adapters includes dedicated hardware that unleashes cutting-edge RDMA features, delivering compute and data intensive applications with highest performance and scalability. The RDMA subsystem verbs library is a generic API library serving all vendors and all RDMA applications.

In order to enable the usage of new cutting-edge technologies as soon as they became available, NVIDIA came up with an interim solution to work around the long wait periods before these features become supported and incorporated in Linux Upstream. As such, and until a faster acceptance process for the verbs API was implemented for the RDMA subsystem, NVIDIA has developed and maintained a private API with the prefix “ibv_exp” (known as “experimental verbs”). This private API enabled NVIDIA to expose early on the latest hardware offloads to its customers’ applications via the NVIDIA Linux driver (MLNX_OFED).

Nowadays, the acceptance process has become faster. Therefore, the private verbs API has become redundant. NVIDIA has set new APIs in Linux Upstream, which can be used to achieve most of the legacy experimental verbs’ functionalities and more. Going forward, future RDMA-related features will only be available through the Upstream RDMA subsystem.

The userspace RDMA subsystem (known as RDMA-Core) library introduces the following new libmlx5 API channels:

  • mlx5dv—a channel for configuring hardware objects with NVIDIA specific attributes.

  • mlx5dv_devx—a channel for exposing raw NVIDIA hardware objects, enabling the use of hardware features with minimal Kernel changes (some of the mlx5dv APIs use mlx5dv_devx at userspace level).

Application owners who wish to use the latest RDMA-based in-network computing acceleration engines and hardware offloads are kindly asked to use the RDMA-Core APIs and avoid using the experimental verbs APIs.

Application owners currently using the experimental verbs (ibv_exp) APIs are highly encouraged to begin porting their applications to use the RDMA-core APIs instead. The porting process is for the most part straight-forward.

Important Notes:

  • NVIDIA plans to deprecate the experimental verbs API in the near future, following an advance twelve-month notice.

  • As of MLNX_OFED v4.7:

    • No new features will be added to the experimental verbs API.

    • mlx5dv and mlx5dv_devx APIs will become default APIs. However, the experimental verbs API will not be exposed.

    • Application owners can still choose to install the experimental verbs API and use the verbs (using a special installation flag).

For further assistance, please contact Support.

© Copyright 2023, NVIDIA. Last updated on Aug 31, 2023.