Skip to main content
Version: 1.0.0

User Guide

Prerequisites​

info

Before reading this guide, refer to SDK Installation for the SDK package download and common configurations.

Mango GPUBoost packages can be easily installed using the package manager in each Linux distribution. Follow the below instructions. Be aware that EPEL packages should be installed on RHEL distros prior to the mango package installation.

~$ sudo apt install ibverbs-providers mango-drivers mango-cli
# install below packages for testing RDMA through perftest
~$ sudo apt install linux-modules-extra-`uname -r` perftest
info

Uninstall any other packages (e.g., MLNX_OFED) that may conflict with ours.

Verifying Card Installation​

If a Mango GPUBoost card is installed correctly, the mango-ctl dev command will show two PCIe devices. mango-ctl is a CLI tool for managing MangoBoost devices.

~$ mango-ctl dev
PCI FPGA Devices
  0000:98:00.0
    Vendor ID: 1f52
    Device ID: 1008
    Revision ID: 0
    Description: Mango GPUBoost - RDMA
    BAR[0]: 0xd8c00000 [size=0x40000]
    BAR[2]: 0xd8800000 [size=0x400000]
    BAR[4]: 0xd8c40000 [size=0x4000]
    NUMA Node: 1
    Kernel module:
  0000:99:00.0
    Vendor ID: 1f52
    Device ID: 1008
    Revision ID: 0
    Description: Mango GPUBoost - RDMA
    BAR[0]: 0xd8400000 [size=0x40000]
    BAR[2]: 0xd8000000 [size=0x400000]
    BAR[4]: 0xd8440000 [size=0x4000]
    NUMA Node: 1
    Kernel module:

Device Driver Setup​

Driver Status Check​

When the package is installed, the Mango GPUBoost driver is automatically loaded and the device becomes available. Additionally, the driver should be automatically loaded on every boot.

~$ lsmod | grep mango
mango_aux_rdma   131072 0
mango_aux_net    212992 1 mango_aux_rdma
mango_core       65536 2 mango_aux_net,mango_aux_rdma

Upon successful loading, you can check the mb_<i> RDMA interface information using the rdma link command.

~$ rdma link
link mb_0/1 state DOWN physical_state DISABLED netdev ens102np0
link mb_1/1 state DOWN physical_state DISABLED netdev ens104np0

Driver Management​

To load the driver manually, use the modprobe mango-aux-rdma command.

~$ sudo modprobe mango-aux-rdma
~$ rdma link
link mb_0/1 state DOWN physical_state DISABLED netdev ens102np0
link mb_1/1 state DOWN physical_state DISABLED netdev ens104np0

To unload the driver, use the modprobe -r mango-aux-rdma command.

~$ sudo modprobe -r mango-aux-rdma
~$ rdma link
(empty)

Network Configuration​

Interface Activation​

Assigning an IP to the ethernet network interface completes the setup for using the RDMA interface.

~$ sudo ifconfig ens102np0 100.0.10.1/24 up
~$ sudo ifconfig ens104np0 100.0.11.1/24 up

Now check if the RDMA interfaces are active.

~$ rdma link link mb_0/1 state ACTIVE physical_state LINK_UP netdev ens102np0 link mb_1/1 state ACTIVE physical_state LINK_UP netdev ens104np0


### Setting the MTU size

Properly configuring the MTU (Maximum Transmission Unit) can improve RDMA performance. Mango GPUBoost -
RNIC currently supports two RDMA MTU sizes: **1024** and **4096** bytes.

Note that the Ethernet MTU includes headers for Ethernet, IP, UDP, and RoCEv2. This results in a difference between the
Ethernet MTU and the RDMA MTU. The following mappings are recommended:
- Ethernet MTU 1500 → RDMA MTU 1024
- Ethernet MTU 4500 → RDMA MTU 4096

To check the current MTU setting:
```bash
~$ ip link show ens102np0
2: ens102np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...

To set a new Ethernet MTU (e.g., 4500 bytes):

~$ sudo ip link set dev ens102np0 mtu 4500
warning

Do NOT set Ethernet MTU directly to RDMA MTU values (e.g., 1024). Always set it to standard Ethernet frame sizes (e.g., 1500 or 4500) to ensure proper operation.

warning

Ensure MTU compatibility across all networking equipment (NICs, switches, routers) to prevent fragmentation or packet drops.

RDMA Functionality Testing​

perftest is a standard benchmarking tool for RDMA NICs. This section provides a simple example of bandwidth testing using Mango GPUBoost.

We assume two Mango GPUBoost cards are installed in separate servers and connected directly or connected through a switch. Both sides must have completed the above RDMA setup step to run the test.

Here is an example of measuring RDMA write bandwidth. Both sides use the same command, but the client side additionally requires the server’s IP address. The -d option specifies the RDMA interface to use.

Server-side command
(server) ~$ ib_write_bw -d mb_0
Client-side command
(client) ~$ ib_write_bw -d mb_0 100.0.10.1
Expected output
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mb_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x000b PSN 0x3d2367 RKey 0x000880 VAddr 0x007feadd1b7000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:0:10:1
 remote address: LID 0000 QPN 0x000a PSN 0xbe580f RKey 0x0007c0 VAddr 0x007faa4544c000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:0:10:2
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             11100.88            11099.90                  0.177598
---------------------------------------------------------------------------------------
info

If it fails, check the network connection between the server and the client, including the firewall configurations. By default, the firewall on RHEL systems blocks the default port for ib_write_bw, 18515.

Perftest supports RDMA Read, Write, Send, and Atomic operations, offering both latency and bandwidth tests. It provides a comprehensive set of options, including message size, number of QPs, queue depth, bidirectional testing, and eventbased CQ processing, among others.

For more details on test options, refer to https://github.com/linux-rdma/perftest.