Software Setup

This page covers package installation, driver management, IP configuration, device monitoring, and the functional RDMA test for Mango BoostX™ RoCE AI.

info

Before proceeding, ensure the host BIOS / kernel cmdline / cable / switch prerequisites in Hardware Setup are complete.

Prerequisites

Officially Supported Linux Distributions

Mango BoostX™ RoCE AI is officially tested on the following Linux distributions on the x86_64 architecture:

Ubuntu 24.04 (Noble): Kernel version 6.8.x
Ubuntu 22.04 (Jammy): Kernel version 5.15.x
Rocky Linux 9.6: Kernel version 5.14.x

Package Installation

First, install the delivered local-repository package and refresh apt:

~$ sudo dpkg -i mango-repo-rdma_26XXXX_amd64.deb
~$ sudo apt update
Get:1 file:/var/mango-rdma-noble-local  InRelease [1512 B]
Get:1 file:/var/mango-rdma-noble-local  InRelease [1512 B]
...

Then install the Mango BoostX™ RoCE AI packages:

~$ sudo apt install ibverbs-providers mango-drivers mango-drivers-dkms mango-cli libmango0 libmango-dev
# install below packages for testing RDMA through perftest
~$ sudo apt install linux-modules-extra-`uname -r` perftest

info

Uninstall any other packages (e.g., MLNX_OFED) that may conflict with Mango packages.

Verifying Card Installation

If a Mango BoostX™ RoCE AI card is installed correctly, the mango-ctl dev command shows four PCIe devices: two 200GbE RDMA-enabled Ethernet interfaces and two management PFs. mango-ctl is the CLI tool for managing MangoBoost devices.

~$ mango-ctl dev
PCI FPGA Devices
  0000:02:00.2
    Vendor ID: 1f52
    Device ID: 3000
    Revision ID: 0
    Description: Mango Management PF
    BAR[0]: 0x4030800000 [size=0x80000]
    BAR[2]: 0x4000000000 [size=0x10000000]
    NUMA Node: 1
    PCIe MSI-X: 8
    PCIe Link: Speed 32GT/s (ok), Width x8 (ok)
    Kernel module: mango_mgmt
  0000:02:00.0
    Vendor ID: 1f52
    Device ID: 1008
    Revision ID: 0
    Description: Mango BoostX - RoCE AI
    BAR[0]: 0x4030880000 [size=0x80000]
    BAR[2]: 0x4020000000 [size=0x10000000]
    BAR[4]: 0x4030000000 [size=0x800000]
    NUMA Node: 1
    PCIe MSI-X: 16
    PCIe Link: Speed 32GT/s (ok), Width x8 (ok)
    Character device: /dev/mango1
    Enabled projects: net pcc rdma
    Kernel module: mango_core
  0000:01:00.0
    Vendor ID: 1f52
    Device ID: 1008
    Revision ID: 1
    Description: Mango BoostX - RoCE AI
    BAR[0]: 0x4050880000 [size=0x80000]
    BAR[2]: 0x4040000000 [size=0x10000000]
    BAR[4]: 0x4050000000 [size=0x800000]
    NUMA Node: 1
    PCIe MSI-X: 16
    PCIe Link: Speed 32GT/s (ok), Width x8 (ok)
    Character device: /dev/mango0
    Enabled projects: net pcc rdma
    Kernel module: mango_core
  0000:02:00.1
    Vendor ID: 1f52
    Device ID: 50b4
    Revision ID: 0
    Description: Mango Management PF (SMI)
    BAR[0]: 0x4010000000 [size=0x10000000]
    NUMA Node: 1
    PCIe MSI-X: 0
    PCIe Link: Speed 32GT/s (ok), Width x8 (ok)
    Kernel module: mango_smi

Driver Management

Checking Driver Load Status

When the package is installed, the Mango BoostX™ RoCE AI driver is automatically loaded and the device becomes available. The driver is also automatically loaded on every boot.

~$ lsmod | grep mango
mango_aux_rdma        270336  0
mango_aux_net         495616  1 mango_aux_rdma
mango_aux_pcc          28672  0
mango_core            913408  15 mango_aux_net,mango_aux_pcc,mango_aux_rdma
mango_mgmt            339968  3 mango_core,mango_aux_pcc,mango_aux_rdma

Upon successful loading, you can check the mb_<i> RDMA interface information using the rdma link command:

~$ rdma link
link mb_0/1 state DOWN physical_state DISABLED netdev ens102np0
link mb_1/1 state DOWN physical_state DISABLED netdev ens104np0

Loading the Driver

To load the driver manually:

~$ sudo modprobe mango-aux-rdma
~$ rdma link
link mb_0/1 state DOWN physical_state DISABLED netdev ens102np0
link mb_1/1 state DOWN physical_state DISABLED netdev ens104np0

Unloading the Driver

To unload the driver:

~$ sudo modprobe -r mango-aux-rdma
~$ rdma link
(empty)

Network Configuration

Interface Activation

Assigning an IP address to the Ethernet interface completes the setup for using the RDMA interface.

~$ sudo ifconfig ens102np0 100.0.10.1/24 up
~$ sudo ifconfig ens104np0 100.0.11.1/24 up

Confirm the RDMA interfaces are active:

~$ rdma link
link mb_0/1 state ACTIVE physical_state LINK_UP netdev ens102np0
link mb_1/1 state ACTIVE physical_state LINK_UP netdev ens104np0

Setting the MTU Size

Properly configuring the MTU (Maximum Transmission Unit) can improve RDMA performance. Mango BoostX™ RoCE AI currently supports two RDMA MTU sizes: 1024 and 4096 bytes.

Note that the Ethernet MTU includes headers for Ethernet, IP, UDP, and RoCEv2. This results in a difference between the Ethernet MTU and the RDMA MTU. The following mappings are recommended:

Ethernet MTU 1500 → RDMA MTU 1024
Ethernet MTU 4500 → RDMA MTU 4096

Check the current MTU setting:

~$ ip link show ens102np0
2: ens102np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...

Set a new Ethernet MTU (e.g., 4500 bytes):

~$ sudo ip link set dev ens102np0 mtu 4500

warning

Do NOT set the Ethernet MTU directly to RDMA MTU values (e.g., 1024). Always use standard Ethernet frame sizes (e.g., 1500 or 4500) to ensure proper operation.

warning

Ensure MTU compatibility across all networking equipment (NICs, switches, routers) to prevent fragmentation or packet drops.

Device Monitoring

Monitoring Device Temperature

Temperature information can be monitored using the mango-ctl CLI tool. The FPGA sensor reports the temperature of the FPGA chip, Board reports the board temperature, and Ethernet reports the QSFP-DD transceiver temperature.

~$ sudo mango-ctl mgmt sensor 0000:44:00.1

Device: 0000:44:00.1

Label    | Value
---------+---------
Board    | 40.000 C
Ethernet | 49.000 C
FPGA     | 72.000 C

Ethtool Statistics

The following RDMA statistics can be monitored using the ethtool -S command.

Counter	Description
`rdma_tx_packets`	Number of transmitted RoCEv2 packets
`rdma_tx_bytes`	Number of transmitted RoCEv2 bytes
`rdma_rx_packets`	Number of received RoCEv2 packets
`rdma_rx_bytes`	Number of received RoCEv2 bytes

RDMA Functionality Test

perftest is the standard benchmarking tool for RDMA NICs. This section shows a simple bandwidth test using Mango BoostX™ RoCE AI.

The test assumes two Mango BoostX™ RoCE AI cards installed in separate servers, connected either directly or through a switch. Both sides must have completed the setup steps above before running the test.

The example below measures RDMA write bandwidth. Both sides issue the same command, but the client side additionally takes the server's IP address. The -d option specifies the RDMA interface to use.

Server-side command
(server) ~$ ib_write_bw -d mb_0

Client-side command
(client) ~$ ib_write_bw -d mb_0 100.0.10.1

Expected output

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mb_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0005 PSN 0xd355d0 RKey 0x000204 VAddr 0x007f32c9b64000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:00:10:01
 remote address: LID 0000 QPN 0x0005 PSN 0x14867a RKey 0x000204 VAddr 0x007f125610c000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:00:10:02
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             21941.37            21937.89                  0.351006
---------------------------------------------------------------------------------------

perftest supports RDMA Read, Write, and Send operations, with both latency and bandwidth tests. It offers a comprehensive set of options, including message size, number of QPs, queue depth, bidirectional testing, and event-based CQ processing.

For more details, refer to linux-rdma/perftest.

Prerequisites​

Officially Supported Linux Distributions​

Package Installation​

Verifying Card Installation​

Driver Management​

Checking Driver Load Status​

Loading the Driver​

Unloading the Driver​

Network Configuration​

Interface Activation​

Setting the MTU Size​

Device Monitoring​

Monitoring Device Temperature​

Ethtool Statistics​

RDMA Functionality Test​