Initiator Setup
This page covers the host side: connecting to the subsystems exported by Mango BoostX™ NRT with the standard nvme-cli tools, sizing the number of I/O queues, and running a quick performance check.
Complete the appliance-side configuration first, either through Automatic Configuration or Manual Configuration, and confirm with nvmf show that every subsystem has a listener.
The procedure below uses the Linux in-kernel NVMe-oF initiator, driven with nvme-cli. Other initiator implementations are not covered by this guide.
Prerequisites
Install nvme-cli from your distribution, and fio if you intend to measure performance.
(initiator) ~$ sudo apt install nvme-cli fio
The host needs a working RoCEv2-capable RDMA interface on the same network as the appliance. Before connecting, confirm with rdma link that the interface you intend to use reports state ACTIVE and physical_state LINK_UP.
(initiator) ~$ rdma link
If the appliance restricts access to specific hosts, the NQN of this host must be registered on the subsystem. Read it with:
(initiator) ~$ cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:a734913e-15f4-4bcb-9c1b-1ef805909638
Discover and Connect
(initiator) ~$ sudo nvme discover -t rdma -a <target-ip> -s 4420
(initiator) ~$ sudo nvme connect -t rdma -a <target-ip> -s 4420 -n <subsystem-nqn> -i <queues>
Discovery lists the subsystems the appliance exports, one entry per subsystem:
(initiator) ~$ sudo nvme discover -t rdma -a 200.1.1.100 -s 4420
Connect to each subsystem you want to use:
(initiator) ~$ sudo nvme connect -t rdma -a 200.1.1.100 -s 4420 -n nqn.2022-02.io.mangoboost:subsys0 -i 8
(initiator) ~$ sudo nvme connect -t rdma -a 200.1.1.100 -s 4420 -n nqn.2022-02.io.mangoboost:subsys1 -i 8
(initiator) ~$ sudo nvme connect -t rdma -a 200.1.1.100 -s 4420 -n nqn.2022-02.io.mangoboost:subsys2 -i 8
The first few connect attempts after the target has just started may fail. This is expected — retry, and the connection succeeds once the target has settled.
Confirm the namespaces appeared as block devices:
(initiator) ~$ sudo nvme list
A connect that failed with a timeout leaves its controller behind instead of removing it, so the subsystem still appears in nvme list — but with namespace ID 0 and a capacity of 0.00 B:
/dev/nvme2n1 ... 0 0.00 B / 0.00 B 1 B + 0 B
A namespace listed with zero capacity is a failed connection, not a working one. It never becomes a usable block device, and running fio against it reports open() Invalid argument, which is easily mistaken for a data path problem. Disconnect that subsystem and connect again; the namespace is then reported with its real capacity.
I/O Queue Budget
The appliance serves up to 32 NVMe-oF I/O queue pairs per network port, with a queue depth of 32 entries, and up to 32 hosts. That pool is shared by every connection. The per-connection limit is enforced by the target's -m/--max-queues and by the controller's queue size, but nothing caps the sum across connections: if the total requested by all connections exceeds the pool, the connections made later fail while creating their queue pairs.
Size the values so that the total fits:
- Multiply the number of subsystems you will connect to by the queues per connection, and keep the product within the pool.
- With many subsystems, lower both the target's
-mand the initiator's-i. For example, 12 subsystems at 2 queues each requests 24 queues in total. - Automatic configuration already does this for you: it divides its queue budget by the number of subsystems it creates, so a multi-subsystem appliance comes up with values that fit.
Use a recent software release on both sides. Older target software cannot tell the initiator why an over-budget connection was rejected, and the failure surfaces on the host as an unrecognized reason rather than a clear error.
Performance Check
Once connected, the namespaces are ordinary block devices and can be measured with fio:
(initiator) ~$ sudo fio --filename=/dev/nvme0n1 --rw=randrw --bs=4k --ioengine=libaio \
--iodepth=64 --runtime=30 --numjobs=4 --time_based \
--group_reporting --name=nrt --eta-newline=1
Scale numjobs and iodepth to the number of namespaces under test, and run against several namespaces at once to exercise more than one SSD. Confirm the run reported err=0 before reading the result.
The maximum data transfer size (MDTS) is 128 KiB, so a single request larger than that is split by the initiator before it reaches the appliance. Namespaces use a 512-byte logical block size.
Disconnecting
Disconnect by subsystem NQN, using the same NQN given to connect:
(initiator) ~$ sudo nvme disconnect -n nqn.2022-02.io.mangoboost:subsys0
Disconnect every initiator before stopping the target on the appliance. Otherwise nvmf stop is refused, and forcing it tears down connections the host still believes are alive.