Monitoring and Diagnostics
This page covers how to read the current state of a Mango BoostX™ NRT appliance, where the logs are, and what to collect when you open a support request.
Configuration State
nvmf show is the primary status command. It reports the target state on the first line, any warnings on the second, and then places every SSD in the appliance into one of three groups.
(target) ~# mango-ctl nvmf show
NVMe-oF target (group 0): running (RDMA) subsystems 3 capacity 10.9 TB
warnings: none
Exported
NQN LISTENER NAMESPACES
nqn.2022-02.io.mangoboost:subsys0 200.1.1.100:4420 Nvme0n1 (nsid 1, 3.2 TB, 0000:04:00.0)
nqn.2022-02.io.mangoboost:subsys1 200.1.1.100:4420 Nvme1n1 (nsid 1, 3.8 TB, 0000:05:00.0)
nqn.2022-02.io.mangoboost:subsys2 200.1.1.100:4420 Nvme2n1 (nsid 1, 3.8 TB, 0000:06:00.0)
Attached, not exported
(none)
Not attached
(none)
A subsystem can hold several namespaces, and a device the target has not taken over is listed with the identity the kernel reports for it. The example below shows both: two subsystems carrying five namespaces in total, and two drives the target never attached.
(target) ~# mango-ctl nvmf show
NVMe-oF target (group 0): running (RDMA) subsystems 2 capacity 16.0 TB
warnings: 2 NVMe device(s) not attached
Exported
NQN LISTENER NAMESPACES
nqn.2022-02.io.mangoboost:subsys0 200.1.1.100:4420 Nvme0n1 (nsid 1, 3.2 TB, 0000:0b:00.0)
Nvme1n1 (nsid 2, 3.2 TB, 0000:0c:00.0)
nqn.2022-02.io.mangoboost:subsys1 200.1.1.100:4420 Nvme2n1 (nsid 1, 3.2 TB, 0000:11:00.0)
Nvme3n1 (nsid 2, 3.2 TB, 0000:12:00.0)
Nvme4n1 (nsid 3, 3.2 TB, 0000:13:00.0)
Not attached
BDF VENDOR MODEL SIZE
0000:14:00.0 KIOXIA - -
0000:15:00.0 KIOXIA - -
| Group | Meaning |
|---|---|
| Exported | Published in a subsystem. Reachable by an initiator, provided a listener is present |
| Attached, not exported | Held by the target but not published in any subsystem. Not reachable |
| Not attached | Still owned by the kernel NVMe driver. Not part of the target at all |
Read the warnings: line every time. A subsystem with no listener is reported as N subsystem(s) exported without a listener and its LISTENER column shows -. Such a subsystem exists and holds namespaces, but no initiator can connect to it, so it is easy to mistake for a working export.
Devices in the Not attached group are shown with their model and size, read from the kernel. A device that has already been detached to the userspace driver has no such attributes and is shown as -.
For a narrower view, or for machine-readable output:
(target) ~# mango-ctl nvmf show --status # first two lines only
(target) ~# mango-ctl nvmf show --subsystem # subsystems only
(target) ~# mango-ctl nvmf show --devices # device groups only
(target) ~# mango-ctl nvmf show --json # full state as JSON
(target) ~# mango-ctl nvmf list_subsys # NQNs only, also accepts --json
Device View
mango-ctl dev reports the PCIe view of the appliance, which is the quickest way to confirm which driver currently owns each SSD.
(target) ~# mango-ctl dev show nvme
An SSD attached to the target shows uio_pci_generic as its kernel module; one still owned by the kernel shows nvme. To see which device types the command accepts:
(target) ~# mango-ctl dev list type
Passing a type that does not exist prints the list of valid ones rather than failing silently.
Workload Statistics
dump stat is the most useful single command for a first look at performance or errors. It reports the hardware counters in two groups: [NRT] for the NVMe-oF command and data path, and [RDMA] for the RoCEv2 transport underneath it. All counters are per port.
The argument is a device node name or a BDF, not a product name. List the node names first:
(target) ~# mango-smi list
400000000.mb-nrt
(target) ~# mango-ctl dump stat 400000000.mb-nrt
[NRT]
commands_received : 1,234,567
completions_sent : 1,234,560
completions_sent_with_error : 3
read_bytes_completed : 10.24 GiB
write_bytes_completed : 5.41 GiB
[RDMA]
crc_drop : 12
ip_length_drop : 0
invalid_request : 0
recv_not_ready : 5
retransmitted_wqes : 2
local_timeout : 1
NVMe-oF Counters
| Counter | Meaning |
|---|---|
commands_received | NVMe-oF commands received from the hosts |
completions_sent | Completions returned to the hosts |
completions_sent_with_error | Completions returned with an error status |
read_bytes_completed | Data returned for read commands |
write_bytes_completed | Data accepted for write commands |
commands_received minus completions_sent is the number of commands still outstanding. A steadily growing gap means commands are not completing.
RDMA Counters
| Counter | Meaning |
|---|---|
crc_drop | Packets dropped because of a CRC error |
ip_length_drop | Packets dropped because of an invalid IP length |
invalid_request | Requests the responder rejected as invalid |
recv_not_ready | Requests that arrived with no receive buffer posted |
retransmitted_wqes | Work requests that had to be retransmitted |
local_timeout | Transport timeouts detected locally |
These accumulate from the last port reset. crc_drop and ip_length_drop point at the physical link or the cabling; retransmitted_wqes and local_timeout rising together usually mean congestion or packet loss in the fabric.
Selecting a Group
(target) ~# mango-ctl dump stat 400000000.mb-nrt --nrt # NVMe-oF counters only
(target) ~# mango-ctl dump stat 400000000.mb-nrt --rdma # RDMA counters only
(target) ~# mango-ctl dump stat 400000000.mb-nrt --json # machine-readable output
Use --json when a monitoring system collects the counters rather than a person reading them.
Register Dump
dump reg writes a register snapshot to a file and prints the path it used. MangoBoost support may ask for this file.
(target) ~# mango-ctl dump reg 400000000.mb-nrt
The file is written under /var/log/mango/nrt_log/. An alternative directory can be given as a second argument.
Logs
| Source | Command | Contents |
|---|---|---|
| Configuration daemon | journalctl -u mango-jbof -f | Interface setup, SSD discovery, and each configuration step |
| Log files | /var/log/mango/nrt_log/ | Register dump files, named by timestamp |
Collecting Data for a Support Request
When reporting a problem to contact@mangoboost.io, collect the following from the appliance and attach the result as a tar.gz.
(target) ~# mango-ctl nvmf show --json # configuration snapshot
(target) ~# mango-ctl nvmf autoconfig --show # automatic configuration in effect
(target) ~# mango-ctl nvmf autoconfig --diff # pinned vs. running configuration
(target) ~# mango-ctl dump stat 400000000.mb-nrt # workload and error statistics
(target) ~# mango-ctl dump reg 400000000.mb-nrt # register dump
(target) ~# journalctl -u mango-jbof --no-pager -n 500 # configuration daemon log
(target) ~# dmesg | tail -200 # SSD and kernel errors
(target) ~# cat /etc/mango-cli/mango_jbof_config.yaml # policy file
(target) ~# cat /etc/mango-cli/mango_jbof_nvmf_state.yaml # pinned configuration, if any
From the initiator, add the host's view of the connection:
(initiator) ~$ sudo nvme list
(initiator) ~$ sudo nvme list-subsys
(initiator) ~$ rdma link
(initiator) ~$ sudo dmesg | grep nvme