Skip to main content

Troubleshooting

The appliance loses its IP address after a restart or reboot​

  • This is the expected result of leaving the network configuration on DHCP when no address is actually being handed out. The configuration daemon re-applies the interface address every time it starts, so the appliance drops off the network on each restart and on each boot, which looks like a unit that fails intermittently.
  • Check the DHCP server first. Confirm it is running, that it serves the network the appliance is connected to, and that it still has a lease available for the appliance. If DHCP is meant to be used, fixing the server is the correct repair.
  • If the network has no DHCP server, or the appliance is meant to hold a fixed address, set a static address in the policy file and restart the service.
    /etc/mango-cli/mango_jbof_config.yaml
    network:
      mode: "static"
      interfaces:
        - name: "eth0"
          address: "200.1.1.100/24"
  • See Software Setup → Network Configuration.

nvmf show reports subsystems 0 right after boot​

  • This is normal for a short period after a reboot while the SSDs are being discovered and configured. The startup is retried according to startup_retry_max in the policy file.
  • Wait for the export to complete before connecting from a host, rather than waiting a fixed amount of time. 
    (target) ~# mango-ctl nvmf show --status
  • If the state never changes, check the configuration daemon log for the step that failed. 
    (target) ~# journalctl -u mango-jbof -n 200 --no-pager

A subsystem exists but no initiator can connect to it​

  • Check the warnings: line of nvmf show. A subsystem exported without a listener is reachable by nothing, and its LISTENER column shows -.
  • Add the listener, using the address configured on the appliance interface. 
    (target) ~# mango-ctl nvmf add_listener -n <nqn> -a <target-ip>
  • If access is restricted to specific hosts, also confirm the host NQN is registered on that subsystem. Read it on the host with cat /etc/nvme/hostnqn.

nvmf stop appears to hang​

  • Stopping the target is allowed up to 90 seconds to finish, and the service stop has the same timeout. The command reports progress while it works.
  • Wait for it to return. Interrupting it with Ctrl-C leaves the target half torn down.
  • If it is refused rather than slow, the exit code is 6 and initiators are still connected. Disconnect them, or force the teardown with -f.

The later nvme connect calls fail while the first ones succeed​

  • The connection is refused, often reported on the host as Connection reset by peer.
  • The appliance serves up to 32 I/O queue pairs per network port, shared by all connections. Each connection stays within its own limit, but the total across connections can still exceed that, and the connections made later fail while creating their queue pairs.
  • Lower the queues per connection on both sides — the target's -m/--max-queues and the initiator's -i — so that the number of subsystems multiplied by the queues per connection fits the pool. See I/O Queue Budget.
  • The first few connect attempts after the target has just started can also fail for an unrelated reason. Retry before investigating.

Automatic configuration does not recover while an initiator is connected​

  • On earlier software, a restart of the configuration daemon could not tear down a target that still had connections, so the following start failed and the daemon eventually stopped retrying. The symptom is that automatic configuration never comes back as long as a host stays connected.
  • This is fixed in current software. If you see it, check the installed version first and update the appliance.

A command waits a long time and then fails​

  • Requests to the target daemon time out after 120 seconds by default. A timeout means the daemon is not answering, not that the command was wrong.
  • The timeout can be changed for a single command with an environment variable. 
    (target) ~# MANGO_RPC_TIMEOUT_SEC=30 mango-ctl nvmf show
  • Collect the daemon log before restarting anything, so the cause is not lost. See Collecting Data for a Support Request.

An SSD is missing from the export​

  • Check which group it appears in under nvmf show. A device in Not attached is still owned by the kernel driver; one in Attached, not exported has no namespace in any subsystem.
  • A device reporting I/O errors is skipped on purpose while skip_faulty_devices is enabled, and the reason is written to the daemon log. Check the log and the kernel messages for that PCI address. 
    (target) ~# journalctl -u mango-jbof --no-pager | grep -i skip
    (target) ~# dmesg | grep -i "buffer i/o"
  • Do not disable skip_faulty_devices to force such a device into service. Attaching an SSD that is already failing can destabilize the appliance, and the remaining healthy SSDs stay exported without it.
  • Screening a device out ahead of time is best-effort, not a guarantee. The kernel reports I/O errors under a controller name such as nvme1, and the only way to resolve that name to a PCI address is the line the kernel writes when it first probes the device: 
    (target) ~# dmesg | grep "pci function"
    nvme nvme1: pci function 0000:06:00.0
    An SSD the kernel never probed has no such line, so it cannot be matched to a reported error and is not screened out in advance. It is still handled safely: if attaching it or adding its namespace fails, only that device is skipped — its PCI address is written to the log — and the other SSDs stay exported.
  • Do not rely on the controller name to identify a device. The same name can refer to a different SSD after a reboot, so scripts should key on the PCI address reported by nvmf show rather than on nvme<N>.
  • A subsystem holds at most 26 namespaces. Devices beyond that limit are reported as a warning and left unexported; spread them across more subsystems with max_devices_per_subsystem.

Namespaces stop being exported on an appliance with many SSDs​

  • On appliances with a large number of SSDs, the target can run out of internal message resources part-way through publishing the namespaces. This originates below the CLI and does not occur on small configurations.
  • Use the latest software release on the appliance. If the problem persists, report it with the data listed below, including the SSD count.

A manual configuration is reverted by itself​

  • The configuration daemon re-applies its own policy and overwrites a configuration built by hand.
  • Either pin the configuration you built, or disable the daemon. 
    (target) ~# mango-ctl nvmf autoconfig --commit-state -f
    (target) ~# mango-ctl nvmf autoconfig --diff
  • See Pinning the Running Configuration.

Other issues not listed above​

  • If you encounter an issue not covered here, report it to contact@mangoboost.io. Please attach a debug log capturing the state of both the hardware and the software (a tar.gz of /var/log/mango/nrt_log), together with the data listed in Collecting Data for a Support Request.