Skip to content

Lustre

LNet

Overview.

Uses kernel modules. ksocklnd.ko for TCP/IP and ko2iblnd.ko for RDMA using OpenFabrics Enterprise Distribution (OFED) network driver (Verbs) and supports fabrics running InfiniBand, Omni-Path, and RDMA over Converged Ethernet (RoCE).

The ID of a network interface in a LNet network is called LNet Network Identifyer (NID) and looks like <address>@<LND protocol><lnd#> or with simpler notation <address>@<lnd><#>.

The available LND Protocols are tcp and o2ib. An example is 10.20.30.11@o2ib0 and 10.70.207.11@tcp0.

Note: Even though the o2ib LNet driver uses OFED Verbs for communications, the IP address of the IB interface is used to identify the IB interface for the initial connection with a peer. After the initial connection, the o2ib LND uses only RDMA for all further communications. By default, socklnd uses TCP port 988 to create connections, and this must not be blocked by any firewalls.

  • https://wiki.lustre.org/Static_LNet_Configuration
  • https://wiki.lustre.org/Optimizing_o2iblnd_Performance
  • https://wiki.lustre.org/Starting_and_Stopping_LNet
  • https://wiki.lustre.org/Dynamic_LNet_Configuration_and_lnetctl

Static Configuration

Documentation

Loads the parameters configured in /etc/modprobe.d/lustre.conf.

Simple configuration with the format: options lnet networks="<lnd><#>(<dev>)[,…]" 

options lnet networks="tcp0(eth1),o2ib0(ib0),o2ib1(ib1)"

To use using regular expressions to configure the networks using ip2nets check the documentation. Documentation

Dynamic Configuration

Documentation

Available on Lustre version 2.7.0 and onwards.

Troubleshooting

  • Kernel module:
    • Load: modprobe -v lnet Starts LNet. This does not load the configuration. Starting lnet as a dependency of Lustre does.
    • Unload: lustre_rmmod
    • Check: lsmod | grep lnet Check if LNet is running
  • lnetctl
    • Start the service: lnetctl lnet configure
    • Start the service and configure the networks: lnetctl lnet configure --all
    • View network configuration: lnetctl net show and lnetctl net show --verbose
    • Stop the service and unload the kernel modules: lctl network down or lctl network unconfigure
    • Configure the network: lnetctl net add --net <LND protocol><lnd#> --if <network interface> [<options> …]
    • lnetctl stats
    • lnetctl peer_credits

lctl controls the Lustre Filesystem and lnetctl works from Lustre 2.7 onwards and allows dynamically configure LNet. I will only use the newer version lnetctl.

Glossary

  • DNE - Distributed Namespace Environment feature to aggregate multiple MDTs (possibly on many MDS’s) into a single filesystem namespace
  • IDIF - OST object ID In FID - specific FID range reserved for compatibility with pre-DNE OST objects
  • IGIF - Inode and Generation In FID - specific FID range reserved for compatibility from Lustre 1.x MDT inode objects
  • FID - File IDentifier - unique 128-bit identifier for every object within a single filesystem.
  • LMV - Logical Metadata Volume - client software layer that handles client (llite) access to multiple MDTs
  • LOD - Logical Object Device - MDS software layer that handles access to multiple MDTs and multiple OSTs
  • LOV - Logical Object Volume - client software layer that handles client (llite) access to multiple OSTs
  • MDC - MetaData Client - client software layer that interfaces to the MDS
  • MDD - Metadata Device Driver - MDS software layer that understands POSIX semantics for file access
  • MDS - MetaData Server - software service that manages access to filesystem namespace (inodes, paths, permission) requests from the client.
  • MDT - MetaData Target - storage device that holds the filesystem metadata (attributes, inodes, directories, xattrs, etc)
  • MGS - Management Server - service that helps clients and servers with configuration
  • MGT - Management Target - storage device that holds the configuration logs
  • OFD - Object Filter Device - OSS software layer that handles file IO
  • OSC - Object Storage Client - client software layer that interfaces to the OST
  • OSD - Object Storage Device - server software layer that abstracts MDD and OFD access to underlying disk filesystems like ldiskfs and ZFS
  • OSP - Object Storage Proxy - server software layer that interfaces from one MDS to the OSD on another MDS or another OSS
  • OSS - Object Storage Server - software service that manages access to filesystem data (read, write, truncate, etc)
  • OST - Object Storage Target - storage device that holds the filesystem data (regular data files, not directories, xattrs, or other metadata)