How to Configure an SRP Initiator

As an example of how to specify SRP connection fields, consider the Open Fabrics Enterprise Distribution (OFED) implementation of SRP on Linux initiators. The following notes are based on the file srp_release_notes.txt from the OFED. The current release of the OFED distribution can be found here, and it includes an online copy of srp_release_notes.txt.

For a deeper discussion of SRP Addressing and Discovery that may be helpful when configuring other kinds of SRP Initiator, refer to the optional section on Understanding SRP Addressing and Discovery.

Enabling SRP - Linux Initiator

To load the SRP module, either execute the "modprobe ib_srp" command after the OFED driver is up, or change the value of SRP_LOAD in /etc/infiniband/openib.conf to "yes" (causing the srp module to be loaded at driver boot).

SRP Discovery Example - Linux Initiator

To use IB Device Management services to search for available SRP storage that is visible using port 2 of the first IB HCA (HCA number 0) on the system (known to Linux as mthca0), use the following command. The -c option to srp_daemon produces output in a convenient one-line-per-target format that is re-used below. The -n option causes the output to contain a default setting of the initiator_ext parameter that works well when enabling simultaneous SRP sessions to distinct target ports on the target system, to take advantage of multipath I/O.

# srp_daemon -a -n -c -o -i mthca0 -p 2
id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,dgid=fe800000000000000003ba0001002e49,pkey=ffff,service_id=0003ba0001002e48,initiator_ext=492e000100ba0300
id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,dgid=fe800000000000000003ba0001002e4a,pkey=ffff,service_id=0003ba0001002e48,initiator_ext=4a2e000100ba0300

SRP Connection Command Example - Linux Initiator

In Linux OFED SRP, management capabilities are exported through a synthetic directory that corresponds to a particular HCA and port. Each port has a subdirectory under /sys/class/infiniband_srp named mthcaHCA-PORT. For example, the directory corresponding to the 2nd port on the 1st HCA (HCA number 0) in the system is named /sys/class/infiniband_srp/mthca0-2.

To create an SRP connection, create a file named add-target in the directory that contains an entry of name-value pairs that describes the desired connection. For example, to create a connection from port 2 on the initiator HCA (mthca0) to a COMSTAR SRP target HCA with HCA GUID 0003ba0001002e48, use the following commands:

echo -n id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,\
dgid=fe800000000000000003ba0001002e4a,pkey=ffff,\
service_id=0003ba0001002e48,initiator_ext=4a2e000100ba0300\
 > /sys/class/infiniband_srp/mthca0-2/add_target

Note that the output format of srp_daemon -c is suitable for sending to add_target. As a simple way to create a connection to all visible SRP targets, use the single command srp_daemon -e. This form of srp_daemon performs the add_target function directly.

srp_daemon -o -e -i mthca0 -p 2

Verifying Available Storage Logical Units

Different initiator systems have different mechanisms for displaying the set of available logical units. There may be separate commands to display the available SCSI targets and to display the available storage logical units. For example, on some Linux varieties, you can cat /proc/scsi/scsi. You can also install the optional lsscsi command. The following example is from Red Hat.

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST380013AS       Rev: 3.18
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: SUN      Model: COMSTAR          Rev: 1.0
  Type:   Direct-Access                    ANSI SCSI revision: 05

# lsscsi
[0:0:0:0]    disk    ATA      ST380013AS       3.18  /dev/sda
[2:0:0:0]    disk    SUN      COMSTAR          1.0   /dev/sdc

The first entry above is a locally attached disk and the second entry represents one particular COMSTAR SRP target.

To display available disks on Linux, use fdisk -l. The disk /dev/sda corresponds to the locally attached SCSI disk identified above, and contains two partitions. The disks /dev/sdc and /dev/sdd  are remote logical units that correspond to a SCSI host called Host 2. The disk /dev/sdc and /dev/sdd in this example do not contain a valid partition table because neither logical unit has yet been formatted.

# fdisk -l

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        9729    78043770   8e  Linux LVM

Disk /dev/sdc: 53.6 GB, 53687025664 bytes
64 heads, 32 sectors/track, 51199 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 53.6 GB, 53687025664 bytes
64 heads, 32 sectors/track, 51199 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdd doesn't contain a valid partition table

SRP Connection Parameters Example - Linux Initiator

When creating an SRP connection, the following parameters must be specified:

• IB connection parameters - To identify the endpoints of the desired IB channel
• SRP parameters - To further specify the desired connection

The following table describes each required item as well as the short keyword that OFED uses for each one.

IB Connection Parameters

Desired Service Parameters

Where to Go Next

• Understanding SRP Addressing and Discovery