As an example of how to specify SRP connection fields, consider the Open Fabrics Enterprise Distribution (OFED) implementation of SRP on Linux initiators. The following notes are based on the file {{srp_release_notes.txt}} from the OFED. The current release of the OFED distribution can be found [here|http://www.openfabrics.org/downloads/OFED/], and it includes an online copy of [srp_release_notes.txt|http://www.openfabrics.org/downloads/OFED/ofed-1.4/OFED-1.4-docs/srp_release_notes.txt].
For a deeper discussion of SRP Addressing and Discovery that may be helpful when configuring other kinds of SRP Initiator, refer to the optional section on [Understanding SRP Addressing and Discovery].
h3. Enabling SRP - Linux Initiator
To load the SRP module, either execute the "{{modprobe ib_srp}}" command after the OFED driver is up, or change the value of {{SRP_LOAD}} in {{/etc/infiniband/openib.conf}} to "yes" (causing the srp module to be loaded at driver boot).
h3. SRP Discovery Example - Linux Initiator
To use IB Device Management services to search for available SRP storage that is visible using port 2 of the first IB HCA (HCA number 0) on the system (known to Linux as {{mthca0}}), use the following command. The {{\-c}} option to {{srp_daemon}} produces output in a convenient one-line-per-target format that is re-used below. The {{\-n}} option causes the output to contain a default setting of the {{initiator_ext}} parameter that works well when enabling simultaneous SRP sessions to distinct target ports on the target system, to take advantage of multipath I/O.
{code:none}
# srp_daemon -a -n -c -o -i mthca0 -p 2
id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,dgid=fe800000000000000003ba0001002e49,pkey=ffff,service_id=0003ba0001002e48,initiator_ext=492e000100ba0300
id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,dgid=fe800000000000000003ba0001002e4a,pkey=ffff,service_id=0003ba0001002e48,initiator_ext=4a2e000100ba0300
{code}
h3. SRP Connection Command Example - Linux Initiator
In Linux OFED SRP, management capabilities are exported through a synthetic directory that corresponds to a particular HCA and port. Each port has a subdirectory under {{/sys/class/infiniband_srp}} named {{mthcaHCA-PORT}}. For example, the directory corresponding to the _2nd_ port on the _1st_ HCA (HCA number 0) in the system is named {{/sys/class/infiniband_srp/mthca0-2}}.
To create an SRP connection, create a file named {{add-target}} in the directory that contains an entry of name-value pairs that describes the desired connection. For example, to create a connection from port 2 on the initiator HCA (mthca0) to a COMSTAR SRP target HCA with HCA GUID {{0003ba0001002e48}}, use the following commands:
{code:none}
echo -n id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,\
dgid=fe800000000000000003ba0001002e4a,pkey=ffff,\
service_id=0003ba0001002e48,initiator_ext=4a2e000100ba0300\
> /sys/class/infiniband_srp/mthca0-2/add_target
{code}
Note that the output format of {{srp_daemon -c}} is suitable for sending to {{add_target}}. As a simple way to create a connection to all visible SRP targets, use the single command {{srp_daemon -e}}. This form of {{srp_daemon}} performs the {{add_target}} function directly.
{code:none}
srp_daemon -o -e -i mthca0 -p 2
{code}
h3. Verifying Available Storage Logical Units
Different initiator systems have different mechanisms for displaying the set of available logical units. There may be separate commands to display the available SCSI targets and to display the available storage logical units. For example, on some Linux varieties, you can {{cat /proc/scsi/scsi}}. You can also install the optional {{lsscsi}} command. The following example is from Red Hat.
{code:none}
# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: ST380013AS Rev: 3.18
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: SUN Model: COMSTAR Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 05
{code}
{code:none}
# lsscsi
[0:0:0:0] disk ATA ST380013AS 3.18 /dev/sda
[2:0:0:0] disk SUN COMSTAR 1.0 /dev/sdc
{code}
The first entry above is a locally attached disk and the second entry represents one particular COMSTAR SRP target.
To display available disks on Linux, use {{fdisk \-l}}. The disk {{/dev/sda}} corresponds to the locally attached SCSI disk identified above, and contains two partitions. The disks {{/dev/sdc}} and {{/dev/sdd}} are remote logical units that correspond to a SCSI host called {{Host 2}}. The disk {{/dev/sdc}} and {{/dev/sdd}} in this example do not contain a valid partition table because neither logical unit has yet been formatted.
{code:none}
# fdisk -l
Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 9729 78043770 8e Linux LVM
Disk /dev/sdc: 53.6 GB, 53687025664 bytes
64 heads, 32 sectors/track, 51199 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 53.6 GB, 53687025664 bytes
64 heads, 32 sectors/track, 51199 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdd doesn't contain a valid partition table
{code}
h3. SRP Connection Parameters Example - Linux Initiator
When creating an SRP connection, the following parameters must be specified:
* IB connection parameters - To identify the endpoints of the desired IB channel
* SRP parameters - To further specify the desired connection
The following table describes each required item as well as the short keyword that OFED uses for each one.
*IB Connection Parameters*
|| OFED keyword || Item Name || Description ||
| _directory name_ | Initiator HCA and port | The initiator IB endpoint, specified as HCA and port |
| {{dgid}} | Target port | IB subnet address (destination GID) of the HCA physical port |
| {{pkey}} | Partition key | IB partition identifier for this connection |
*Desired Service Parameters*
|| OFED keyword || Item Name || Description || Set To ||
| {{ini-ext}} | Initiator extension | Set arbitrarily by initiator administrator | arbitrary |
| {{ioc_guid}} | Target I/O controller GUID | GUID of target SRP-capable I/O Controller | tgt HCA GUID |
| {{id_ext}} | SCSI target ID | ID of desired SCSI target | tgt HCA GUID |
| {{service_id}} | Target service ID | IB service ID of target SRP IB service | tgt HCA GUID |
h2. Where to Go Next
* [Understanding SRP Addressing and Discovery]
For a deeper discussion of SRP Addressing and Discovery that may be helpful when configuring other kinds of SRP Initiator, refer to the optional section on [Understanding SRP Addressing and Discovery].
h3. Enabling SRP - Linux Initiator
To load the SRP module, either execute the "{{modprobe ib_srp}}" command after the OFED driver is up, or change the value of {{SRP_LOAD}} in {{/etc/infiniband/openib.conf}} to "yes" (causing the srp module to be loaded at driver boot).
h3. SRP Discovery Example - Linux Initiator
To use IB Device Management services to search for available SRP storage that is visible using port 2 of the first IB HCA (HCA number 0) on the system (known to Linux as {{mthca0}}), use the following command. The {{\-c}} option to {{srp_daemon}} produces output in a convenient one-line-per-target format that is re-used below. The {{\-n}} option causes the output to contain a default setting of the {{initiator_ext}} parameter that works well when enabling simultaneous SRP sessions to distinct target ports on the target system, to take advantage of multipath I/O.
{code:none}
# srp_daemon -a -n -c -o -i mthca0 -p 2
id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,dgid=fe800000000000000003ba0001002e49,pkey=ffff,service_id=0003ba0001002e48,initiator_ext=492e000100ba0300
id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,dgid=fe800000000000000003ba0001002e4a,pkey=ffff,service_id=0003ba0001002e48,initiator_ext=4a2e000100ba0300
{code}
h3. SRP Connection Command Example - Linux Initiator
In Linux OFED SRP, management capabilities are exported through a synthetic directory that corresponds to a particular HCA and port. Each port has a subdirectory under {{/sys/class/infiniband_srp}} named {{mthcaHCA-PORT}}. For example, the directory corresponding to the _2nd_ port on the _1st_ HCA (HCA number 0) in the system is named {{/sys/class/infiniband_srp/mthca0-2}}.
To create an SRP connection, create a file named {{add-target}} in the directory that contains an entry of name-value pairs that describes the desired connection. For example, to create a connection from port 2 on the initiator HCA (mthca0) to a COMSTAR SRP target HCA with HCA GUID {{0003ba0001002e48}}, use the following commands:
{code:none}
echo -n id_ext=0003ba0001002e48,ioc_guid=0003ba0001002e48,\
dgid=fe800000000000000003ba0001002e4a,pkey=ffff,\
service_id=0003ba0001002e48,initiator_ext=4a2e000100ba0300\
> /sys/class/infiniband_srp/mthca0-2/add_target
{code}
Note that the output format of {{srp_daemon -c}} is suitable for sending to {{add_target}}. As a simple way to create a connection to all visible SRP targets, use the single command {{srp_daemon -e}}. This form of {{srp_daemon}} performs the {{add_target}} function directly.
{code:none}
srp_daemon -o -e -i mthca0 -p 2
{code}
h3. Verifying Available Storage Logical Units
Different initiator systems have different mechanisms for displaying the set of available logical units. There may be separate commands to display the available SCSI targets and to display the available storage logical units. For example, on some Linux varieties, you can {{cat /proc/scsi/scsi}}. You can also install the optional {{lsscsi}} command. The following example is from Red Hat.
{code:none}
# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: ATA Model: ST380013AS Rev: 3.18
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: SUN Model: COMSTAR Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 05
{code}
{code:none}
# lsscsi
[0:0:0:0] disk ATA ST380013AS 3.18 /dev/sda
[2:0:0:0] disk SUN COMSTAR 1.0 /dev/sdc
{code}
The first entry above is a locally attached disk and the second entry represents one particular COMSTAR SRP target.
To display available disks on Linux, use {{fdisk \-l}}. The disk {{/dev/sda}} corresponds to the locally attached SCSI disk identified above, and contains two partitions. The disks {{/dev/sdc}} and {{/dev/sdd}} are remote logical units that correspond to a SCSI host called {{Host 2}}. The disk {{/dev/sdc}} and {{/dev/sdd}} in this example do not contain a valid partition table because neither logical unit has yet been formatted.
{code:none}
# fdisk -l
Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 9729 78043770 8e Linux LVM
Disk /dev/sdc: 53.6 GB, 53687025664 bytes
64 heads, 32 sectors/track, 51199 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 53.6 GB, 53687025664 bytes
64 heads, 32 sectors/track, 51199 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdd doesn't contain a valid partition table
{code}
h3. SRP Connection Parameters Example - Linux Initiator
When creating an SRP connection, the following parameters must be specified:
* IB connection parameters - To identify the endpoints of the desired IB channel
* SRP parameters - To further specify the desired connection
The following table describes each required item as well as the short keyword that OFED uses for each one.
*IB Connection Parameters*
|| OFED keyword || Item Name || Description ||
| _directory name_ | Initiator HCA and port | The initiator IB endpoint, specified as HCA and port |
| {{dgid}} | Target port | IB subnet address (destination GID) of the HCA physical port |
| {{pkey}} | Partition key | IB partition identifier for this connection |
*Desired Service Parameters*
|| OFED keyword || Item Name || Description || Set To ||
| {{ini-ext}} | Initiator extension | Set arbitrarily by initiator administrator | arbitrary |
| {{ioc_guid}} | Target I/O controller GUID | GUID of target SRP-capable I/O Controller | tgt HCA GUID |
| {{id_ext}} | SCSI target ID | ID of desired SCSI target | tgt HCA GUID |
| {{service_id}} | Target service ID | IB service ID of target SRP IB service | tgt HCA GUID |
h2. Where to Go Next
* [Understanding SRP Addressing and Discovery]