Instructions for Setting Up and Using pNFS
 | Notice These instructions apply to the July 2008 release. If you have an earlier release (i.e. March 2008, October 2007, August 2007 or June 2007) please refer to this set of instructions. |
There are 3 main components of a pNFS configuration; a client, a metadata server (MDS) and a data server (DS). The BFU archives that you have installed allow you to configure and run each of these components. What follows is information on how to get each component configured and running. If you have any problems with these instructions or find any errors in them along the way, send an email to nfsv41-discuss@opensolaris.org.
Currently, the DS and the MDS must be setup on separate machines. For the basic setup, we recommend that you have three machines. One to use as the client, one to use as the DS and one to use as the MDS. The following instructions give specific examples in order to create this exact setup, but the examples can be extended in order to create different configurations (e.g. MDS and multiple data server(s)). None the less, if you are just starting out with pNFS, we recommend for you to try the basic setup first and then build off of that.
Please note that the command line interfaces shown here are prototypical and will be changing in the future.
Quick and Easy Reference
This section is a quick reference to the commands you will need to issue in order to get your systems configured and running. More detailed information on what the commands do and an example on how to use them is available below!
On the Metadata Server
Step 1: Share a file system:
#sharemgr create -P nfs <group>
#sharemgr add-share -s <path> <group>
Step 2: Add an authorization record for each data server:
#mdsadm -o add -t auth -a ip=<ip-addr-of-ds>
On the Data Server
Step 1: Share a file system:
#sharemgr create -P nfs <group>
#sharemgr add-share -s <path> <group>
Step 2: Create the zpool to use:
#zpool create <pool> <vdev>
Step 3: Create the pNFS dataset to use:
#zfs create -t pnfsdata <pnfs-dataset>
Step 4: Assign the pNFS dataset to the data server:
#dservadm addstor <pnfs-dataset-name>
Step 5: Specify the MDS to use:
#dservadm addmds <ip-addr-and-port-number-of-mds>
Step 6: Lastly, start the data server:
#dservadm enable
On the Client
#mount -F nfs -o vers=4 <resource> <mount point>
Tips and Tricks
- Every time you reboot the MDS, you need to reconfigure it using mdsadm. This is because the data server information is currently not stored persistently. This means that you will also have to re-enable the your data servers in order to allow them to report to the MDS. If the MDS reboots, execute the following commands:
| Potential gotcha! The following commands must be executed in order. |
Step 1: On the MDS - Authorize each data server:
#mdsadm -o add -t auth -a ip=<ip-addr-of-ds>
Step 2: On the DS - Restart the data server:
#dservadm disable
#dservadm enable
- Anytime ZFS adds a new DMU object type, you need to rebuild your pools. If this occurs, it will be mentioned in the appropriate release notes.
- To verify pNFS is working you can use snoop to watch the traffic between the client and the data servers. In your capture file, search for "pnfs write" or "pnfs read"
- Do NOT create your zpools using files in /tmp as the backing store.
Detailed Information on the Commands Used to Manage pNFS
This section covers the same information as the "Quick and Easy Reference", but goes into more detail on what each command does.
Pre-configuration - Turn on Debugging (optional)
- To turn on debugging, add the following entry to /etc/syslog.conf:
- *.info /var/adm/messages - Set dserv:dserv_debug to 1 in /etc/system
Instructions for Setting up a Metadata Server
Share a file system:
#sharemgr create <group>
#sharemgr add-share -s <path> <group>
mdsadm Command
The configuration of the metadata server (MDS) is done with the mdsadm command. It allows you to input a list of data server "authorization" records and layouts for the MDS. The syntax is as follows:
mdsadm -o <operation> -t <type>
Valid values for operation include "add"
Valid values for type include "auth" and "layout"
A data server "authorization" record notifies the metadata server that the given data server will be communicating with it.
A layout determines how the file will be laid out at creation time (e.g. 3 way stripe, 32K stripe unit or interlace).
A default layout will be generated based on the device information reported via the control-protocol by the data-servers. This information is not currently stored persistently. Therefore the above step will need to be done after every reboot of the metadata server.
Instructions for Setting up a Data Server
Create a zpool
#zpool create <pool> <vdev>
Create a pNFS dataset
There should be at least one pNFS dataset per data server instance, so if you are planning on having multiple instances on your data server, be sure to create at least one dataset per instance. Although, for now we recommend using a single instance of the data server.
#zfs create -t pnfsdata <pnfs-dataset>
Simple (single instance) Data Server Configuration
dservadm Command
The configuration of the data server is done using the dservadm command. Please note that the dservadm command line interface is an interim solution. This command will be replaced with a more complete CLI in the near future. The syntax is as follows:
|
Command |
Description |
|
dservadm create <instance> |
Creates an instance of the data server with the name as given. This can be used to create multiple instances of the data server on one physical machine. By default, there is one instance of the data server. This is referred to as the default instance. This capability is experimental and is not expected to be delivered with the final pNFS implementation. |
|
dservadm destroy <instance> |
Deletes the instance of the data server with the name given. This capability is experimental and is not expected to be delivered with the final pNFS implementation. |
|
dservadm enable [ -i <instance> ] |
Starts the data server SMF service. If given an instance name, that instance will be started. If not given an instance name, the default instance will be started. |
|
dservadm disable [ -i <instance> ] |
Stops the data server SMF service. If given an instance name, that instance will be stopped. If not given an instance name, the default instance will be stopped. |
|
dservadm addstorage [ -i <instance> ] <pnfs-dataset-name> |
Adds the given pNFS dataset to the data server for use in doing I/O. If given an instance name, the dataset will be added to that instance. If not given an instance name, the dataset will be added to the default instance. |
|
dservadm addstor [ -i <instance> ] <pnfs-dataset-name> |
Alias for dservadm addstorage. |
|
dservadm dropstorage [ -i <instance> ] <pnfs-dataset-name> |
Deletes the given pNFS dataset from the data server. This will stop the data server from using the dataset to do I/O. If given an instance name, the dataset will be deleted from that instance. If not given an instance name, the dataset will be deleted from the default instance. |
|
dservadm dropstor [ -i <instance> ] <zpool-name> |
Alias for dservadm dropstorage. |
|
dservadm liststorage [ -i <instance> ] |
Lists the pNFS datasets that are available to the data server. If given an instance name, the datasets that are available to that instance are displayed. If not given an instance name, the datasets for the default instance are displayed. |
|
dservadm liststor [ -i <instance> ] |
Alias for dservadm liststorage. |
|
dservadm addmds [ -i <instance> ] <mds-universal-address> |
Adds the given metadata server universal address to the data server. This will allow the data server and metadata server to communicate. If given an instance name, the mds is added to that instance. If not given an instance name, the mds is added to the default instance. There can only be one metadata server added to any instance of the data server. |
|
dservadm dropmds [ -i <instance> ] |
Deletes the metadata server. This will remove the metadata server from the data server and will stop the communication between the two. If given an instance name, the mds will be deleted from that instance. If not given an instance name, the mds will be deleted from the default instance. |
|
dservadm listmds [ -i <instance> ] |
Lists the metadata server that the data server is in communication with. If given an instance name, the mds that is in use by that instance is displayed. If not given an instance name, the mds being used by the default instance are displayed. |
Instructions for Setting up a Client
#mount -F nfs -o vers=4 <resource> <mount point>
The resource will be the shared file system on the metadata server (e.g. mds:/export).
The mount point is where you want the file system mounted on the client.
Example of using the commands above to configure pNFS
(Note: In this example, the IP of the MDS is 172.20.48.143 and the IP of the DS is 172.20.48.115)
On the MDS:
Share a file system:
#sharemgr create -P nfs mygroup
#sharemgr add-share -s /export mygroup
Specify the data server that is allowed to talk to this MDS:
#mdsadm -o add -t auth -a ip=172.20.48.115
On the DS:
Share a file system:
(Note: This step of sharing a file system on the data server is a temporary workaround. The data server will not actually use this file system. The data server SMF service (svc:/network/dserv:default) depends on the NFS server SMF service (svc:/network/nfs/server:default). Sharing a file system guarantees that the NFS server service is up and running.)
#sharemgr create -P nfs mygroup
#sharemgr add-share -s /export mygroup
Add the address of the MDS for use by the data server:
#dservadm addmds 172.20.48.143.8.1
 | Handy Hint 172.20.48.143 is the IP address, 8.1 is the port number (8.1 or (8*256 + 1) = 2049). Unless you went out of your way to change the port that the Metadata Server is running on, this will always be 8.1 |
To list the MDS that has been assigned to the data server:
#dservadm listmds
mds:
172.20.48.143.8.1
To create a pNFS dataset
#zpool create dspool /dev/dsk/c1t1d0
#zfs create -t pnfsdata dspool/pnfsds
To view all pNFS datasets on a data server:
#zfs list -t pnfsdata
NAME USED AVAIL REFER MOUNTPOINT
dspool/pnfsds 572M 66.4G 15K -
Add the pNFS dataset to the list of datasets for use by the data server:
#dservadm addstor dspool/pnfsds
To list the datasets that have been added for use by the data server:
#dservadm liststor
storage:
dspool/pnfsds
Start the data server service:
#dservadm enable
To determine the state of the data server service:
#svcs dserv
STATE STIME FMRI
online 19:29:24 svc:/network/dserv:default
On the client:
To mount the pNFS file system:
#mount -F nfs -o vers=4 172.20.48.143:/export /mnt