pNFS Control Protocol Specification

h3.Solaris pNFS Control Protocol 1.1

The control protocol facilitates the management of NFSv4.1 state and data-sets at the DSs. It handles:
* MDS and DS restart / network partition indication.
* Filehandle, File StateID, and layout validation.
* Client access validation (has access to MDS-FS and is using appropriate security mechanism).
* Reporting of DS resources.
* Management of Inter-DS data movement.
* MDS Proxy I/O.
* DS state invalidation.

The control protocol is broken up into 3 RPC programs, that have been assigned the program numbers of 104000, 104001 and 104002.

Messages that flow from the DS to MDS use program number 104001 that has the name of PNFS_CTL_DS, messages that flow from the MDS to a DS use program number 104000 that has the name of PNFS_CTL_MDS, messages that flow between DS and DS to copy and move file data use program number 104002 and has the name PNFS_CTL_MV.

The DS receives filehandles that are in the following format:
* Type A type to indicate filehandle usage.
* Version A version number to allow changes.
* Flags Flags.
* Generation A generation to expire invalid filehandles
* MDS id The owning MDS identifier
* MDS SID The MDS storage identifier which identifies a piece of DS storage (DS_GUID)
* MDS dataset id The containing dataset id
* Object id The object identifier.

The control protocol provides the means to identify the MDS id via the DS_EXIBI message, and the mapping for the MDS_SID to DS_GUID via messages DS_REPORTAVAIL and DS_MAP_MDSSID.

The MDS embeds the MDS dataset id so that it may subsequently invalidate all state held by the data server for a specific file system. This is needed in situations such as the un-sharing of a file system or when the security mechanism in use by the client and metadata server have changed (re-share).

h4.Control Protocol Messages:

h5.DS to MDS: (PNFS_CTL_DS / 1040001)

The general model is for the DS to lazily request validation of presented state associated with an I/O operation from a NFS Client. It is expected that the DS will cache the results, and when needed the MDS will invalidate the state via the DS_INVALIDATE message.

h6.DS_CHECKSTATE - Validate state.

Issued by the DS asking MDS to validate the provided FH, Client and stateid.

On successful validation MDS provides back the layout segments that are associated with the client/stateid, the MDS dataset id, the MDS clientid corresponding to the stateid and the effective open mode.

*Arguments:*
{code}
struct DS_CHECKSTATEargs
{
stateid4 stateid;

*Results:*
{code}
struct layout_info
{
stateid4 layoutstateid;
offset4 offset;
length4 length;
layoutiomode4 iomode;
uint32_t device_count;
uint32_t positions_in_stripe<>;
};

struct ds_filestate
{
clientid4 mds_clid;
layout_info layout;
int open_mode;
};

union DS_CHECKSTATEres switch (ds_status status)
{
case DS_OK:
ds_filestate file_state;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_ACCESS, DSERR_STALE,
DSERR_BAD_FH, DSERR_EXPIRED, DSERR_GRACE, DSERR_WRONGSEC,
DSERR_PNFS_NO_LAYOUT
{code}

*Notes:*

The files' layout(s) for the associated client returned to the DS can be used by the DS to validate that the client is issuing I/O to an appropriate region of the file held by the DS.

The MDS returns the short-hand clientid that is associated with the stateid in order to invalidate the state held at the DS should the NFS Client lose the lease at the MDS.

If the client holds no active layout segment the MDS will return DSERR_PNFS_NO_LAYOUT.

h6.DS_EXIBI -- Exchange Identity and Boot Instance.

Exchange identities and boot instances (DS_EXIBI) follows the general intent for the NFSv4.0 OP_SETCLIENTID, and the NFSv4.1 OP_EXCHANGE_ID.

*Description:*

DS_EXIBI should be issued by the DS to an MDS to establish an identity with the MDS. The data server provides a unique identity string and boot verifier. The MDS replies with a short-hand DS identifier, the MDS's boot verifier, a short-hand MDS identifier and the current lease period.

The DS will present the short version identifier on subsequent DS_REPORTAVAIL and DS_RENEW procedure calls.

*Data Types:*

{code}
typedef uint64_t cp_id;

typedef identity
{
ds_verifier boot_verifier;
opaque instance;
};
{code}

*Argument:*

{code}
struct DS_EXIBIargs
{
identity ds_ident;
};
{code}

*Result:*

{code}
struct DS_EXIBIresok
{
cp_id ds_id;
cp_id mds_id;
ds_verifier mds_boot_verifier;
uint32_t mds_lease_period;
};

union DS_EXIBIres switch (ds_status status)
{
case DS_OK:
DS_EXIBIresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL
{code}

*Notes:*

The MDS implementation can use the DS identity to logically collect the information it holds for that particular identity.

It could for example have quick access to the collection of universal addresses for the control-protocol, so when the MDS is acting in proxy_io mode (on behalf of a NFS Client) it may trunk the I/O over multiple interfaces.

The DS may hold state on behalf of a NFS Client for multiple MDSs. The mds_lease_periods from all the MDSs should be taken into account when calculating the lease period of the state, and idle sessions/clientids at the DS.

h6.DS_MAP_MDSSID - Return a mapping for an mds sid.

This is a DS to MDS message to retrieve a mapping for a DS_GUID.

*Description:*

Although the DS and MDS establish the DS_GUID to MDS_SID mapping at DS_REPORTAVIL time, it is likely that an MDS administrator will create a new pnfs zfs filesystem(s). When this occurs the DS will not have the DS_GUID to MDS_SID mapping when a new MDS_SID is presented in a layout filehandle.

This message provides a method for the DS to request the mapping if it should encounter an unknown MDS_SID embedded in a filehandle. The DS will know to which MDS it should direct the message since the filehandle also carries the mds_id.

The mds_id is obtained by the DS in the reply to the DS_EXIBI message.

*Data Types:*

{code}
struct ds_guid_map
{
ds_guid ds_guid;
mds_sid mds_sid_array<>;
};
{code}

*Arguments:*

{code}
DS_MAP_MDSSIDargs
{
mds_sid mma_sid;
};
{code}

*Results:*

{code}
struct DS_MAP_MDSSIDresok
{
mds_guid_map guid_map;
};

union DS_MAP_MDSSIDres switch (ds_status status)
{
case DS_OK:
DS_MAP_MDSSIDresok resok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_GRACE, DSERR_BAD_MDSSID
{code}

h6.DS_MAP_MDS_DATASET_ID - Query root path.

This is a data server to MDS message to query the root path for a MDS dataset id.

*Description:*

The MDS dataset id is one that the data server has obtained from the filehandle. The MDS will return the root path of the file system that corresponds to the MDS dataset id.

*Arguments:*

{code}
struct DS_MAP_MDS_DATASET_IDargs
{
mds_dataset_id mds_dataset_id;
};

struct DS_MAP_MDS_DATASET_IDresok
{
ds_status status;
utf8string pathname;
};
{code}

*Results:*

{code}
union DS_MAP_MDS_DATASET_IDres switch (ds_status status)
{
case DS_OK:
DS_MAP_MDS_DATASET_IDresok res_ok;
default:
void;
};
{code}

h6.DS_RENEW - Exchange boot instance verifier.

A message from the DS to MDS to test the connection to MDS and to detect restarts.

*Description:*

A message from the DS to MDS used to exchange a ds_verifier. This can be used to indicate to the MDS when a data server has restarted, and also to the DS when the MDS has restarted.

In issuing this message on a regular basis the DS can also detect when there is a network partition.

*Arguments:*

{code}
struct DS_RENEWargs
{
cp_id ds_id;
ds_verifier ds_boottime;
};
{code}

*Results:*

{code}
union DS_RENEWres switch (ds_status status)
{
case DS_OK:
ds_verifier mds_boottime;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_STALE_DSID
{code}

*Notes:*

When the DS or the MDS detect that a restart has occurred. The DS should drop all state that it has cached for that particular MDS and the MDS should drop any state that it believes the DS is holding.

If the MDS has restarted then it will not know about ds_id, in this case MDS will return DSERR_STALE_DSID.

h6.DS_REPORTAVAIL - Data Server resource availability and status.

Issued by the data server informing the MDS of available storage (e.g. pNFS datasets), attributes of available storage (e.g. size available, size used) and IP Addresses.

*Description*

The data server supplies the short-hand version ds_id obtained from DS_EXIBI so the MDS can locate the correct DS Identity. The DS and MDS communicate which set of storage attributes each understands by setting the ds_attrvers member of DS_REPORTAVAILargs and DS_REPORTAVAILres, respectively.

*Data Types:*

{code}
struct ds_zfsattr
{
utf8string attrname;
opaque attrvalue<>;
};

typedef uint32_t ds_addruse;

/* Intended use for the addrs */
const NFS = 0x00000001;
const DSERV = 0x00000002;

struct ds_addr
{
netaddr4 addr;
ds_addruse validuse;
};

struct ds_zfsguid
{
uint64_t zpool_guid;
uint64_t dataset_guid;
};

enum storage_type
{
ZFS = 1
};

union ds_guid switch (storage_type stor_type)
{
case ZFS:
opaque zfsguid<>;
default:
void;
};

struct ds_guid_map
{
ds_guid ds_guid;
mds_sid mds_sid_array<>;
};

struct ds_zfsinfo
{
ds_guid_map guid_map;
ds_zfsattr attrs<>;
};

union ds_storinfo switch (storage_type type)
{
case ZFS:
ds_zfsinfo zfs_info;
default:
void;
};

enum ds_attr_version
{
DS_ATTR_v1 = 1
};
{code}

*Arguments:*

{code}
struct DS_REPORTAVAILargs
{
cp_id ds_id;
ds_verifier ds_verifier;
struct ds_addr ds_addrs<>;
ds_attr_version ds_attrvers;
ds_storinfo ds_storinfo<>;
};
{code}

*Results:*

{code}
struct DS_REPORTAVAILresok
{
ds_attr_version ds_attrvers;
ds_guid_map guid_map<>;
};

union DS_REPORTAVAILres switch (ds_status status)
{
case DS_OK:
DS_REPORTAVAILresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_BAD_DSID
{code}

h6.DS_SECINFO - Return security information

A message from the DS to the MDS to query object security flavors.

*Description:*

There is a need for the data server to inquire on the security flavors used for an object at the MDS. The case in particular is when the NFS Client uses an invalid security flavor, the DS replies with the NFSv4 error NFS4ERR_WRONGSEC and the NFS Client then issues a SECINFO_NO_NAME.

- Arguments:

{code}
struct DS_SECINFOargs
{
nfs_fh4 object;
netaddr4 *cl_addr;
};

union ds_secinfo switch (uint32_t flavor)
{
case RPCSEC_GSS:
rpcsec_gss_info flavor_info;
default:
void;
};
{code}

*Results:*

{code}
typedef ds_secinfo DS_SECINFOresok<>;

union DS_SECINFOres switch (ds_status status)
{
case DS_OK:
DS_SECINFOresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_STALE_FH, DSERR_BAD_FH
{code}

h6.DS_SHUTDOWN - Orderly DS shutdown

This is a message from the DS to the MDS informing the MDS that the DS is in the process of an orderly shutdown.

*Arguments:*

{code}
struct DS_SHUTDOWNargs
{
cp_id ds_id;
};
{code}

*Results:*

{code}
struct DS_SHUTDOWNres
{
ds_status status;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DSERR_NOT_AUTH, DSERR_STALE_DSID
{code}

h6.DS_FMATPT - transport FMA event(s)

This message is a DS to MDS message that will facilitate the transportation of FMA events to the MDS.

*Arguments:*

{code}
struct DS_FMATPTargs {
opaque fma_msg<>;
};
{code}

*Results:*

{code}
struct DS_FMATPTres {
ds_status status;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DSERR_NOTSUPP
{code}

h5.MDS to DS: (PNFS_CTL_MDS / 104000)

h6.DS_COMMIT - Commit write.

Issued by the MDS requesting that the DS write any non-stable data for the given filehandle, for the given file segments.

*Data Types:*

{code}
struct ds_fileseg
{
offset4 offset;
count4 count;
};
{code}

*Arguments:*

{code}
struct DS_COMMITargs {
nfs_fh4 fh;
ds_fileseg cmv<>;
};
{code}

*Results:*

{code}
struct DS_COMMITresok {
ds_verifier writeverf;
count4 count<>;
};

union DS_COMMITres switch (ds_status status) {
case DS_OK:
DS_COMMITresok resok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_NOSPC, DSERR_BAD_FH
{code}

h6.DS_LIST - List objects.

Get a list of objects from the DS matching the provided criteria.

*Description:*

Issued by the MDS asking the DS for a list of objects matching the provided criteria (either mds_dataset_id or MDS_SID)

This message is modeled after READDIR, and so MDS provides a starting offset cookie, and returning buffer-size parameters.

DS returns as many entries that will fit in the buffer-size parameters, starting from the provided cookie.

The information returned will be a counted array of filehandles and also the.

*Data Types:*

{code}
enum ds_list_type {
DS_LIST_MDS_SID,
DS_LIST_MDS_DATASET_ID
};
{code}

*Arguments:*

{code}
struct ds_list_sid_arg {
mds_sid mds_sid;
uint64_t cookie;
count4 maxcount;
};

struct ds_list_dataset_arg {
mds_dataset_id dataset_id;
uint64_t cookie;
count4 maxcount;
};

union DS_LISTargs switch (ds_list_type dla_type) {
case DS_LIST_MDS_SID:
ds_list_sid_arg sid;

case DS_LIST_MDS_DATASET_ID:
ds_list_dataset_arg dataset_id;

default:
void;
};

struct DS_LISTresok {
mds_ds_fh fh_list<>;
uint64_t cookie;
};
{code}

*Results:*

{code}
union DS_LISTres switch (ds_status status) {
case DS_OK:
DS_LISTresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_TOOSMALL, DSERR_NOENT
{code}

h6.DS_READ - Read an object.

Issued by the MDS to read from the DS for a given DS filehandle, for the provided data segments.

*Data Types:*

{code}
struct ds_fileseg {
offset4 offset;
count4 count;
};
{code}

*Arguments:*

{code}
struct DS_READargs
{
nfs_fh4 fh;
count4 count;
ds_fileseg rdv<>;
};
{code}

*Results:*

{code}
struct ds_filesegbuf
{
offset4 offset;
opaque data<>;
};

struct DS_READresok {
bool eof;
count4 count;
ds_filesegbuf rdv<>;
};

union DS_READres switch (ds_status status) {
case DS_OK:
DS_READresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_BAD_FH, DSERR_NOENT
{code}

h6.CTL_MDS_REMOVE - Remove.

From MDS to a DS, for the given filehandle(s) or MDS dataset id(s), remove the object(s).

*Description:*

Allows the MDS to remove one or more objects by filehandle or one or more MDS datasets from the DS.

*Arguments:*

{code}
enum ctl_mds_rm_type {
CTL_MDS_RM_OBJ,
CTL_MDS_RM_MDS_DATASET_ID
};


union CTL_MDS_REMOVEargs switch (ds_rm_type type) {
case CTL_MDS_RM_OBJ:
nfs_fh4 obj<>;

case CTL_MDS_RM_MDS_DATASET_ID:
mds_sid mds_sid;
mds_dataset_id dataset_id<>;

default:
void;
};
{code}

*Results:*

{code}
struct CTL_MDS_REMOVEres {
ds_status status;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_BAD_FH, DSERR_NOENT
{code}

h6.DS_WRITE - Write an Object.

From the MDS to a DS, for the given filehandle, write the data segments.

*Description:*

This procedure provides a way for the MDS to write file data segments to the DS. Most likely due to the NFS Client performing writes through the MDS, due to a network partition between the NFS Client and DS, or the MDS has recalled/revoked the layout and the client was unable to write/commit data to the DS.

The filehandle used in the arguments *MUST* be obtained from the layout allocation.

*Arguments:*

{code}
struct ds_filesegbuf {
offset4 offset;
opaque data<>;
};

struct DS_WRITEargs
{
nfs_fh4 fh;
stable_how4 stable;
count4 count;
ds_filesegbuf wrv<>;
};
{code}

*Results:*

{code}
struct DS_WRITEresok
{
stable_how4 committed;
ds_verifier writeverf;
count4 wrv<>;
};

union DS_WRITEres switch (ds_status status)
{
case DS_OK:
DS_WRITEresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_BAD_FH
{code}

h6.DS_STAT - Inquire on status of a data server object

From MDS to data sever to return information on the requested object.

*Arguments:*

{code}
struct DS_STATargs
{
nfs_fh4 object;
};
{code}

*Results:*

{code}
union DS_STATres switch (ds_status status)
{
case DS_OK:
ds_attr dattr;
default:
void;
};
{code}

h6.DS_GETATTR - Get attributes for a data server object

Issued by a MDS to a DS in order to query the attributes of a DS object.

*Description:*

This may be used when a MDS wants to verify the attributes (for example, size) provided by a client upon LAYOUTCOMMIT.

*Arguments:*

{code}
struct DS_GETATTRargs
{
nfs_fh4 fh;
ds_attr dattrs;
};
{code}

*Results:*

{code}
union DS_GETATTRres switch (ds_status status)
{
case DS_OK:
ds_attr dattrs;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DSERR_ATTR_NOTSUPP DSERR_NOENT
{code}

h6.DS_SETATTR - Set attributes for a data server object

Issued by a MDS to a DS to set attributes of a DS object.

*Description:*

This procedure allows a MDS to set attributes of the specified DS object. When an NFS Client truncates the file size down at the MDS, the MDS MUST issue a DS_SETATTR size to all the effected DSs. The DS_SETATTR messages can occur in parallel, however the MDS must not reply to the NFS Client until all the DS_SETATTRs have completed.

*Arguments:*

{code}
struct DS_SETATTRargs
{
nfs_fh4 fh;
ds_attr dattrs;
};
{code}

*Results:*

{code}
struct DS_SETATTRres
{
ds_status status;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DS_ATTR_NOTSUPP, DSERR_NOENT
{code}

h6.DS_SNAP - Snapshot a dataset on the DS that is related to the given MDS Dataset ID.

*Arguments:*

{code}
struct DS_SNAPargs
{
mds_dataset_id dataset_id;
};
{code}

*Results:*

{code}
struct DS_SNAPresok
{
utf8string name;
};

union DS_SNAPres switch (ds_status status)
{
case DS_OK:
DS_SNAPresok resok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOTSUPP, DSERR_INVAL
{code}

h6.DS_INVALIDATE - Invalidate state.

Issued by the MDS to a DS to invalidate state that the DS may hold.

*Description:*

The MDS may wish to invalidate state at the DS due to many factors. The DS_INVALIDATE message carries a type to indicate the scope of the invalidation, and the DS uses the type to interpret the associated parameter.

Examples of when the different invalidate types will be used are as follows:


* DS_INVALIDATE_ALL - Used on nfs/server service offlined on the MDS
* DS_INVALIDATE_CLIENTID - Used on client lease (with the MDS) expiration
* DS_INVALIDATE_LAYOUT_BY_CLIENT - Used on LAYOUTRETURN when invalidating all layouts held by a client
* DS_INVALIDATE_LAYOUT_BY_FH - Used on REMOVE issued from client to MDS
* DS_INVALIDATE_LAYOUT_BY_STATEID - Used on LAYOUTRETURN when invalidating a specific layout held by a client
* DS_INVALIDATE_MDS_DATASET_ID - Used on unshare of MDS FS
* DS_INVALIDATE_STATEID - Used on CLOSE, UNLOCK, DELEGRETURN issued from client to MDS

*Arguments:*

{code}
enum ds_invalidate_type
{
DS_INVALIDATE_ALL,
DS_INVALIDATE_CLIENTID,
DS_INVALIDATE_LAYOUT_BY_CLIENT,
DS_INVALIDATE_LAYOUT_BY_FH,
DS_INVALIDATE_LAYOUT_BY_STATEID,
DS_INVALIDATE_MDS_DATASET_ID,
DS_INVALIDATE_STATEID
};

struct ds_inval_layout_by_clid
{
clientid4 mds_clid;
nfs_fh4 fh;
};

struct ds_inval_stateid_lo_stateid
{
stateid4 stateid; /* MUST be layout stateid */
nfs_fh4 fh;
};

struct ds_inval_state_id
{
stateid4 stateid; /* MUST be open, lock or delegation stateid */
nfs_fh4 fh;
};

union DS_INVALIDATEargs switch (ds_invalidate_type obj)
{
case DS_INVALIDATE_ALL:
void;
case DS_INVALIDATE_CLIENTID:
clientid4 clid;
case DS_INVALIDATE_LAYOUT_BY_CLIENT:
ds_inval_layout_by_clid layout;
case DS_INVALIDATE_LAYOUT_BY_FH:
nfs_fh4 fh;
case DS_INVALIDATE_LAYOUT_BY_STATEID:
ds_inval_layout_by_lo_stateid stateid;
case DS_INVALIDATE_MDS_DATASET_ID:
mds_dataset_id dataset_id;
case DS_INVALIDATE_STATEID:
ds_inval_stateid stateid;
};

struct DS_INVALIDATEres
{
ds_status status;
};
{code}

*Results:*

{code}
struct DS_INVALIDATEres
{
ds_status status;
};
{code}

*Possible Error Codes:*

{code}
DSERR_NOT_AUTH, DSERR_INVAL, DSERR_NOENT
{code}

*Notes:*

_DS_INVALIDATE_ALL_

Invalidates all state at the data server. This is perhaps sent when the MDS knows it is in the process of an orderly shutdown.

_DS_INVALIDATE_LAYOUT_

To invalidate all layout segments held at the DS for this particular filehandle clientid combination. The MDS could use this message when a client returns the layout via OP_LAYOUTRETURN, or when the MDS has revoked the layout from the client.

_DS_INVALIDATE_MDS_DATASET_ID_

To invalidate all state for a MDS dataset id. The MDS would use this message for the un-share (or re-share) of a filesystem.

_DS_INVALIDATE_STATEID_

To invalidate state for the specified stateid. The MDS would use this when the client has issued either OP_CLOSE or OP_DELEGRETURN. The MDS could also use this message when it has revoked a delegation.

_DS_INVALIDATE_CLIENTID_

To invalidate all state for a particular clientid. The MDS would use this message once the client has lost it lease with the MDS, or the client has issues an explicit OP_DESTROY_CLIENTID.

h6.DS_PNFSSTAT - Collect kstats

This is a message from the MDS to DS to ask DS to report all pnfs/rpc kstats

The indices will be reported as nvpairs. The name will possibly be constructed from the corresponding kstat <class>_<name>_ <statistic>_< instance>

*Arguments:*

{code}
/* RPC kstats */
const DS_NFSSTAT_RPC = 0x000000001;

/* NFS kstats */
const DS_NFSSTAT_NFS = 0x000000002;

/* the DMOV protocol kstats */
const DS_NFSSTAT_DMOV = 0x000000004;

/* the control protocol kstats */
const DS_NFSSTAT_CP = 0x000000008;

/* CPU kstat (all stats for module cpu)*/
const DS_NFSSTAT_CPU = 0x000000010;

struct DS_PNFSSTATargs
{
uint64_t stat_wanted;
};
{code}

*Results:*

{code}
struct DS_PNFSSTATresok
{
opaque nvlist<>;
};

union DS_PNFSSTATres switch (ds_status status)
{
case DS_OK:
DS_PNFSSTATresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DSERR_NOTSUPP
{code}

h6.DS_CHANGE_DATASET_ID - Notify the data server that a change to the dataset ID on the MDS has been made (e.g. as a result of a zfs send/recv of a pNFS file system on the MDS).

*Arguments:*
{code}
struct DS_CHANGE_DATASET_IDargs
{
mds_sid mds_sid_owner;
mds_dataset_id old_dataset_id;
mds_dataset_id new_dataset_id;
};
{code}

*Results:*
{code}
struct DS_CHANGE_DATASET_IDres
{
ds_status status;
};
{code}

h6.DS_OBJ_MOVE_STATUS - To inquire on the status of a move

Since MDS supplies a task id for the data server move, it can inquire as to the status of the move.

*Arguments:*

{code}
struct DS_OBJ_MOVE_STATUSargs
{
uint64_t taskid;
};
{code}

*Results:*

{code}
struct DS_OBJ_MOVE_STATUSresok
{
uint64_t maxoffset;
bool complete;
};

union DS_OBJ_MOVEres switch (ds_status status) {
case DS_OK:
DS_OBJ_MOVEresok res_ok;
default:
void;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DSERR_NOTSUPP
{code}

h6.DS_OBJ_MOVE_ABORT - To stop an in-progress move.

For the given taskid, abort, abort, abort

*Arguments:*
{code}
struct DS_OBJ_MOVE_ABORTargs
{
uint64_t taskid;
};
{code}

*Results:*

{code}
struct DS_OBJ_MOVE_ABORTres
{
ds_status status;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DSERR_NOTSUPP, DSERR_NOMATCHING_TASKID
{code}

h6.DS_OBJ_MOVE - To initiate a data server to data server move

From the MDS to data server, a message to instruct a data server to move the files data blocks for a given FH.

The MDS supplies source and destination FH and target data server and taskid.

*Arguments:*

{code}
struct DS_OBJ_MOVEargs
{
uint64_t taskid;
nfs_fh4 source;
nfs_fh4 target;
netaddr4 targetserver;
};
{code}

*Results:*

{code}
struct DS_OBJ_MOVEres
{
ds_status status;
};
{code}

*Possible Error Codes:*

{code}
DSERR_INVAL, DSERR_NOTSUPP
{code}

h5.Object move/relocation: (PNFSCTLMV / 1040002)

The PNFSCTLMV protocol is exclusively for moving data from one data server to another. This would occur as the result of the MDS making a request (DS_OBJ_MOVE) via the PNFSCTLMDS protocol.

h6.MOVE - Move data from one data server to another.

*Arguments:*

{code}
struct MOVEargs {
uint64_t taskid;
uint64_t minoffset;
uint64_t maxoffset;
uint32_t maxbytes;
};
{code}

*Results:*

{code}
struct MOVEsegment {
uint64_t fileoffset;
uint32_t len;
};

struct MOVEres {
ds_status status;
opaque data<>;
struct MOVEsegment segments<>;
};
{code}

*Possible Error Codes:*
DSERR_NOENT

*Notes:*
The MOVE message is roughly equivalent to an NFS read, but with other properties. It is specially designed to deal with objects with holes.

MOVE is pull-based, rather than push-based. That is, the target data server contacts the source data server to ask it for data. NFS/RDMA has shown better performance with READ than WRITE. Also, pull-based avoids the problem of the stability of the data on the target server -- there will be no need for write verifiers and the like.

The disadvantage with pull-based data movement is that the source data server is the one that is aware of any holes in the source object. The dmov protocol reflects this. The result of a MOVE operation is a block of data, and an array of offsets and lengths. The arguments give a minimum and maximum file offset, which enables multiple MOVE operations to be in flight -- the target server can arrange for non-overlapping MOVEs.

h5. Error codes (ds_status)

The control protocol carries a status word in the reply to convey successful execution of the request or to report a problem. The status word uses an enumeration 'ds_status' and currently has the following assigned values.

- DS_OK
The request completed successfully.

- DSERR_NOENT
The item does not exist.

- DSERR_NOT_AUTH
The DS has been blocked from accessing the MDS through system administration.

- DSERR_INVAL
One or more of the arguments provided in the RPC Call are invalid.

- DSERR_IO
The server encountered a I/O error with the underlying data store.

- DSERR_ACCESS
During the DS_CHECKSTATE validation it was determined that the client does not have the appropriate access rights.

- DSERR_NOSPC
The MDS issued a DS_WRITE to a DS and the DS has no space left.

- DSERR_STALE_FH
The provided filehandle is stale.

- DSERR_BADHANDLE
The format of the filehandle was not recognized.

- DSERR_BAD_COOKIE
The MDS presented an invalid cookie to the DS in the DS_LIST message

- DSERR_NOTSUPP
The control protocol message you issued has yet to be implemented.

- DSERR_TOOSMALL
The DS_LIST results will not fit into the provided size.

- DSERR_EXPIRED

- DSERR_GRACE
The MDS is in the process of allowing clients to reclaim state after the MDS restarted

- DSERR_FHEXPIRED
The generation count within the filehandle has been incremented.

- DSERR_WRONGSEC
The security flavors for ds_checkstate do no match the required security flavor for the target object.

- DSERR_RESOURCE
The MDS has no available resource to complete the requested operation.

- DSERR_STALE_DSID
The ds_id used is not known to the MDS.

- DSERR_STALE_STATEID

- DSERR_SERVERFAULT
The data server encountered an unrecoverable error.

- DSERR_OLD_STATEID

- DSERR_BAD_MDSSID
The MDS was unable to provide a mapping for the requested MDS_SID.

- DSERR_PNFS_NO_LAYOUT
The client, identified by the co_owner member of the CHECKSTATE4args, that contacted the DS for I/O does not hold a layout.

- DSERR_NOMATCHING_TASKID
Returned from DS_MOVE_OBJ_STATUS and DS_MOVE_OBJ_ABORT if the taskid given is not recognized by the DS.

- DSERR_ATTR_NOTSUPP
Attribute is not supported.