CELESTESH COMMAND LINE INTERFACE

The celestesh command line utility is an interface to the low-level Celeste data store. This interface gives you basic capabilities to create and manipulate Celeste files, namespaces, and authentication/authorisation credentials at the lowest level.

The general form of the command line invocation is:

celestesh -help | [--celeste-address <ipaddress>:<port>] [--verbose] [operation]

Where --help causes a help message to be printed.

The optional parameter --celeste-address ipaddress:port specifies the address and tcp-port number of the Celeste node to contact to perform the specified operation (See ENVIRONMENT VARIABLES below). If --celeste-address is left unspecified, its value defaults to 127.0.0.1:14000. Any Celeste node is capable of performing the operation and multiple operations can be distributed among many nodes.

ENVIRONMENT VARIABLES

Several runtime parameters are controlled by environment variables:

CELESTESH OPERATIONS

Paramters to the following operations are positional. Unless otherwise specified, all parameters are encoded as UTF-8 strings. An object-id is always expressed as a 64 digit hexadecimal number. In cases where a null value may be supplied, it is specified by the string null.

create-file

Create a named file within a namespace, owned by an owner and group.

The specified namespace and the owner and group credentials must have already been created (see the new-namespace and new-credential operations).

• [--unsigned-writes | --signed-writes] Subsequent writes to this file include (in the case of --signed-writes or do not include (in the case of --unsigned-writes) signatures on the write operation that include the data written. By not signing the data (--unsigned-writes) write operations perform faster because the signature on the data is not computed nor is it checked. The risk for unsigned-writes is the potential for malicious nodes recording previous writes and then writing old data to the file at a later time. The default is --signed-writes

• requestor The name or the object-id of the credential of the requestor of this operation.

• password The password string to unlock the specified requestor credential.

• namespace The name or the object-id of the namespace to use when creating this file (See the operation new-namespace).

• password The password string to unlock the specified namespace.

• file-id The name or object-id of the file to create. File identifiers are unique within namespaces and must not already exist.

• owner The name or the object-id of the credential for the owner of this file. The credential must already exist (see the operation new-credential).

• group The name or the object-id of the credential of group for this file. The credential must already exist (see the operationnew-credential).

• deleteToken The secret string or object-id to encode with the created file's Celeste metadata to enable a subsequent deletion. (See Celeste File Deletion)

• replication-parameters The specification of the replication technique and parameters for the data and metadata that comprise the file (see Celeste Data Replication).

• blockObjectSize The decimal value of the maximum size of an individual block of file data (a Block Object). When data is written to this data file, it will be aggregated into blockObjectSize chunks. Choosing a large value improves read performance at the cost of increasing the load on the object store in storing large objects. If the Celeste system consists of nodes that cannot store blockObjectSize chunks, performance suffers and writes may fail entirely if a node is not available to store the chunk.

• timeToLive The decimal value of the number of seconds for this data file to exist.

$ celestesh create-file celestesh-id myPass celestesh-ns nspw celestesh-file \
celestesh-Owner celestesh-Group deleteMe \
'AObject.Replication.Store=3;VObject.Replication.Store=3;BObject.Replication.Store=3' \
8388608 86400

delete-file

Delete an existing file.

• requestor The name or the object-id of the credential to use for authorising this operation.<password>The password string to unlock the specified requestor credential.

• namespace The name or the object-id of the namespace used when this file was created. (See the operations new-namespace and create-file)

• file-id The name or object-id of the file to delete.

• deleteToken The secret delete-token that was previously set when this file was created. (See the create-file operation and Celeste File Deletion)

• timeToLive The decimal value of the number of seconds that the system should continue to look for undeleted components of the original file and delete them. Choosing this value is a function of the duration of time that nodes in the system are absent and return. The risk is that if this value is too small, a node the contains some part of a deleted data file may be absent at the time of deletion and then rejoin the Celeste system after the deletion time-to-live has gone to zero. In this case, the node will never be notified to delete the part of the original data file that it still has. Ultimately the part of the deleted file will be removed because of the time-to-live set when the file was created (see create-file). But this may be far into the future.

$celestesh delete-file myId myPass myNs myFile deleteMe 900

inspect-file

Obtain the Celeste-metadata and user-metadata for the specified file.

requestor The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the specified requestor credential.

namespace The name or the object-id of the file namespace specified when this file was created. (See the operations new-namespace and create-file)

file-name The name or object-id of the file to inspect.

version The object-id of the specific version of this file. The string "null" signifies the most recent version.

celeste-meta-data File into which to write the content of the Celeste metadata for this file. The string "-" signifies standard output.The results returned from this operation are still in flux. If you have specific requirements, or depend upon the formatting of the result, please notify the developers.

$ celestesh inspect-file myId myPass myNs myFile null clientMetaData celesteMetaData

lock-file

Place an advisory write lock on a file.

requestor The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the specified requestor credential.

namespace The name or the object-id of the file namespace specified when this file was created. (See the operations new-namespace and create-file)

file-name The name or object-id of the file to inspect.

celeste-meta-data File into which to write the content of the Celeste metadata for this file. The string "-" signifies standard output.The results returned from this operation are still in flux. If you have specific requirements, or depend upon the formatting of the result, please notify the developers.

new-credential

Create a new Celeste credential.

Credentials represent responsible entities (both internal and external to Celeste) and are used to determine authorisation, sign and attest to operations, and provide auditing.

name The name or specific object-id of the credential to create. The credential must not already exist.

password The password string to unlock the resulting credential.

replication-parameters The specification of the replication technique and parameters for the data and metadata that comprise the credential (See Celeste Data Replication).

$ celestesh new-credential myId myPass ErasureCodeReplica/3

new-namespace

Create a new namespace in the Celeste system.

name The name or specific object-id of the namespace to create. The namespace must not already exist.

password The password string to unlock the resulting namespace. Creators of files in this namespace must have this password in order to unlock the namespace.

replication-parameters The specification of the replication technique and parameters for the data and metadata that comprise the namespace metadata. (See Celeste Data Replication)

$ celestesh new-namespace myNs nspw ErasureCodeReplica/3

read-credential

Read a raw credential.

name The name or the object-id of the credential to read. (See the operation new-credential)

result-filename The local file name into which to store the result. The special name "-" signifies the standard output.

$ celestesh read-credential myClient result-filename

read-file

Read data from an existing data file.<requestor>The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the requestor's credential.

namespace The name or the object-id of the namespace used when this file was created. (See the operations new-namespace and create-file)

file-name The name or object-id of the file to read (see the operation create-file).

offset The decimal value of the starting position to read from.

length The decimal value of the number of bytes to read. If the value is -1, the entire contents of the file from the specified starting position to end of file are read and returned.

result-data The name of the local file into which to write the result. The special name "-" signifies the standard output.

result-metadata The name of the local file into which to write the result metadata. The special name "-" signifies the standard output.

$ celestesh read-file myId myPass myNs myFile 0 -1 -

read-file-version

Read data from a specific version of a data file. Each write to a given file creates a new version of that file and makes that version the current version; this command allows older versions to be accessed.

requestor The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the requestor's credential.

namespace The name or the object-id of the namespace used when this file was created (See the operations new-namespace and create-file).

file-name The name or object-id of the file to read (see the operation create-file).

version-Id The object-id of the specific version of this file. The string "null" signifies the most recent version (see the operation read-file).

offset The decimal value of the starting position to read from.

_ length_ The decimal value of the bumber of bytes to read.

result-filename The name of the local file into which to write the result. The special name "-" signifies the standard output.

result-metadata-filename The name of the local file into which to write the result metadata. The special name "-" signifies the standard output.

$ celestesh read-file-version myId myPass myNs myFile \
FA5541E46DF5C7ECF037F50E71661565D7B8270CD6A47E501467A648301B0DB4 0 -1 -

set-file-length

Set the length of an existing data file. The length may be shorter or longer than the amount of data currently in the file. Setting the length to a value greater than the amount of data stored in the file will cause subsequent reads of the segment of the data file beyond the length of the actual data to return bytes with a value of 0.

requestor The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the specified requestor credential.

namespace The name or the object-id of the namespace used when this file was created (See the operations new-namespace and create-file).

file-name The name or object-id of the file to read (see the operation create-file).

length The decimal value of the new length of the data file. Extending the length of the file beyond the natural length of the data is permitted.

$ celestesh set-file-length myId myPass myNs myFile 6

set-file-owner-and-group

Set the owner and group identities of an existing file. The file and the owner and group credentials must already exist. (See the operation new-credential)

requestor The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the specified requestor credential.

namespace The name or the object-id of the namespace used when this file was created (See the operations new-namespace and create-file).

file-name The name or object-id of the file to read (see the operation create-file).

owner The name or object-id of the credential of the new owner.

group The name or object-id of the credential of the new group.

$ celestesh set-file-owner-and-group myId myPass myNs myFile Owner Group

test

Test for the existence of a file or credential.

Exits with a zero status if the Credential exists, non-zero if it does not.

Exits with a zero status if the file exists, non-zero if it does not.

unlock-file

Remove a previously placed write lock on a file.

requestor The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the specified requestor credential.

namespace The name or the object-id of the file namespace specified when this file was created. (See the operations new-namespace and create-file)

file-name The name or object-id of the file to inspect.

celeste-meta-data File into which to write the content of the Celeste metadata for this file. The string "-" signifies standard output.The results returned from this operation are still in flux. If you have specific requirements, or depend upon the formatting of the result, please notify the developers.

write-file

Write data to an existing data file. This operation will write the entire contents of the input data to the Celeste data file starting at the specified offset. The file's length is automatically extended if the resulting write extends beyond the current end of the file. Note however, that a write that is less than the current length of the file does not truncate the file. Use set-file-length to explicitly set the length of the file if necessary.

requestor The name or the object-id of the credential to use for authorising this operation.

password The password string to unlock the specified requestor credential.

namespace The name or the object-id of the namespace used when this file was created. (See the operations new-namespace and {{create-file}])

file-name The name or object-id of the file to write (see the operation create-file).

offset The decimal value of the starting offset to write.

input-FileName The local file name of the data to write. The special name "-" signifies the standard input.

[result] Optional file name to store the result returned from this operation.

$ echo 'Hello World' | celestesh write-file myId myPass myNs myFile 0 -

PUTTING IT ALL TOGETHER

The example below is from the shell-script languages/bash/celestesh-example in the top-level directory of the Celeste installation.

$ celestesh new-namespace celestesh-ns nspw 'AObject.Replication.Store=3;VObject.Replication.Store=3;BObject.Replication.Store=3'
$ celestesh new-credential celestesh-Owner ownerpw \
  'AObject.Replication.Store=3;VObject.Replication.Store=3;BObject.Replication.Store=3'
$ celestesh new-credential celestesh-Group grouppw \
  'AObject.Replication.Store=3;VObject.Replication.Store=3;BObject.Replication.Store=3'
$ celestesh new-credential celestesh-id myPass \
  'AObject.Replication.Store=3;VObject.Replication.Store=3;BObject.Replication.Store=3'
$ celestesh create-file celestesh-id myPass celestesh-ns nspw celestesh-file \
  celestesh-Owner celestesh-Group deleteMe \
  'AObject.Replication.Store=3;VObject.Replication.Store=3;BObject.Replication.Store=3' 8388608 86400
$ celestesh set-file-owner-and-group \
  celestesh-id myPass celestesh-ns celestesh-file celestesh-Owner celestesh-Group
$ echo 'Hello World' | \
    celestesh write-file celestesh-id myPass celestesh-ns celestesh-file 0 -
$ celestesh read-file celestesh-id myPass celestesh-ns celestesh-file 0 -1 -
Hello World
$ echo '' | \
    celestesh write-file celestesh-id myPass celestesh-ns celestesh-file 5 -
$ celestesh read-file celestesh-id myPass celestesh-ns celestesh-file 0 -1 -
Hello
World
$ celestesh set-file-length celestesh-id myPass celestesh-ns celestesh-file 6
$ celestesh read-file celestesh-id myPass celestesh-ns celestesh-file 0 -1 -
Hello
$ celestesh inspect-file celestesh-id myPass celestesh-ns celestesh-file null clientMetaData celesteMetaData
$ cat celesteMetaData
#
#
Celeste.BObjectSize=8388608
Celeste.DeleteTokenId=707740A12E713CFF1658006C08FA429C5438E3128EDA7CACB17207D0298183B3
Celeste.FileSize=6
Celeste.GroupId=F8F299F5B3EA52D5CE61C22BDE40FCCAD45A7ED2A4A904431CDB3BD5BC42701C
Celeste.OwnerId=284072C57E661D9A2742D8F50E68DEB2B7313F3458697A6F7EA8F22E65848071
Celeste.ReplicationParameters={VObject.Replication.Store=3, AObject.Replication.Store=3, BObject.Replication.Store=3}
Celeste.SignData=true
Celeste.VObjectId=0FE6B786F99A430235234A017B092B91E752626AD7E3B90665489C1024CB6756
VObject.ClientMetaData=1\:
VObject.WritersSignature=E76A7E891200FA97809697D9D9F73AA85BCEBFE99F893E4F7AC1710F608A1868 SHA256withRSA 8049553758337738881320504909421465532413840983339336487670304210231643518150201258634764730908077926644553753687278955220481722340375772026814787198698833798700643262611320978365818039900057646627310921314938421044474293434832351300738723233822096574931749769182130260239720091852829148052405873775729512467091643909103550726963627539792593154915624506317645172147933830785663910073214182181817118139328929324238449484164375603215047582580388278067357356435481304314876293802503161405781445862415340414327947019557316123281274853059079429989759606723137595386917733867348899794364588273081582126825977467755468861952
$ celestesh set-file-length celestesh-id myPass celestesh-ns celestesh-file 10
$ celestesh read-file celestesh-id myPass celestesh-ns celestesh-file 0 -1 - | \
    cat -v
Hello
^@^@^@^@
$  echo 'Hello Again' | \
    celestesh write-file celestesh-id myPass celestesh-ns celestesh-file 0 -
$  celestesh read-file celestesh-id myPass celestesh-ns celestesh-file 0 -1 -
Hello Again
$  echo 'Do you like my hat?' | \
    celestesh write-file celestesh-id myPass celestesh-ns celestesh-file 0 -
$  celestesh read-file celestesh-id myPass celestesh-ns celestesh-file 0 -1 -
Do you like my hat?
$  celestesh delete-file celestesh-id myPass celestesh-ns celestesh-file deleteMe 900
$  celestesh read-file celestesh-id myPass celestesh-ns celestesh-file 0 -1 -
Not Found