Using a SVM submirror on a ramdisk to increase read performance

Version 1 by bnsmb
on Nov 17, 2008 13:10.
compared with
Current by bnsmb
on Nov 08, 2009 05:31.
Key
This line was removed.
This word was removed. This word was added.
This line was added.

Changes (92)

View page history
!http://www.sun.com/bigadmin/home/images/bigadminHeaderWikiThumb.jpg!
{section:border=false}
{column:width=20%}
{include:TOC for Tech Tips}
{column}
{column:width=55%}

h1. Using a SVM submirror on a ramdisk to increase read performance

by Bernd Schemmer, November 2008
Table of Contents
Homepage: [http://www.bnsmb.de/]

*Table of Contents*
{toc}
...

First create the SVM mirror on the disks in the normal way, e.g.:
{code}
{code}bash-3.00# metainit d11 1 1 c0d1s0
d11: Concat/Stripe is setup
...
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d10       249127    1041  223174     1%    /mnt
{code}
{code}The next step is to create the  ramdisk for the 3rd submirror using ramdiskadm:
{code}
{code}bash-3.00# ramdiskadm -a testdisk 230m
/dev/ramdisk/testdisk
{code}
{code}{*}Note:* The ramdisk must be as big as the other submirrors.

Due to a restriction in the SVM commands (metainit, metastat, etc) you must create symbolic links for the ramdisk in the /dev/dsk and /dev/rdsk directories:
{code}
{code}bash-3.00# ln -s /dev/ramdisk/testdisk /dev/dsk/c5t0d0s0

bash-3.00# ln -s /dev/rramdisk/testdisk /dev/rdsk/c5t0d0s0
{code}
{code}{*}Note:* Use a control number (c5 in this example) which is not already in use.

Now create a metadevice on the ramdisk
{code}
{code}bash-3.00# metainit d13 1 1 c5t0d0s0
d13: Concat/Stripe is setup
...
Device   Reloc  Device ID
c5t0d0   No     -
{code}
{code}And attach the new metadevice to the existing mirror:
{code}
{code}bash-3.00# metattach d10 d13
d10: submirror d13 is attached
...
d12 1 1 c0d1s0
d13 1 1 c5t0d0s0
{code}
{code}That's it \-\- now  we have a SVM mirror with one submirror on a ramdisk.

But as can be seen in the iostat output below this configuration not really improves the read performance of the mirror. For this we must  change the read policy for the mirror.

The default read policy for SVM mirror devices is round robin:
{code}
{code}bash-3.00# metaparam d10
d10: Mirror current parameters are:
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
{code}
{code}We want to change it , so that reads are done from our ramdisk based submirror. Unfortunately there is no option to select a specific submirror as primary submirror for read requests:
{code}
{code}bash-3.00# metaparam
usage:  metaparam [-s setname] [options] concat/stripe | RAID
        metaparam [-s setname] [options] mirror
...
-w      parallel | serial
-p      0-9
{code}
{code}Therefore we must first destroy and recreate the mirror (without loosing the data, of course):

Stop all applications accessing the mirror and ensure that all three submirrors are in sync:
{code}
{code}bash-3.00#  metastat | grep sync
{code}
{code}Now umount the mirror and clear the SVM metadevice for the mirror
{code}
{code}bash-3.00#  umount /mnt

bash-3.00#  metaclear d10
{code}
{code}and recreate the mirror using the ramdisk as first submirror:
{code}
{code}bash-3.00#  metainit d10 -m d13

bash-3.00#  metattach d10 d12
...
d11 1 1 c1d0s0
d12 1 1 c0d1s0
{code}
{code}So now we can change the read policy for the mirror to "first":
{code}
{code}bash-3.00# metaparam -r first d10

bash-3.00# metaparam d10
...
    Read option: first (-r)
    Write option: parallel (default)
{code}
{code}This configuration now improves the read performance for the mirror a lot  (see the iostat output below).

h2. What about write performance?
...

*Writing to a mirror with two submirrors on disk*
{noformat}
{noformat}bash-3.00# time dd if=/dev/zero of=/mnt/testfile bs=512
write: No space left on device
429777+0 records in
...
user 0m0.249s
sys 0m23.197s
{noformat}
{noformat}{*}Writing to a mirror with two submirrors on disk and one submirror on a ramdisk:*
{noformat}
{noformat}bash-3.00# time dd if=/dev/zero of=/mnt/testfile bs=512
write: No space left on device
429777+0 records in
...

After the machine is rebooted, the ramdisk does not exist and therefore the submirror on the ramdisk also does not exist and the mirror needs maintenance:
{code}
# {code}# bash
bash-3.00# metastat
d10: Mirror
...
c5t0d0   No     -
c0d1     Yes    id1,cmdk@AQEMU_HARDDISK=QM00002
{code}
{code}This is not a big problem because the data still exists on the existing submirrors and the mirror also works without the 3rd submirror - but without the read performance improvement, of course.

To repair the mirror recreate the ramdisk
{code}
{code}bash-3.00# ramdiskadm -a testdisk 230m
{code}
{code}and reattach the submirror on the ramdisk
{code}
{code}bash-3.00# metareplace -e d10 c5t0d0s0
d10: device c5t0d0s0 is enabled
...
c0d1     Yes    id1,cmdk@AQEMU_HARDDISK=QM00002
c1d0     Yes    id1,cmdk@AQEMU_HARDDISK=QM00003
{code}
{code}That's it - now the 3rd submirror on the ramdisk is again in place and after the synchronisation is done we have a mirror with faster read performance again.

h1. Performance tests

h2. In a virtual QEMU machine:

h3. SVM Mirror configuration
h3. Configuring SVM Mirror
 
{noformat}
{noformat}bash-3.00# metastat -p d10
d10 -m d11 d12 d13 1
d11 1 1 c1d0s0
d12 1 1 c0d1s0
d13 1 1 c5t0d0s0
{noformat}
{noformat} d10 is the mirror, d11 and d12 are the sub mirrors on two virtual harddisks, and d13 is the submirror on the ramdisk.

h3. Using a mirror with two disk based submirrors

{anchor:testvirt000}
{noformat}
{noformat}bash-3.00# dd if=/dev/md/rdsk/d10 of=/dev/null bs=512
    
bash-3.00#  iostat -xn 1 5555
...

{anchor:testvirt001}
{noformat}
{noformat}bash-3.00# dd if=/dev/md/rdsk/d10 of=/dev/null bs=512

bash-3.00#  iostat -xn 1 5555
...

{anchor:testvirt000}
{noformat}
{noformat}bash-3.00# dd if=/dev/md/rdsk/d10 of=/dev/null bs=512

bash-3.00#  iostat -xn 1 5555
...
{noformat}
h2. On a physical machine with SCSI disks:

h3. SVM Mirror configuration
h3. Configuring SVM Mirror
 
{noformat}
noname:/# {noformat}noname:/# metastat -p d80
d80 -m d81 d82 d83 1
d81 1 1 c3t2d0s0
d82 1 1 c3t3d0s0
d83 1 1 c7t0d0s0
{noformat}
{noformat}d80 is the mirror, d81 and d82 are the submirrors on two harddisks, and d83 is the submirror on the ramdisk


h3. Using a mirror with two disk based submirrors
{noformat}
noame:/var# {noformat}noame:/var# time dd if=/dev/md/rdsk/d80 of=/dev/null bs=512
522000+0 records in
522000+0 records out
...
h3. Using a mirror with two disk based submirrors and one ramdisk based submirror with the default read policy
{noformat}
noname:/var# {noformat}noname:/var# time dd if=/dev/md/rdsk/d80 of=/dev/null bs=512
522000+0 records in
522000+0 records out
...
h3. Using a mirror with two disk based submirrors and one ramdisk based submirror and the changed read-policy
{noformat}
noname:/var# {noformat}noname:/var# time dd if=/dev/md/rdsk/d80 of=/dev/null bs=512
522000+0 records in
522000+0 records out
...

So it's possible to use this approach but it's only useful if an application really needs it and it's not possible to get the necessary read performance with other methods (like tuning the cache, for example).


{column}
{section}

The individuals who post here are part of the extended Sun Microsystems community and they might not be employed or in any way formally affiliated with Sun Microsystems. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Sun nor any other party necessarily agrees with them.

Copyright 1994-2009 Sun Microsystems, Inc.
Powered by Atlassian Confluence Sun Guidelines on Public Discourse Privacy Policy Terms of Use Trademarks Site Map Employment Investor Relations Contact