|
SysAdmin Topics
|
Using on-the-fly decompression for UFS filesystemsby Bernd Schemmer, December 2008 Homepage: http://www.bnsmb.de/ Table of contents
IntroductionPurposeThis article describes the usage of fiocompress and dcfs to create compressed files on UFS filesystems. ScopeDue to the fact there is only a little information about fiocompress and dcfs, this article may contain wrong or incomplete information and should be read as draft documentation by a user of the tool. PrerequisitesSolaris 10 Update 6 for SPARC or a current release of OpenSolaris for SPARC Please be aware that these tools are marked with
in the man page. Referencesman pagesfiocompress(1M) man page dcfs (7FS) man page Source codeBlog entriesPorting OpenSolaris Project Indiana to SPARC - Part 2 Compression in Ramdisk - Dcfs (Moinak's blog) Installationdcfs is part of the Solaris package SUNWckr - Core Solaris Kernel (Root); fiocompress is part of the Solaris package SUNWcsr Core Solaris, (Root). These packages should be available on every Solaris 10 Update 6 (SPARC) or OpenSolaris installation. UsageThe program fiocompress(1M) can be used to compress or decompress files on UFS formatted filesystems. fiocompress uses the dcfs(7FS) filesystem which uses gzip to compress or decompress the files. Note: dcfs is a "layered filesystem" - you can not use it as primary filesystem. The man page for fiocompress says
But that's surely not the only useful usage of this tool. The usage syntax for fiocompress is /sbin/fiocompress -c [-m] [-b block_size] | -d {file}
The parameter -c is used to compress a file and the parameter -d is used to decompress a file (see fiocompress(1M) for a more detailed documentation for fiocompress). Example usage: [Fri Dec 05 22:15:07 root@sol9 /var/develop/fiocompress] # fiocompress -c SunTools.pkg SunTools.pkg.compressed [Fri Dec 05 22:15:25 root@sol9 /var/develop/fiocompress] # ls -l SunTools.pkg SunTools.pkg.compressed -rw-r--r-- 1 root root 7895552 Dec 5 22:14 SunTools.pkg -rw-r--r-- 1 root root 2030863 Dec 5 22:15 SunTools.pkg.compressed [Fri Dec 05 22:16:30 root@sol9 /var/develop/fiocompress] # file SunTools.pkg.compressed SunTools.pkg.compressed: data To decompress the file again use [Fri Dec 05 22:17:19 root@sol9 /var/develop/fiocompress] # fiocompress -d SunTools.pkg.compressed SunTools.pkg.decompressed [Fri Dec 05 22:17:26 root@sol9 /var/develop/fiocompress] # ls -l SunTools.pkg.compressed SunTools.pkg.decompressed SunTools.pkg -rw-r--r-- 1 root root 7895552 Dec 5 22:14 SunTools.pkg -rw-r--r-- 1 root root 2030863 Dec 5 22:15 SunTools.pkg.compressed -rw-r--r-- 1 root root 7895552 Dec 5 22:17 SunTools.pkg.decompressed [Fri Dec 05 22:17:44 root@sol9 /var/develop/fiocompress] # diff SunTools.pkg SunTools.pkg.decompressed So, that's not different to the usage of gzip or any other program to compress files - so why should you use fiocompress? Advanced UsageThe real advantage of fiocompress is hidden in the man page for fiocompress; there it says:
So, let's test this option: [Fri Dec 05 22:21:09 root@sol9 /var/develop/fiocompress] # fiocompress -m -c SunTools.pkg SunTools.pkg.compressed_with_m [Fri Dec 05 22:21:37 root@sol9 /var/develop/fiocompress] # ls -l SunTools.pkg SunTools.pkg.compressed_with_m -rw-r--r-- 1 root root 7895552 Dec 5 22:14 SunTools.pkg -rw-r--r-- 1 root root 7895552 Dec 5 22:21 SunTools.pkg.compressed_with_m [Fri Dec 05 22:21:47 root@sol9 /var/develop/fiocompress] # du SunTools.pkg SunTools.pkg.compressed_with_m 15440 SunTools.pkg 3984 SunTools.pkg.compressed_with_m [Fri Dec 05 22:21:53 root@sol9 /var/develop/fiocompress] # file SunTools.pkg.compressed_with_m SunTools.pkg.compressed_with_m: package datastream [Fri Dec 05 22:22:00 root@sol9 /var/develop/fiocompress] # diff SunTools.pkg.compressed_with_m SunTools.pkg Hopefully you see the difference: The dcfs filesystem adds a decompression on the fly for UFS formatted filesystems! That's a really cool feature. Using this feature you can compress your files and still use them without thinking about the compression also on UFS formatted filesystems. One disadvantage of fiocompress is that it only supports read-only operations, e.g. copying a file compressed with the -m option decompresses the file, e.g. [Fri Dec 05 22:22:09 root@sol9 /var/develop/fiocompress] # cp SunTools.pkg.compressed_with_m SunTools.pkg.compressed_with_m.copy [Fri Dec 05 22:24:25 root@sol9 /var/develop/fiocompress] # du -ks SunTools.pkg.compressed_with_m SunTools.pkg.compressed_with_m.copy 1992 SunTools.pkg.compressed_with_m 7720 SunTools.pkg.compressed_with_m.copy From another point of view this an advantage because you do not have to think about how to decompress a file ... And you can not edit or change a compressed file without first decompressing it. But for files that are only used in read-only mode fiocompress is really useful. ExamplesCreate a directory with compressed man pages# Create the directory structure # [Fri Dec 05 22:49:32 root@sol9 /var/develop/fiocompress] # mkdir man [Fri Dec 05 22:49:35 root@sol9 /var/develop/fiocompress] # cd /usr/share/man ; find . -type d | cpio -pdum /var/develop/fiocompress/man/ 0 blocks # copy and compress the files # [Fri Dec 05 22:50:51 root@sol9 /usr/share/man] # time for i in $( find . -type f ) ; do echo fiocompress -m -c $i /var/develop/fiocompress/man/$i ; done [Fri Dec 05 22:51:34 root@sol9 /usr/share/man] # time for i in $( find . -type f ) ; do fiocompress -m -c $i /var/develop/fiocompress/man/$i ; done real 3m16.741s user 0m37.202s sys 1m39.979s # check the results # [Fri Dec 05 23:03:49 root@sol9 /usr/share/man] # du -ks /usr/share/man /var/develop/fiocompress/man 58941 /usr/share/man 28890 /var/develop/fiocompress/man # and test the new man directory tree # [Fri Dec 05 23:04:40 root@sol9 /usr/share/man] # man -M /var/develop/fiocompress/man fiocompress Reformatting page. Please Wait... done System Administration Commands fiocompress(1M) NAME fiocompress - file compression utility SYNOPSIS /sbin/fiocompress -c [-m] [-b block_size] | -d {file} DESCRIPTION The fiocompress utility is a file compression tool that works together with the dcfs(7FS) file system to perform per-file compression. You can use fiocompress to decompress a compressed file or mark a compressed file as compressed, causing automatic decompression on read. The primary use of fiocompress is to compress files in the boot archive. Note that this utility is not a Committed interface. See attributes(5). OPTIONS The following options are supported: -b block_size Specify a block size for compression. The default block size is 8192.i -c Compress the specified file. -d Decompress the specified file. -m Mark the compressed file for automatic decompression on read. Can be used only in conjunction with -c. EXIT STATUS 0 Compressing ISO images# Compress the ISO image # [Sat Dec 06 09:44:58 root@sol9 /] # time fiocompress -m -c /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed real 1m23.849s user 1m15.220s sys 0m4.894s # check the result # [Sat Dec 06 09:47:05 root@sol9 /] # ls -l [Sat Dec 06 09:55:16 root@sol9 /] # ls -l /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed -rw-rw-r-- 1 root root 426311680 Apr 5 2008 /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso -rw-r--r-- 1 root root 426311680 Dec 6 09:47 /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed [Sat Dec 06 09:55:23 root@sol9 /] # du -ks /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed 416536 /export/iso/Solaris10/sol-10-u5-doc-ga-dvd.iso 305624 /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed # Use the compressed ISO image # [Sat Dec 06 09:55:28 root@sol9 /] # lofiadm -a /var/daten/sol-10-u5-doc-ga-dvd.iso.compressed /dev/lofi/24 [Sat Dec 06 09:56:01 root@sol9 /] # mkdir /tmp/sol10u5 [Sat Dec 06 09:56:07 root@sol9 /] # mount -o ro -F hsfs /dev/lofi/24 /tmp/sol10u5 [Sat Dec 06 09:56:24 root@sol9 /] # ls /tmp/sol10u5 Copyright css graphics index.html pdf Troubleshootingfiocompress does not preserve the attributes of the compressed filesThis is a bug in the current version of fiocompress; see the blog entry Porting OpenSolaris Project Indiana to SPARC - Part 2 for a workaround for this bug. The bugid for this bug is 6724205.
Size restrictionThere seems to be a size restriction for fiocompress or dcfs, example: [Sat Dec 06 10:01:51 root@sol9 /] # ls -l /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso -rw-r--r-- 1 root root 2661416960 Oct 28 20:12 /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso [Sat Dec 06 10:01:52 root@sol9 /] # du -h /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso 2.5G /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso [Sat Dec 06 10:02:23 root@sol9 /] # fiocompress -m -c /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso /var/daten/sol-10-u6-ga1-sparc-dvd.iso fiocompress: cannot open /export/iso/Solaris10/sol-10-u6-ga1-sparc-dvd.iso - Value too large for defined data type |
Sign up or Log in to add a comment or watch this page.