Using "lsof" in the Real World

Using "lsof" in the Real World

by Chris Harrison, October 2008

lsof ("LiSt Open Files") is a tool to list open file handles on a server. Being able to filter these handles using the range of options and switches for lsof allows the Solaris administrator to diagnose a number of problems.

This guide explains a couple of the command-line options to lsof, and then shows several examples of how lsof can be used in the "real world". (Note: This should work on all Solaris versions from 8 onwards; it might work on earlier versions, but I've not tried it.)

Obtaining "lsof"

lsof does not come standard on a default Solaris install, but compiled Solaris packages for SPARC and x86 versions are available from http://sunfreeware.com.

Examining Open Network Ports

Syntax:

lsof -i [<protocol>][@<interface name>][:<port>]

You can specify some or all of the three parameters shown above. Any that are omitted are treated as wildcards. <port> can be either a single port or a range. Using lsof like this is similar to the behavior of netstat -a but also provides the process and owner information.

Listing the File Handles a Given Process Has Open

Syntax:

lsof -p <process id>[,<process id>[,<process id>[...]]]

You can specify one or more process IDs (comma separated) to have lsof show all associated open file handles. This can be useful when trying to determine which libraries a particular running job is using. Doing this can also give you an indication as to how the process interacts with its log and configuration files.

Listing the File Handles a Given User Has Open

Syntax:

lsof -u <username>[,<username>[,<username>[...]]]

This command will show all open handles belonging to the given user(s).

Finding All the Open File Handles on a Given File System

Syntax:

lsof <file system or device>

This option is a considerable help locating the processes that are preventing a umount command from completing, or tracking down an open file with no corresponding directory entry (see the examples below).

Some Other Useful Flags

• -b – Avoid using system calls that could cause blocking on the kernel
• -w – Suppress warning messages
• -P – Don't map network port numbers in /etc/services to their service names
• -a – Place this between two other flags to have lsof list only the handles that match both flags. By default, lsof will list handles that match either of the flags specified.

Examples

List all the ports that inetd is listening on:
Find out exactly what is open via inetd right now.

gouda# ps -ef | grep inetd

root  2849     1  0   May 25 ?        6:43 /usr/sbin/inetd -s
gouda# lsof -i -a -p 2849
COMMAND  PID USER   FD   TYPE        DEVICE SIZE/OFF NODE NAME
inetd   2849 root   11u  IPv6 0x30006ec96d0      0t0  TCP *:ftp(LISTEN)
inetd   2849 root   13u  IPv4 0x30006ec9550      0t0  UDP *:name(Idle)
inetd   2849 root   16u  IPv4 0x30008f26ad8      0t0  TCP *:52783(LISTEN)
inetd   2849 root   17u  IPv4 0x30006ec9b50      0t0  UDP *:57504(Idle)
inetd   2849 root   23u  IPv6 0x30008f264d8      0t0  UDP *:servicetag(Idle)
inetd   2849 root   24u  IPv4 0x30008f26358      0t0  TCP *:servicetag(LISTEN) 

First we identify the process ID of inetd, then we have lsof list all IP handles (-i) that also (-a) are opened by that port (-p <process id>).

List all the currently open network connections for a particular user:
To trace a possible security breach in progress, or to verify that our application is connecting to the correct database server, lsof can show us a given user's current network connections.

gouda# lsof -i -a -u fred
COMMAND PID USER FD  TYPE DEVICE SIZE NODE NAME
sshd   9696 fred  5u IPv4 0x3001 0t527 TCP gouda:22->edam:46528(ESTABLISHED)
sshd   9696 fred 10u IPv4 0x300e   0t0 TCP localhost:6011(LISTEN)
sshd  14710 fred  5u IPv4 0x3008 0t404 TCP gouda:22->edam:46590(ESTABLISHED)
sshd  14710 fred 10u IPv4 0x307d   0t0 TCP localhost:6012(LISTEN)
sshd  17013 fred  5u IPv4 0x300f 0t146 TCP gouda:22->edam:50019(ESTABLISHED)
sshd  17013 fred 10u IPv4 0x30e0   0t0 TCP localhost:6010(LISTEN)
ssh   25259 fred  4u IPv4 0x3000 0t157 TCP gouda:54435->colby:22(ESTABLISHED)
sshd  25383 fred  5u IPv4 0x3000 0t126 TCP gouda:22->brie:34480(ESTABLISHED)
ssh   25438 fred  4u IPv4 0x3000  0t91 TCP gouda:54448->smokey:22(ESTABLISHED)

This shows us that fred has three incoming ssh connections and has opened two more outgoing ssh connections. Running lsof on one of the other servers, looking for the originating port (for example, port 46528 on edam), will allow us to continue following the trail from there.

Find what process is preventing a particular file system from unmounting:
A stray process can prevent a umount command from succeeding. Rather than use the -f flag and potentially cause corruptions, lsof can show you which processes need to be stopped first.

gouda# lsof /mnt/scratchdisk
COMMAND   PID USER    FD  TYPE DEVICE SIZE/OFF NODE NAME
bash     7268 fred    cwd VDIR 85,204      512    2 /mnt/scratchdisk/edam
dcm     15375 george  txt VREG 85,204   292352  202 /mnt/scratchdisk/bin/dcm
dcm     15375 george  cwd VDIR 85,204      512    5 /mnt/scratchdisk/bin/dcm

This shows that fred has a bash shell whose current working directory is on our mount point, and george has started a copy of dcm from another directory on the same file system.

Find what process is listening on a given port and who's connecting to it:
It can often be highly frustrating to the administrator when a daemon fails to start, displaying the error port already in use. lsof can be used to track down what process is holding the port we want to use.

gouda# lsof -i :7507
COMMAND  PID   USER FD  TYPE  DEVICE  SIZE NODE NAME
inetd    668   root 34u IPv4 0x303f0   0t0  TCP *: 7507 (LISTEN)
answer 19757 nobody  0u IPv4 0x58028 0t132  TCP gouda:7507->edam:46111 (ESTABLISHED)
answer 19757 nobody  1u IPv4 0x58028 0t132  TCP gouda:7507->edam:46111 (ESTABLISHED)
answer 19757 nobody  2u IPv4 0x58028 0t132  TCP gouda:7507->edam:46111 (ESTABLISHED)

This shows that inetd is running a service called answer on that port and that something running on edam is connecting to it. Running lsof -i :46111 on edam will show which process this is.

Locate the process that has a file open with no corresponding directory entry:
If someone has deleted a file that a running process has open, the file system will not actually remove it until the last link to it has closed. In this case, the running process is the only thing left. If we can find the correct process, we can either shut it down (if we want the file gone) or save the contents of the file via the /proc file system (Solaris 8 onwards) if we need to salvage it.

gouda # df -k /opt/myapp/data
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d100     4129290 1458133 2629865    36%    /opt/myapp

gouda # lsof /opt/myapp | grep '/dev/md/dsk/d100'
less      15043 fred    4r  VREG  102,0       144 2050050 /opt/myapp (/dev/md/dsk/d100)

This shows us that the file is being held open by a copy of less (process ID 15043). If the goal is to get rid of the file, ending the less process will achieve this. If the goal is to salvage the contents of the file, though, it is still available to be copied out. The 4th column shows a (r)ead-only handle on file descriptor 4. This means the file is available for reading from /proc/15043/fd/4.