sysinfo Provider

{anchor:top}
h1. {anchor:CHP-SYSINFO} {{sysinfo}} Provider
The {{sysinfo}} provider makes available probes that correspond to kernel statistics classified by the name {{sys}}. Because these statistics provide the input for system monitoring utilities like [{{mpstat}}(1M)|http://docs.sun.com/doc/819-2240/mpstat-1m?a=view], the {{sysinfo}} provider enables quick exploration of observed aberrant behavior.

[Top|#top]
h2. {anchor:CHP-SYSINFO-1} Probes
The {{sysinfo}} provider makes available probes that correspond to the fields in the {{sys}} named kernel statistic: a probe provided by {{sysinfo}} fires immediately before the corresponding {{sys}} value is incremented. The following example shows how to display both the names and the current values of the {{sys}} named kernel statistic using the [{{kstat}}(1M)|http://docs.sun.com/doc/819-2240/kstat-1m?a=view] command.
{noformat}
$ kstat -n sys
module: cpu instance: 0
name: sys class: misc
bawrite 123
bread 2899
bwrite 17995
cpu_ticks_idle 73743866
cpu_ticks_kernel 2096277
cpu_ticks_user 1010122
cpu_ticks_wait 46413
...
{noformat}
The {{sysinfo}} probes are described in [Table 23–1|#TBL-SYSINFO].
h6. {anchor:TBL-SYSINFO} {{sysinfo}} Probes
|{{bawrite}}|Probe that fires whenever a buffer is about to be asynchronously written out to a device.|
|{{bread}}|Probe that fires whenever a buffer is physically read from a device. {{bread}} fires _after_ the buffer has been requested from the device, but _before_ blocking pending its completion.|

|{{idlethread}}|Probe that fires whenever a CPU enters the idle loop.|
|{{intrblk}}|Probe that fires whenever an interrupt thread blocks.|
|{{inv_swtch}}|Probe that fires whenever a running thread is forced to involuntarily give up the CPU.|
|{{lread}}|Probe that fires whenever a buffer is logically read from a device.|
|{{lwrite}}|Probe that fires whenever a buffer is logically written to a device|
|{{modload}}|Probe that fires whenever a kernel module is loaded.|
|{{modunload}}|Probe that fires whenever a kernel module is unloaded.|
|{{msg}}|Probe that fires whenever a [{{msgsnd}}(2)|http://docs.sun.com/doc/819-2241/msgsnd-2?a=view] or [{{msgrcv}}(2)|http://docs.sun.com/doc/819-2241/msgrcv-2?a=view] system call is made, but before the message queue operations have been performed.|
|{{mutex_adenters}}|Probe that fires whenever an attempt is made to acquire an owned adaptive lock. If this probe fires, one of the {{lockstat}} provider's {{adaptive-block}} or {{adaptive-spin}} probes will also fire. See [Chapter 18, lockstat Provider|lockstat Provider] for details.|
|{{namei}}|Probe that fires whenever a name lookup is attempted in the filesystem.|
|{{nthreads}}|Probe that fires whenever a thread is created.|
|{{phread}}|Probe that fires whenever a raw I/O read is about to be performed.|
|{{phwrite}}|Probe that fires whenever a raw I/O write is about to be performed.|
|{{procovf}}|Probe that fires whenever a new process cannot be created because the system is out of process table entries.|
|{{pswitch}}|Probe that fires whenever a CPU switches from executing one thread to executing another.|
|{{readch}}|Probe that fires after each successful read, but before control is returned to the thread performing the read. A read may occur through the [{{read}}(2)|http://docs.sun.com/doc/819-2241/read-2?a=view], [{{readv}}(2)|http://docs.sun.com/doc/819-2241/readv-2?a=view] or [{{pread}}(2)|http://docs.sun.com/doc/819-2241/pread-2?a=view] system calls. {{arg0}} contains the number of bytes that were successfully read.|
|{{rw_rdfails}}|Probe that fires whenever an attempt is made to read-lock a readers/writer when the lock is either held by a writer, or desired by a writer. If this probe fires, the {{lockstat}} provider's {{rw-block}} probe will also fire. See [Chapter 18, lockstat Provider|lockstat Provider] for details.|
|{{rw_wrfails}}|Probe that fires whenever an attempt is made to write-lock a readers/writer lock when the lock is held either by some number of readers or by another writer. If this probe fires, the {{lockstat}} provider's {{rw-block}} probe will also fire. See [Chapter 18, lockstat Provider|lockstat Provider] for details.|
|{{sema}}|Probe that fires whenever a [{{semop}}(2)|http://docs.sun.com/doc/819-2241/semop-2?a=view] system call is made, but before any semaphore operations have been performed.|
|{{sysexec}}|Probe that fires whenever an [{{exec}}(2)|http://docs.sun.com/doc/819-2241/exec-2?a=view] system call is made.|
|{{sysfork}}|Probe that fires whenever a [{{fork}}(2)|http://docs.sun.com/doc/819-2241/fork-2?a=view] system call is made.|
|{{sysread}}|Probe that fires whenever a [{{read}}(2)|http://docs.sun.com/doc/819-2241/read-2?a=view], [{{readv}}(2)|http://docs.sun.com/doc/819-2241/readv-2?a=view], or [{{pread}}(2)|http://docs.sun.com/doc/819-2241/pread-2?a=view] system call is made.|
|{{sysvfork}}|Probe that fires whenever a [{{vfork}}(2)|http://docs.sun.com/doc/819-2241/vfork-2?a=view] system call is made.|
|{{syswrite}}|Probe that fires whenever a [{{write}}(2)|http://docs.sun.com/doc/819-2241/write-2?a=view], [{{writev}}(2)|http://docs.sun.com/doc/819-2241/writev-2?a=view], or [{{pwrite}}(2)|http://docs.sun.com/doc/819-2241/pwrite-2?a=view] system call is made.|
|{{trap}}|Probe that fires whenever a processor trap occurs. Note that some processors, in particular UltraSPARC variants, handle some light-weight traps through a mechanism that does not cause this probe to fire.|
|{{ufsdirblk}}|Probe that fires whenever a directory block is read from the UFS file system. See [{{ufs}}(7FS)|http://docs.sun.com/doc/819-2254/ufs-7fs?a=view] for details on UFS.|
|{{ufsiget}}|Probe that fires whenever an inode is retrieved. See [{{ufs}}(7FS)|http://docs.sun.com/doc/819-2254/ufs-7fs?a=view] for details on UFS.|
|{{ufsinopage}}|Probe that fires after an in-core inode _without_ any associated data pages has been made available for reuse. See [{{ufs}}(7FS)|http://docs.sun.com/doc/819-2254/ufs-7fs?a=view] for details on UFS.|
|{{ufsipage}}|Probe that fires after an in-core inode _with_ associated data pages has been made available for reuse. This probe fires after the associated data pages have been flushed to disk. See [{{ufs}}(7FS)|http://docs.sun.com/doc/819-2254/ufs-7fs?a=view] for details on UFS.|
|{{wait_ticks_io}}|Probe that fires when the periodic system clock has made the determination that a CPU is otherwise idle but some threads are waiting for I/O on the CPU. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The {{cpu_t}} argument ({{arg2}}) indicates the CPU that is described as waiting for I/O. See [Arguments|#CHP-SYSINFO-ARGS] for details on {{arg2}}. No semantic difference between {{wait_ticks_io}} and {{cpu_ticks_wait}}; {{wait_ticks_io}} exists solely for historical reasons.|
|{{writech}}|Probe that fires after each successful write, but before control is returned to the thread performing the write. A write may occur through the [{{write}}(2)|http://docs.sun.com/doc/819-2241/write-2?a=view], [{{writev}}(2)|http://docs.sun.com/doc/819-2241/writev-2?a=view] or [{{pwrite}}(2)|http://docs.sun.com/doc/819-2241/pwrite-2?a=view] system calls. {{arg0}} contains the number of bytes that were successfully written.|
|{{xcalls}}|Probe that fires whenever a cross-call is about to be made. A cross-call is the operating system's mechanism for one CPU to request immediate work of another CPU.|

[Top|#top]
h2. {anchor:CHP-SYSINFO-ARGS} Arguments
The arguments to {{sysinfo}} probes are as follows:
|{{arg0}}|The value by which the statistic is to be incremented. For most probes, this argument is always 1, but for some probes this argument may take other values.|
|{{arg1}}|A pointer to the current value of the statistic to be incremented. This value is a 64–bit quantity that will be incremented by the value in {{arg0}}. Dereferencing this pointer enables consumers to determine the current count of the statistic corresponding to the probe.|
|{{arg2}}|A pointer to the {{cpu_t}} structure that corresponds to the CPU on which the statistic is to be incremented. This structure is defined in {{<sys/cpuvar.h>}}, but it is part of the kernel implementation and should be considered Private.|
The value of {{arg0}} is 1 for most {{sysinfo}} probes. However, the {{readch}} and {{writech}} probes set {{arg0}} to the number of bytes read or written, respectively. This features permits you to determine the size of reads by executable name, as shown in the following example:
{noformat}
# dtrace -n readch'{@[execname] = quantize(arg0)}'
dtrace: description 'readch' matched 4 probes
^C
xclock
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
64 | 0

acroread
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
64 | 0

FvwmAuto
value ------------- Distribution ------------- count
2 | 0
4 |@@@@@@@@@@@@@ 13
8 |@@@@@@@@@@@@@@@@@@@@@ 21
16 |@@@@@ 5
32 | 0

xterm
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@ 19
64 |@@@@@@@@@ 7
128 |@@@@@@ 5
256 | 0

fvwm2
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@ 186
1 | 0
2 | 0
4 |@@ 51
8 | 17
16 | 0
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@ 503
64 | 9
128 | 0

Xsun
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@ 269
1 | 0
2 | 0
4 | 2
8 |@ 31
16 |@@@@@ 128
32 |@@@@@@@ 171
64 |@ 33
128 |@@@ 85
256 |@ 24
512 | 8
1024 | 21
2048 |@ 26
4096 | 21
8192 |@@@@ 94
16384 | 0
{noformat}
The {{sysinfo}} provider sets {{arg2}} to be a pointer to a {{cpu_t}}, a structure internal to the kernel implementation. Most {{sysinfo}} probes fire on the CPU on which the statistic is being incremented, but some probes do not. The exceptional probes include {{cpu_ticks_idle}}, {{cpu_ticks_kernel}}, {{cpu_ticks_user}} and {{cpu_ticks_wait}}, which always fire on the CPU executing the system clock. Use the {{cpu_id}} member of the {{cpu_t}} structure to determine the CPU of interest. The following D script runs for about ten seconds and gives a quick snapshot of relative CPU behavior on a statistic-by-statistic basis:
{noformat}
cpu_ticks_*
{
@[probename] = lquantize(((cpu_t *)arg2)->cpu_id, 0, 1024, 1);
}

tick-1sec
/x++ >= 10/
{
exit(0);
}
{noformat}
Running the above script results in output similar to the following example:
{noformat}
# dtrace -s ./tick.d
dtrace: script './tick.d' matched 5 probes
CPU ID FUNCTION:NAME
22 37588 :tick-1sec

cpu_ticks_user
value ------------- Distribution ------------- count
11 | 0
12 |@@@@@@@@ 14
13 |@@@@ 7
14 |@ 3
15 |@ 2
16 |@@ 4
17 |@@@@@@ 10
18 | 0
19 |@ 2
20 |@@@ 6
21 |@@@ 5
22 | 1
23 |@@@@@@ 10
24 | 0

cpu_ticks_wait
value ------------- Distribution ------------- count
11 | 0
12 |@@@@@@@@@@@@@ 241
13 |@@@@@@@@@@@@@ 236
14 | 16
15 |@@@@@@@ 132
16 | 11
17 | 10
18 | 7
19 |@ 18
20 | 4
21 | 16
22 | 13
23 | 10
24 | 0

cpu_ticks_kernel
value ------------- Distribution ------------- count
11 | 0
12 |@@@@@@@@ 234
13 |@@@@@ 159
14 |@@@ 104
15 |@@@@ 131
16 |@@ 66
17 |@ 40
18 |@ 51
19 |@ 36
20 |@@ 56
21 |@ 42
22 |@@@ 96
23 |@@ 57
24 | 0

cpu_ticks_idle
value ------------- Distribution ------------- count
11 | 0
12 |@@ 534
13 |@@ 621
14 |@@@ 900
15 |@@ 758
16 |@@@ 942
17 |@@@ 963
18 |@@@ 965
19 |@@@ 967
20 |@@@ 957
21 |@@@ 960
22 |@@@ 913
23 |@@@ 946
24 | 0
{noformat}

[Top|#top]
h2. {anchor:CHP-SYSINFO-6} Example
Examine the following output from [{{mpstat}}(1M)|http://docs.sun.com/doc/819-2240/mpstat-1m?a=view]:
{noformat}
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
12 90 22 5760 422 299 435 26 71 116 11 1372 5 19 17 60
13 46 18 4585 193 162 431 25 69 117 12 1039 3 17 14 66
14 33 13 3186 405 381 397 21 58 105 10 770 2 17 11 70
15 34 19 4769 109 78 417 23 57 115 13 962 3 14 14 69
16 74 16 4421 437 406 448 29 77 111 8 1020 4 23 14 59
17 51 15 4493 139 110 378 23 62 109 9 928 4 18 14 65
18 41 14 4204 494 468 360 23 56 102 9 849 4 17 12 68
19 37 14 4229 115 87 363 22 50 106 10 845 3 15 14 67
20 78 17 5170 200 169 456 26 69 108 9 1119 5 21 25 49
21 53 16 4817 78 51 394 22 56 106 9 978 4 17 22 57
22 32 13 3474 486 463 347 22 48 106 9 769 3 17 17 63
23 43 15 4572 59 34 361 21 46 102 10 947 4 15 22 59
{noformat}
From the above output, you might conclude that the {{xcal}} field seems too high, especially given the relative idleness of the system. {{mpstat}} determines the value in the {{xcal}} field by examining the {{xcalls}} field of the {{sys}} kernel statistic. This aberration can therefore be explored easily by enabling the {{xcalls}} {{sysinfo}} probe, as shown in the following example:
{noformat}
# dtrace -n xcalls'{@[execname] = count()}'
dtrace: description 'xcalls' matched 4 probes
^C
dtterm 1
nsrd 1
in.mpathd 2
top 3
lockd 4
java_vm 10
ksh 19
iCald.pl6+RPATH 28
nwadmin 30
fsflush 34
nsrindexd 45
in.rlogind 56
in.routed 100
dtrace 153
rpc.rstatd 246
imapd 377
sched 431
nfsd 1227
find 3767
{noformat}
The output shows where to look for the source of the cross-calls. Some number of [{{find}}(1)|http://docs.sun.com/doc/819-2239/find-1?a=view] processes are causing the majority of the cross-calls. The following D script can be used to understand the problem in further detail:
{noformat}
syscall:::entry
/execname == "find"/
{
self->syscall = probefunc;
self->insys = 1;
}

sysinfo:::xcalls
/execname == "find"/
{
@[self->insys ? self->syscall : "<none>"] = count();
}

syscall:::return
/self->insys/
{
self->insys = 0;
self->syscall = NULL;
}
{noformat}
This script uses the {{syscall}} provider to attribute cross-calls from {{find}} to a particular system call. Some cross-calls, such as those resulting from page faults, might not emanate from system calls. The script prints &ldquo;{{&lt;none>}}&rdquo; in these cases. Running the script results in output similar to the following example:
{noformat}
# dtrace -s ./find.d
dtrace: script './find.d' matched 444 probes
^C
<none> 2
lstat64 2433
getdents64 14873
{noformat}
This output indicates that the majority of cross-calls induced by {{find}} are in turn induced by [{{getdents}}(2)|http://docs.sun.com/doc/819-2241/getdents-2?a=view] system calls. Further exploration would depend on the direction you want to explore. If you want to understand why {{find}} processes are making calls to {{getdents}}, you could write a D script to aggregate on {{ustack}} when {{find}} induces a cross-call. If you want to understand why calls to {{getdents}} are inducing cross-calls, you could write a D script to aggregate on {{stack}} when {{find}} induces a cross-call. Whatever your next step, the presence of the {{xcalls}} probe has enabled you to quickly discover the root cause of the unusual monitoring output.

[Top|#top]
h2. {anchor:CHP-SYSINFO-STABILITY} Stability
The {{sysinfo}} provider uses DTrace's stability mechanism to describe its stabilities, as shown in the following table. For more information about the stability mechanism, see [Chapter&nbsp;39, Stability|Stability].
||Element||Name stability||Data stability||Dependency class||
|Provider|Evolving|Evolving|ISA|
|Module|Private|Private|Unknown|
|Function|Private|Private|Unknown|
|Name|Evolving|Evolving|ISA|
|Arguments|Private|Private|ISA|
{excerpt:hidden=true}Converted by tech dogg's sgml2wiki on Tue 20 Nov 2007 at 9:29:59 PM{excerpt}