Identifying disk I/O-bound systems

A system is I/O bound, or has an I/O bottleneck, if the peripheral devices (hard disk, tape, and so on) cannot transfer data as fast as the system requests it. This causes processes to be put to sleep, ``waiting for I/O'', and leaves the CPU(s) idle for much of the time. To determine if the system is disk I/O bound run sar -u (or cpusar -u for SMP) and look at the %wio value. This displays the percentage of time that each CPU spends waiting for I/O to complete while there are no runnable processes. If this value is high then it is possible that I/O is not keeping up with the rest of the system. (You should not always assume that there is a problem with disks; for example, %wio might be high because a tape drive is being accessed.) Other indications of a disk I/O bottleneck can be seen using sar -d (or mpsar -d for SMP). Note that sar -d can be also be used to view the activity of block I/O devices including hard disk drives, SCSI tape drives, and floppy disks.

If the values for %busy and avque are both consistently high then the devices cannot keep up with the requests to transfer data. Devices such as floppy disks and some older types of tape drive are inherently slow. As these devices are generally infrequently used -- for system backup, software installation, and so on -- there is little that performance tuning can usefully accomplish.

The value of blks/s displayed by sar -d can be combined with %busy to give an indication of the maximum I/O throughput of a disk, and may suggest where a I/O bottleneck can occur:

Maximum disk throughput (KB/s) = blks/s 50 / %busy

High values of the ratio of avwait to avserv also suggest that the device is saturated with requests.

If the number of transfers, r+w/s, is high but the amount of data being transferred, blks/s, is low, it may be possible to modify the application to transfer larger amounts of data less frequently. This should reduce the number of requests for the disk and reduce contention for it.

The read and write hit rates (%rcache and %wcache) shown by sar -b should show high values. If these values fall, the system is having to access blocks on disk (or other block devices) rather than in the buffer cache. If this happens, increasing the size of the buffer cache may help to alleviate a disk I/O bottleneck.

A low hit rate for the namei cache could lead to the disk being accessed more often in order to convert pathname components to inode numbers. If sar -n displays results showing that hit % is consistently low then the namei cache for the corresponding filesystem type is too small. It is not possible to give a general definition of what is a low value since this depends on the application mix that you run on your system. Because the performance of the namei cache does not depend linearly on its size, you will find that improving cache hit rates that are already high requires a significantly greater cache size. For example, you might have to increase the size of the namei cache by 30% in order to increase the namei cache hit rate from 90% to 95% giving a relative gain of 5.6%.

Note that namei caching is only performed for pathname components that are 14 characters or less in length. To take advantage of the namei cache, you should use directory names and filenames that are less than 15 characters long.

The following table is a summary of the commands that can be used to determine if a system is I/O bound:

Identifying an I/O-bound system

Command	Field	Description
[cpu]sar -u	%wio	percent of time that the CPU spends waiting for I/O with no processes to run
[mp]sar -b	bread/s	average number of blocks read into the buffer cache per second
	bwrit/s	average number of blocks written from the buffer cache per second
	%rcache	percent of read disk block requests satisfied by reading the buffer cache
	%wcache	percent of write disk block requests satisfied by writing to the buffer cache
[mp]sar -d	avque	average number of requests on queue waiting for device per second
	avserv	average time transfer takes to complete in milliseconds
	blks/s	number of 512 byte blocks transferred to and from the device per second
	%busy	percent of time device was servicing transfer request
	r+w/s	number of read/write transfers to device per second
[mp]sar -n	hit %	namei cache hit rate as a percentage