How to find the process causing IO wait?

To identify if there is any IO wait happening in the system, run the top command

# top


Cpu(s):  0.2%us,  0.5%sy,  0.0%ni, 98.8%id,  0.5%wa,  0.0%hi,  0.0%si,  0.0%st

The %wa tells if there is an IO wait.

How to identify the process causing high IO wait?

1) Using iotop command - iotop watches I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O usage by processes or threads on the system.

# iotop

Total DISK READ: 0.00 B/s | Total DISK WRITE: 99.97 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  361 be/3 root        0.00 B/s    7.69 K/s  0.00 %  4.20 % [jbd2/sda2-8]
 6673 be/4 root        0.00 B/s  315.30 K/s  0.00 %  0.00 % wget -c http://mirror.steadfast.net/centos/6.3/isos/i386/CentOS-6.3-i386-bin-DVD1.iso

In case if iotop command is not available, one can try dstat and pidstat commands

2) Using dstat command - dstat is a versatile replacement for vmstat, iostat and ifstat. dstat overcomes some of the limitations and adds some extra features.

#  dstat -ta --top-bio

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ----most-expensive----
  date/time   |usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw |  block i/o process
27-01 01:10:09|  0   0  99   0   0   0|  15k   27k|   0     0 |   0     0 |  81   124 |bash         11B   17k
27-01 01:10:10|  1   1  99   0   0   0|   0     0 | 187k  188k|   0     0 | 307   439 |wget          0   168k
27-01 01:10:11|  1   1  97   1   0   0|   0    12k| 179k  179k|   0     0 | 289   422 |wget          0   176k
27-01 01:10:12|  1   1  98   0   0   0|   0     0 | 195k  196k|   0     0 | 330   445 |wget          0   180k
27-01 01:10:13|  1   1  98   0   0   0|   0     0 | 177k  178k|   0     0 | 274   383 |wget          0   164k
27-01 01:10:14|  1   1  98   0   0   0|   0     0 | 189k  190k|   0     0 | 302   412 |wget          0   172k
27-01 01:10:15|  1   1  98   0   0   0|   0     0 | 189k  190k|   0     0 | 307   440 |wget          0   188k
27-01 01:10:16|  1   1  98   1   0   0|   0    12k| 188k  188k|   0     0 | 291   427 |wget          0   168k

3) Using pidstat command - pidstat command is provided by sysstat package


# pidstat -d 2
Linux 2.6.32-042stab068.8 (dhcppc5)     01/27/2013      _x86_64_        (2 CPU)

12:39:02 AM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
12:39:04 AM       361      0.00      9.90      0.00  jbd2/sda2-8
12:39:04 AM       563      0.00      1.98      0.00  flush-8:0
12:39:04 AM      1466      0.00      3.96      0.00  java
12:39:04 AM      6519      0.00    178.22      0.00  wget

So from the above commands we understand that the processes  jbd2/sda2-8 and wget are causing IO wait.