Disk IO

KUPI

New Member
Joined
Aug 30, 2019
Messages
27
Reaction score
5
Credits
192
Could someone guide me how to check disk IO during performance issue. I have got some idea to check the memory & cpu utilization.
But I'm not very sure how to check disk IO.
I ran iostat -x command. But I'm not sure which part I need to look in iostat output.
do we have any formula, to calculate the disk IO values. Upto what value, we can consider disk IO is normal.
If disk IO found high, what steps we need to take to improve the performance. (If at all no issues found in filesystem or disk).

For example, I have given the below output of iostat -x. Could someone guide me to learn about troubleshooting IO.

avg-cpu: %user %nice %system %iowait %steal %idle
1.61 0.14 0.65 5.32 0.00 92.28

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sdb 28.26 905.20 244.46 783.11 60380.59 19881.09 78.11 1.03 1.00 0.60 61.21
sda 0.10 4.55 0.78 2.53 73.40 56.62 39.34 0.00 0.87 0.58 0.19
dm-0 0.00 0.00 0.64 2.59 31.99 20.71 16.33 0.04 11.72 0.17 0.06
dm-1 0.00 0.00 0.01 0.00 0.07 0.00 8.00 0.00 0.13 0.13 0.00
dm-2 0.00 0.00 0.21 4.42 40.34 35.37 16.35 0.01 2.97 0.29 0.13
dm-3 0.00 0.00 0.45 0.08 74.92 0.60 143.86 0.00 3.34 0.10 0.01
dm-4 0.00 0.00 267.01 280.13 56234.29 5587.78 112.99 0.72 1.31 0.12 6.31
dm-5 0.00 0.00 0.08 0.13 4.07 1.05 24.09 0.00 0.42 0.17 0.00
dm-6 0.00 0.00 1.48 399.61 1269.71 3196.91 11.14 15.90 39.60 0.01 0.25
dm-7 0.00 0.00 0.09 474.21 1.01 5305.62 11.19 1.34 2.82 0.59 28.03
dm-8 0.00 0.00 3.17 474.22 2781.71 5305.63 16.94 1.35 2.83 0.59 28.31
dm-9 0.00 0.00 0.01 0.06 0.06 0.46 8.00 0.00 0.23 0.09 0.00
dm-10 0.00 0.00 0.30 59.80 11.97 478.41 8.16 0.32 5.27 0.01 0.03
dm-11 0.00 0.00 0.09 0.39 2.48 3.09 11.63 0.00 0.56 0.12 0.01
dm-12 0.00 0.00 0.01 0.06 0.07 0.46 8.00 0.00 0.39 0.14 0.00
dm-13 0.00 0.00 0.01 0.07 0.08 0.54 8.00 0.00 0.26 0.08 0.00
dm-14 0.00 0.00 0.01 0.07 0.07 0.53 8.00 0.00 0.26 0.08 0.00
 


I've found iostat works best over a period of time.

iostat -xtc 5 7

This will run the command 7 times, with a 5 second pause between each run.

03/25/2020 07:28:50 AM
avg-cpu: %user %nice %system %iowait %steal %idle
0.26 0.00 0.10 0.00 0.00 99.64

Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 2.80 41.20 0.00 0.00 0.07 14.71 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.10
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 4.80 34.20 0.00 0.00 0.12 7.12 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.10 0.00 0.10

In my case, I'm not too busy.
avg-cpu = how busy I am. iowait is how long you are waiting to read or write to disk. Mine is 0.0
%idle is the opposite of how busy I am. I am 99% idle.

r/s is reads per second. w/s is writes per second. d/s is deletes (removes) per second. Notice I have a lot of 0.0's under those columns.
wkB/s is how many kilobytes per second are written. All of that doesn't matter a lot.
The columns you really care about are the ones that say "wait" in them. f_await, w_await, d_await, etc...
This is how long your disk is "waiting" to read or write something. In general anything below
2.0 is pretty good. 3.0 and 4.0 means you're slowing down some. 5.0 or higher means you have
a lot of latency on your disk. You can always ignore the first line of data. It always gives bogus data for some reason.
 
Thanks a lot dos2unix for your guidance. It's really helpful for learning.
In the below output, iowait is 4.79. which means something is slowing down. But how to find out that.
it may be the application or could be some faulty on disk as well. correct ?
How to figure out that.

Also Is there any way to overcome this issue & improve the io wait .

avg-cpu: %user %nice %system %iowait %steal %idle
1.51 0.12 0.62 4.79 0.00 92.95
 
Can you include the next few lines so we can see if it's reads, writes, flushes, or deletes?

I'm not sure which linux distro you are using, but do you have iotop, or can you install it?
iotop -a is a handy command.
 
Ways to improve disk i/o

1. use RAID if possible (spread the load across multiple disks)
2. use an SSD drive, NVME drives are even faster.
3. if possible spread your data out, do reads on one disk, writes on another disk.
4. In some cases, using a multi-threaded application will help.
(This is a tricky subject, some will disagree, but I've found sometimes i.o is waiting
on other dependencies).
5. if you have a fast network (gigabyte preferred) split out part of your data to a remote
disk (via NFS or SAN)

I would be interested to see the output of "sudo fdisk -l" just to see what kind of disks you have.
 
Ways to improve disk i/o

1. use RAID if possible (spread the load across multiple disks)
2. use an SSD drive, NVME drives are even faster.
3. if possible spread your data out, do reads on one disk, writes on another disk.
4. In some cases, using a multi-threaded application will help.
(This is a tricky subject, some will disagree, but I've found sometimes i.o is waiting
on other dependencies).
5. if you have a fast network (gigabyte preferred) split out part of your data to a remote
disk (via NFS or SAN)

I would be interested to see the output of "sudo fdisk -l" just to see what kind of disks you have.

Please find the below output of sudo fdisk -l

Disk /dev/sdb: 1395.9 GB, 1395864371200 bytes
255 heads, 63 sectors/track, 169704 cylinders, total 2726297600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a726b

Device Boot Start End Blocks Id System
/dev/sdb1 2048 419430399 209714176 8e Linux LVM
/dev/sdb2 419430400 1258291199 419430400 8e Linux LVM
/dev/sdb3 1258291200 1677721599 209715200 8e Linux LVM
/dev/sdb4 1677721600 2726297599 524288000 8e Linux LVM

Disk /dev/sda: 21.5 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000928e2

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1060863 529408 83 Linux
/dev/sda2 1060864 41943039 20441088 8e Linux LVM
 
Can you include the next few lines so we can see if it's reads, writes, flushes, or deletes?

I'm not sure which linux distro you are using, but do you have iotop, or can you install it?
iotop -a is a handy command.

I'm checking on SUSE Linux Enterprise Server 11 SP4 OS.
iotop is not installed here. I will check if we can install iotop here.
Please see the below full output of iostat -x

avg-cpu: %user %nice %system %iowait %steal %idle
1.47 0.12 0.60 4.58 0.00 93.22

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sdb 24.64 815.88 224.30 672.00 55354.90 17379.60 81.15 0.90 1.00 0.59 52.54
sda 0.08 4.36 0.69 2.52 63.22 55.07 36.85 0.00 0.86 0.59 0.19
dm-0 0.00 0.00 0.57 2.38 27.74 19.01 15.89 0.03 11.05 0.18 0.05
dm-1 0.00 0.00 0.01 0.00 0.06 0.00 8.00 0.00 0.13 0.13 0.00
dm-2 0.00 0.00 0.18 4.44 34.56 35.53 15.16 0.01 2.96 0.29 0.14
dm-3 0.00 0.00 0.38 0.07 64.10 0.58 141.36 0.00 3.29 0.10 0.00
dm-4 0.00 0.00 244.03 239.77 51793.32 4782.17 116.94 0.62 1.28 0.11 5.54
dm-5 0.00 0.00 0.07 0.13 3.48 1.00 23.05 0.00 0.40 0.17 0.00
dm-6 0.00 0.00 1.27 344.08 1086.36 2752.65 11.12 13.69 39.60 0.01 0.21
dm-7 0.00 0.00 0.08 426.21 0.89 4714.38 11.06 1.30 3.05 0.56 23.99
dm-8 0.00 0.00 2.73 426.25 2394.02 4714.39 16.57 1.32 3.07 0.56 24.23
dm-9 0.00 0.00 0.01 0.06 0.05 0.46 8.00 0.00 0.25 0.10 0.00
dm-10 0.00 0.00 0.26 51.17 10.24 409.39 8.16 0.27 5.27 0.01 0.03
dm-11 0.00 0.00 0.08 0.38 2.12 3.04 11.23 0.00 0.55 0.13 0.01
dm-12 0.00 0.00 0.01 0.06 0.06 0.46 8.00 0.00 0.36 0.13 0.00
dm-13 0.00 0.00 0.01 0.07 0.07 0.54 8.00 0.00 0.24 0.07 0.00
dm-14 0.00 0.00 0.01 0.07 0.06 0.53 8.00 0.00 0.24 0.08 0.00
 

Members online


Latest posts

Top