Monday, March 26, 2018

Finding IO statistics in Linux Environment for Slow Disks


I have written 2 articles (AWR 1, AWR 2) in relation to the disk latency and how to read AWR reports to investigate IO slowness. In this article I will explain how we check disks IO performance in Linux systems. We will check how the disks where our datafiles and redo log files are stored are performing. These disks could be simple Linux mount points, or ASM disks. In case of ASM, we will need to find out the disks that are part of ASM diskgroups so that we can check the performance of those disks. For example, following is the way how we can find out the disks that are part of ASM diskgroup.

[root ~]# oracleasm listdisks
DATA01
DATA02
DATA03

[root ~]# oracleasm querydisk -p data01
Disk "DATA01" is a valid ASM disk
/dev/sdj1: LABEL="DATA01" TYPE="oracleasm"

[root ~]# oracleasm querydisk -p data02
Disk "DATA02" is a valid ASM disk
/dev/sdk1: LABEL="DATA02" TYPE="oracleasm"

[root ~]# oracleasm querydisk -p data03
Disk "DATA03" is a valid ASM disk
/dev/sdl1: LABEL="DATA03" TYPE="oracleasm"

As we can see in the outputs of above commands, sdj, sdk, and sdl are the disks that are part of an ASM diskgroup. Now let’s check the IO stats for these disks.

Using iostat Command


Following is an output of “iostat” command that collects 5 samples at an interval of 2 seconds each. Column “await” is the key in this output that shows the time in milliseconds being taken for IO for each of the disks. 
[oracle~]$ iostat -x 2 5
Linux 3.8.13-68.3.2.el6 uek.x86_64 (dbserver)7/03/2017  _x86_64_ (16 CPU)














avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                  33.57    0.00   11.04      26.37        0.00   29.03














Device:
rrqm/s
wrqm/s
r/s
w/s
rsec/s
wsec/s
avgrq-sz
avgqu-sz
await
r_await
w_await
svctm
%util
fd0
0
0
0
0
0.03
0
8
0
97.63
97.63
0
97.63
0.03
sdb
0.03
2.28
3.11
1.7
130
38.47
35.09
0.15
30.74
26.94
37.7
9.17
4.4
sdc
0
0
0.03
0
0.22
0
7.98
0
4.57
4.56
64
4.13
0.01
sdd
0
0
0.28
0
2.23
0
8
0
1.32
1.32
0
1.32
0.04
sde
1.52
2.98
0.29
0.56
14.45
28.52
50.33
0.03
31.56
19.08
37.99
17.2
1.47
sdf
0.81
0
0.55
0
4.39
0
7.99
0.05
83.48
83.48
0
83.38
4.59
sdg
0.81
0
0.55
0
4.39
0
7.99
0.04
80.75
80.75
0
80.7
4.44
sdh
1.88
0
0.55
0
4.38
0
7.96
0.04
80.98
80.98
0
80.92
4.45
sdi
0.02
0.05
1.58
0.71
45.53
97.99
62.73
0.48
211.99
153.91
341.62
47.67
10.91
sdj
0.52
0.03
59.51
37.16
8028.88
2335
107.21
10.88
112.51
118.51
102.9
10.03
96.99
sdk
0.48
0.03
59.09
36.04
8015.36
2298.75
108.43
10.49
110.29
115.96
101
10.19
96.92
sdl
0.15
0.04
10.66
22.81
4269.91
4872.02
273.2
2.59
77.41
111.53
61.46
23.52
78.71
sda
0.1
1.67
3.32
0.71
114.54
30.16
35.92
0.09
22.38
22.21
23.2
8.27
3.33
dm-0
0
0
4.93
3.14
155.24
128.16
35.14
0.63
78.01
64.82
98.71
16.4
13.23
dm-1
0
0
1.79
3.57
14.35
28.52
8
0.19
34.91
19.46
42.68
2.73
1.46
dm-2
0
0
3.15
3.97
129.9
38.47
23.64
0.2
27.71
27.25
28.08
6.19
4.41
dm-3
0
0
0.02
0
0.12
0
8
0
10.28
10.26
64
9.51
0.01














avg-cpu: %user   %nice %system %iowait %steal %idle
                58.44    0.00    29.78      5.78        0              6














Device:
rrqm/s
wrqm/s
r/s
w/s
rsec/s
wsec/s
avgrq-sz
avgqu-sz
await
r_await
w_await
svctm
%util
fd0
0
0
0
0
0
0
0
0
0
0
0
0
0
sdb
0
0
0
1.89
0
15.09
8
0.06
33.5
0
33.5
31.25
5.9
sdc
0
0
0
0
0
0
0
0
0
0
0
0
0
sdd
0
0
0
0
0
0
0
0
0
0
0
0
0
sde
3.3
0
0.47
0
30.19
0
64
0.01
15
15
0
15
0.71
sdf
0
0
0
0
0
0
0
0
0
0
0
0
0
sdg
0
0
0
0
0
0
0
0
0
0
0
0
0
sdh
0
0
0
0
0
0
0
0
0
0
0
0
0
sdi
0
0
3.3
1.42
98.11
11.32
23.2
1.31
247.1
264.86
205.67
212.2
100.1
sdj
0
0
12.26
9.43
211.32
275.47
22.43
7.42
340
391.62
272.9
44.91
97.45
sdk
0
0
7.55
14.15
150.94
1137.26
59.37
5.04
255.3
323.94
218.7
44.91
97.45
sdl
0
0
29.25
4.25
11849.53
623.11
372.42
14.56
375.32
402.21
190.11
29.1
97.45
sda
0
3.3
8.96
1.42
173.58
41.51
20.73
0.43
41.36
39.79
51.33
41.18
42.74
dm-0
0
0
11.32
5.19
139.62
45.28
11.2
1.65
90.34
102.21
64.45
56.6
93.44
dm-1
0
0
3.77
0
30.19
0
8
0.06
15
15
0
1.88
0.71
dm-2
0
0
0
1.89
0
15.09
8
0.06
33.5
0
33.5
31.25
5.9
dm-3
0
0
0
0
0
0
0
0
0
0
0
0
0














avg-cpu: %user   %nice %system %iowait %steal %idle
                32.43     0.00   17.57       28.34     0          21.66














Device:
rrqm/s
wrqm/s
r/s
w/s
rsec/s
wsec/s
avgrq-sz
avgqu-sz
await
r_await
w_await
svctm
%util
fd0
0
0
0
0
0
0
0
0
0
0
0
0
0
sdb
0
4.95
0
0.99
0
47.52
48
0
2.5
0
2.5
2.5
0.25
sdc
0
0
0
0
0
0
0
0
0
0
0
0
0
sdd
0
0
0
0
0
0
0
0
0
0
0
0
0
sde
0
0
0
0
0
0
0
0
0
0
0
0
0
sdf
0
0
0
0
0
0
0
0
0
0
0
0
0
sdg
0
0
0
0
0
0
0
0
0
0
0
0
0
sdh
0
0
0
0
0
0
0
0
0
0
0
0
0
sdi
0
0
3.96
0
95.05
0
24
0.82
258.12
258.12
0
206.3
81.68
sdj
0
0
23.27
13.86
443.56
727.23
31.53
6.07
181.6
182.4
180.25
26.53
98.51
sdk
0
0
14.36
8.42
300.99
616.34
40.28
5.09
221.46
212
237.59
43.26
98.51
sdl
0
0
63.37
7.43
26360.4
559.9
380.27
11.61
205.43
210.75
160
13.92
98.51
sda
0
0
5.45
0
55.45
0
10.18
0.15
27.27
27.27
0
27.36
14.9
dm-0
0
0
9.9
0
154.46
0
15.6
0.97
118.25
118.25
0
89.6
88.71
dm-1
0
0
0
0
0
0
0
0
0
0
0
0
0
dm-2
0
0
0
5.94
0
47.52
8
0
0.42
0
0.42
0.42
0.25
dm-3
0
0
0
0
0
0
0
0
0
0
0
0
0














avg-cpu: %user   %nice %system %iowait %steal %idle
                26.62    0.00    14.45      37.87      0          21.06














Device:
rrqm/s
wrqm/s
r/s
w/s
rsec/s
wsec/s
avgrq-sz
avgqu-sz
await
r_await
w_await
svctm
%util
fd0
0
0
0
0
0
0
0
0
0
0
0
0
0
sdb
0
0
0
0
0
0
0
0
0
0
0
0
0
sdc
0
0
0
0
0
0
0
0
0
0
0
0
0
sdd
0
0
0
0
0
0
0
0
0
0
0
0
0
sde
0
0
0
0
0
0
0
0
0
0
0
0
0
sdf
0
0
0
0
0
0
0
0
0
0
0
0
0
sdg
0
0
0
0
0
0
0
0
0
0
0
0
0
sdh
0
0
0
0
0
0
0
0
0
0
0
0
0
sdi
0
0
9
3.5
220
28
19.84
1.55
95.36
77.78
140.57
84.24
105.3
sdj
0
0
41.5
17
704
1165
31.95
7.33
107.21
116.67
84.09
19.43
113.7
sdk
0
0
22.5
12
376
1432
52.41
4.53
105.43
104.67
106.88
32.54
112.3
sdl
0
0
123
10
49425.5
540.5
375.68
10.18
66.91
68.65
45.6
8.55
113.7
sda
0
3
1
1.5
12
36
19.2
0.03
4.6
7
3
13
3.25
dm-0
0
0
10
8.5
232
68
16.22
1.59
65.24
70.7
58.82
56.92
105.3
dm-1
0
0
0
0
0
0
0
0
0
0
0
0
0
dm-2
0
0
0
0
0
0
0
0
0
0
0
0
0
dm-3
0
0
0
0
0
0
0
0
0
0
0
0
0














avg-cpu: %user   %nice %system %iowait %steal %idle
                27.16    0.00    23.76      38.35      0         10.73














Device:
rrqm/s
wrqm/s
r/s
w/s
rsec/s
wsec/s
avgrq-sz
avgqu-sz
await
r_await
w_await
svctm
%util
fd0
0
0
0
0
0
0
0
0
0
0
0
0
0
sdb
0
4.98
0
1
0
47.76
48
0.07
71
0
71
71
7.06
sdc
0
0
0
0
0
0
0
0
0
0
0
0
0
sdd
0
0
0
0
0
0
0
0
0
0
0
0
0
sde
0
0
0
0
0
0
0
0
0
0
0
0
0
sdf
0
0
0
0
0
0
0
0
0
0
0
0
0
sdg
0
0
0
0
0
0
0
0
0
0
0
0
0
sdh
0
0
0
0
0
0
0
0
0
0
0
0
0
sdi
0
0
8.96
16.42
505.47
318.41
32.47
3.17
132.35
163.33
115.45
33.57
85.17
sdj
0
0
24.38
12.44
421.89
593.53
27.58
5.24
162.54
125.29
235.56
23.47
86.42
sdk
0
0
21.39
15.42
358.21
1857.71
60.19
4.1
131.34
130.67
132.26
23.32
85.87
sdl
0
0
93.03
6.97
38208.96
986.57
391.96
7.49
85.86
78.01
190.79
8.65
86.47
sda
0
0
7.96
1.99
107.46
27.86
13.6
0.29
30.5
23.5
58.5
16.85
16.77
dm-0
0
0
16.92
17.91
632.84
342.29
28
3.44
104.76
97.53
111.58
24.49
85.27
dm-1
0
0
0
0
0
0
0
0
0
0
0
0
0
dm-2
0
0
0
5.97
0
47.76
8
0.78
130.17
0
130.17
11.83
7.06
dm-3
0
0
0
0
0
0
0
0
0
0
0
0
0


Above outputs are during the time that I explained in my article AWR2. Stats highlighted in yellow seemed alarming as “await” column of output should not show this much high waits for the disks IO. Faster the disks are, lower this IO wait time should be. Normally I have observed this wait to be not more than 50 milliseconds on average.
In order to see the stats for only a single disk, “grep” command could be combined with “iostat” command.
[oracle ~]$ iostat -x | grep sdj
sdj
0.53
0.03
59.6
37.13
8085.81
2332.26
107.7
10.91
112.81
118.84
103.15
10.02
96.95

After the disks started performing good, following output shows the average io wait time decreased quite a lot.
[oracle ~]# iostat -x |grep 'sdl|sdj|sdk|await'
Device:
rrqm/s
wrqm/s
r/s
w/s
rsec/s
wsec/s
avgrq-sz
avgqu-sz
await
r_await
w_await
svctm
%util
sdj
0.38
0.02
70.26
51.71
7885.6
2744.98
87.16
9.01
73.88
83.37
60.98
7.78
94.92
sdk
0.35
0.02
70.57
50.81
7898.4
2734.41
87.6
8.87
73.05
82.2
60.35
7.82
94.9
sdl
0.15
0.05
10.52
39.44
4215.1
5670.47
197.84
2.27
45.46
92.99
32.78
14.78
73.84

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts - All Times