A couple of months ago, we had an interesting issue at a customer: an application wasn’t performing well, but the system had more than 20% CPU idle and wasn’t swapping memory, so it wasn’t a lack of resources. After a deeper look into vmstat, we saw a constant 30% of CPU in I/O state. We had some kind of I/O bottleneck.
To discover the root of the issue we used two programs:
- iostat (comes with the sysstat package): similar to vmstat or ifstat, but shows I/O operations per device and partition, updating its output every X seconds.
- iotop: like the classic top, sorting the processes according to their I/O rate.
By using these two utilities it’s quite easy to discover which process is creating the I/O bottleneck, and on which particular device.
In our case, the problem was a RAID controller that was giving a terrible writing performance, coupled with a process that was doing around 15 small, random access writes per second.
Últimos comentarios
RSS