My general IO subsystem performance rule of thumb is read requests must complete in 10 ms or less and write requests must complete in 5 ms or less. Only the most arrogant IO vendors would argue their 20 ms response-time IO subsystem is performing acceptably. When I look at device busyness via iostat or sar, unless the device is at least 5% busy, I don't bother to check the response time. Low-activity devices, when they are active, can result in some crazy-looking response times. Since the low-activity devices are not doing any real work for the system, we can just ignore them.

We can creatively use Oracle's instrumentation to monitor IO subsystem response time. Since Oracle times (instruments) each and every IO request, we simply query from the wait event views to get a very accurate performance view. Regardless of how the IO devices are performing, how the IO subsystem is configured, or how the network-attached storage (NAS) or storage area network (SAN) is performing, we can easily tell how long an Oracle IO request is taking.

Figure 4-19 is an OraPub wait interface report based on v$system_event. I ran the report once (not shown), waited 30 seconds, and then ran the report again. The second run, which contains what occurred during the 30-second interval, is shown in Figure 4-19. The top wait event is db file scattered read, which is what Oracle calls a multiblock read request emanating from an Oracle server process. During the 30-second interval, the average IO subsystem response time was 2.8 ms. That's very good, and I suspect some of the blocks were in a cache, but remember that these blocks are not in Oracle's buffer cache.6