The operating system should have a painfully raging IO subsystem bottleneck. If not, then there must be IO blockage somewhere between the log writer background process and the physical disks. For example, if a vendor-specific tool indicates the IO devices are performing well within tolerance, yet you can clearly demonstrate the jaw-droppingly slow IO system call response times (for example, the output from tracing the log writer background process), you will need to work closely with the IO team to figure out where in the IO stack the bottleneck is occurring.

There are many pieces of hardware and software between the actual physical disk and the log writer background process, and any one of them could be the bottleneck. Performance specialists will tell you they have seen everything from broken Ethernet cables and connectors to nearly all IO being routed to a single controller or a host-based adapter (HBA) card.

Expect the IO team to be defensive and possibly very aggressive in response to this news. After all, when millions of dollars are spent on an IO subsystem and the average write takes 50 ms, people get very nervous. Keep stressing that Oracle's sequential IO write requirements have simply exceeded the IO subsystem's capacity, and that people are working to reduce the IO requirements by focusing on Oracle and on the application. You can also show them the IO requirements Oracle is placing on the IO subsystem. Reports similar to the ones shown n Figures 8-10, 8-15, and 8-16 can be very helpful.