Both the vmstat and sar commands provide average CPU run queue details. Referring back to Figure 4-7, the run queue information is in the far-left column, and its value is zero in every case. Since that particular system contains four CPU cores, the CPU subsystem is not very busy. A run queue of 1, 2, 3, or 4 would be fine. But we do not want a run queue of 5 or more, since this is an OLTP-based system.

Figure 4-13 shows a number of 30-second vmstat samples from a very active and highly volatile Oracle workload. As a review, the average CPU utilization is around 73% percent, or said another way, the average CPU idle time is around 27%. Based on queuing theory, a four-CPU core system running at 73% will have an average run queue of 5.3, and queue time will be around 31% of the response time. So while it may seem surprising, the response time is already significantly degrading. The vmstat report shows the average run queues to be between 3 and 7, with an average of 5.2 (I did the math). Notice the queuing theory prediction of 5.3 was very close. So based both on utilization and run queue length, this Oracle system is running out of CPU resources, and there is a very high likelihood the users are not pleased with performance.

Figure 4-13. A vmstat report on a very active four-CPU core database server. This particular system has a very dynamic workload. This is why even with around 30% idle time, the run queue frequently exceeds the number of CPU cores.