If the problem were centered at the database server, the server process would be either consuming CPU or posting a wait event related to latching, IO, or perhaps an enqueue. But since the server process is posting an SNMFC, we know the problem is beyond the database server. If the problem were with the end user, the user would not be staring idly at the screen.

A story about one of my consulting engagements may clarify this concept. The architecture was classic client/server, with the server process on the database server, the client process on the user's PC, and the network allowing the two processes to communicate. The users told me when they executed a particular application function, it was taking around 30 seconds to complete. The DBAs and the operating system team noticed the database server was essentially idle, and expected the problem to be focused squarely on the vendor application. And as you would expect, the vendor proclaimed the database server was undersized. I told the DBAs that we needed to profile a session while running the key application function that was causing so much pain. Using only Oracle's performance views (that is, no tracing was involved), repeated tests resulted in this situation:

First, there was no denying the users were waiting and staring at their screen for 19 seconds. I was standing there watching them! The Oracle CPU time was gathered from v$sesstat, the network and client time was wait event SNMFC time, database server IO was all IO-related wait event time, and any leftover time was placed into a category I called unaccounted for time. No one disputed the numbers, as they could be easily demonstrated and were based on the rtsess.sql OraPub script.5 Obviously, the majority of time was spent either by the Oracle client process or network activity between the client process and the server process. We did a few quick tnsping executions, and as everyone expected, communication to the database server on the floor below us took around 1 ms. This meant nearly all of the 12 seconds was consumed by the Oracle client process. It turned out the application was doing some fairly advanced numerical calculations, which required significant CPU resources. The vendor could have responded that the user's PC was undersized, but the vendor knew the customer's PCs met the stated sizing recommendations. This squarely placed any real performance improvement on the vendor. It was one of those rare (but not impossible) situations where focusing on the other two circles (Oracle and the operating system) would not materially improve performance.