Figure 4-17. CPU and IO subsystems are fundamentally different. In a CPU subsystem, any core can service any transaction. An IO subsystem device can service only requests directed to itself. This enables hot and cold IO devices. Even with advanced software, the balancing activity cannot be perfect.

This natural seemingly imbalance of IO activity is why so much effort and expense is put into keeping all devices equally busy. But even when all devices on average are just as busy as the next, at a point in time, they can fall into the situation shown in Figure 4-17. This is easy to see with an animated graphical simulation and much more difficult to describe in words, but I'll try.

Suppose you have 100 pieces of paper, 25 with the number 1 printed on them, 25 with the number 2, 25 with the number 3, and 25 with the number 4. You round up 100 people and give each one piece of paper. You find four other people and label them 1, 2, 3, and 4. You tell the 100 people to mix, and then stand in a single line on one side of a room. You place the four people on the other side of the room evenly spaced. You then stand at the beginning of the 100-person queue, and once every 10 seconds, you allow one person to walk to the numbered person that matches the number written on her piece of paper. You also tell the four people on the other side of the room that once a person is in front of them, that person must stay there for 5 seconds and then move away. Then you say, "Go!" You'll see that after a few minutes, some of the queues in front of the four people are very long, some are short, and some have no queue. When you look again in a couple of minutes, while the lines may have moved around, the same general situation exists!