Understanding bottlenecks
Many bottlenecks occur from moving data around. We saw this in Chapter 4, with the CPU, where moving data from RAM into the CPU registers is a relatively slow operation compared to operations within the CPU itself, and moving data over a network is many times slower than this. Moving data between different processes on different computers is something that needs to be considered quite carefully.
The MPI framework forces one to be explicit about moving data, which makes it easier to understand where these kinds of transfer bottlenecks occur. However, these are not the only kind of bottlenecks. Of course, there are computation bottlenecks too, where long computations hold up the progress of the whole system, and there are external bottlenecks, such as loading data from disk or communicating with external services.
Long computations and loading latency are less of a problem if they don’t hold up the whole system. (Here, system refers to the whole...
