These days, systems with a single processing core, with just one logical processor, are known as single core.

When there is only one user running an application in a mono-processor machine and the processor is fast enough to deliver an adequate response time in critical operations, the model will work without any major problems.

For example, consider a robotic servant in the kitchen having just two hands to work with. If you ask him to do one task that requires both his hands, such as washing up, he will be efficient. He has a single processing core.

However, suppose that you ask him to do various tasks—wash up, clean the oven, make your lunch, mop the floor, cook dinner for your friends, and so on. You give him the list of tasks, and he works down the tasks. But since there is so much washing up, its 2 pm before he even starts making your lunch—by which time you get very hungry and make it yourself. You need more robots when you have multiple tasks. You need multiple execution cores, many logical processors.

Each task performed by the robot is a critical operation, because you and your friends are very hungry!

Let's consider another case. We have a mono-processor computer and it has many users connected, requesting services which the computer must process. In this case, we have many input streams and many output streams, one for each connected user. As there is just one microprocessor, there is only one input channel and only one output channel. Therefore the input streams are en-queued (multiplexing) for their processing, and then the same happens with the output streams but is inverted, as shown in the following diagram:

Why does the robot take so long to cook dinner for you and your friends? The robot does a tiny bit of each task, and then goes back to the list to see what else he should be doing. He has to keep moving to the list, read it, and then start a new task. The time it takes to complete the list is much longer because he is not fast enough to finish so many tasks in the required time. That's multiplexing, and the delay is called von Neumann's bottleneck. Multiplexing takes additional time because you have just one robot to do everything you need in the kitchen.

The systems with concurrent access by multiple users are known as multi-user systems.

If the processor is not fast enough to deliver an adequate response time in every critical operation requested by each connected user, a bottleneck will be generated in the processor's input queue. This is well known in computer architecture as von Neumann's bottleneck.

There are three possible solutions to this problem, each consisting of upgrading or increasing one of the following:

No matter which option we pick, we must consider other factors that depend particularly on the kind of operations performed by the computer and which could generate additional bottlenecks. In some cases, the main memory speed access could be too slow (the robot takes too much time to read each task). In some other cases, the disks subsystem could have bad response times (the robot takes too much time to memorize the tasks to be done), and so on. It is important to make a detailed analysis on these topics before taking a decision to troubleshoot bottlenecks.

Moreover, sometimes the amount of data that needs to be processed is too large and the problem is the transfer time between the memory and the processor. The robot is too slow to move each hand. Poor robot! Why don't you buy a new model?

In the last few years, every new micro-architecture developed by microprocessor manufacturers has focused on improving the processors' capacity to run instructions in parallel (a robot with more hands). Some examples of these are the continuous duplication of processing structures like the ALU (Arithmetic and Logic Unit) and the FPU (Floating Point Unit), and the growing number of processing cores that are included in one single physical processor. Hence, you can build a super robot with many independent robots and many hands. Each sub-robot can be made to specialize in a specific task, thus parallelizing the work.

Computers used as servers, with many connected users and running applications, take greater advantage of modern processors' capacity to run instructions in parallel as compared to those computers used by only one user. We will learn how to take full advantage of those features in the applications developed using the C# programming language. You want the robot to get your lunch on time!

A group of researchers need some consulting services of an IT professional specialized in parallel computing. They are not very clear in explaining the kind of research they are doing. However, you decide to help them.

They want to find the best computer micro-architecture needed to parallelize an application.

Research the new micro-architectures that are being prepared by leading PC microprocessor manufacturers, and the schedules for their release, particularly on these topics:

It is a good idea to look inside the micro-architectures available in modern personal computers with multiprocessing. What changes would you consider making in the applications in order to take full advantage of them?

One of the most common mistakes in designing or upgrading systems with multiple processors is making linear projections in their processing speed. It is very common to consider that each additional processor in the system will increase the performance in a way that is directly proportional to its processing capacity.

For instance, when we have a system with just one processor, and if we add three more, we will not have four times the performance. This is because each time we add a processor, the time they dedicate to coordinate their work and the task assignment process increases. Therefore, because of the increased processing power spent on managing tasks, their performance will not increase linearly.

The additional robots added to the kitchen must talk among themselves to coordinate their work.

The coordination costs and the performance increment depend upon a number of factors, including:

All these items represent a problem when we design or upgrade a machine, because we need answers to these questions:

We need a reference, similar to the one offered in the following table in which we can see the coordination cost and the relative performance for an increasing number of processors:

This table was prepared taking into account an overall average performance test with many typical applications well optimized for multiprocessing, and the most modern processors with multiple execution cores used in workstations and servers. These processors were all compatible with AMD64 or EMT64 instruction sets, also known as x86-64. We can take these values as a reference in order to have an idea of the performance improvement that we will see in optimized applications.

As we can see in the following image, the coordination cost grows exponentially as the number of processors or cores increases:

As we can see in the following image, the relative performance grows logarithmically as the number of processors or cores increase:

Nowadays the problem is that, without many concurrent users, multiple processor systems have not proved to be as useful as expected. The use of machines equipped with more than one processor, in workstations used by just one user, is meaningful when the applications executed are designed to work with multiple processors.

Most applications designed for a single user are not optimized to take full advantage of multiple processors. Therefore, if the code is not prepared to use these additional processors, their performance will not improve, as was explained earlier.

But, why does this happen? The answer is simple. The process for developing applications that take full advantage of multiple processors is much more complex than traditional software development (this book will show how to make this task much easier). With the exception of specialized applications requiring a lot of processing capacity and those dedicated to resolving complex calculations, most applications have been developed using a traditional, linear programming scheme.

Nevertheless, the release of physical microprocessors with multiple logical execution cores leads to the widespread availability of multiprocessing systems and an urgent need to take full advantage of these micro-architectures.

A system with multiple processors can be analyzed and measured by the following items:

We are considering neither the storage space nor the amount of memory. We are focused on the parameters that define the operation and the performance of multiple processors.

The group of researchers will ask you many questions regarding the performance improvements that you can achieve in your new applications. You must be prepared to answer those questions.

Take some applications developed in any programming language and measure their performance. Using the procedure and the formulas explained, calculate an estimated performance improvement that can be achieved by optimizing them.

The numbers are a temptation to begin recoding old-fashioned linear programming applications, aren't they? Let's go on to study the technical concepts we will use in the rest of the book. We need them in order to begin coding lots of practical samples.

Many bottlenecks can arise in systems with multiple processors, besides the von Neumann's one (that is reduced, but still alive). The problems appear when the system runs out of processing capacity.

The following list enumerates the components in which the bottlenecks may appear with some possible solutions:

It is a good idea to detect bottlenecks in many applications using the tools provided by the operating system to monitor different subsystems. What changes would you consider making in your system to avoid them? Do you think you have enough memory to avoid continuous slow accesses to the hard disks?

There is a lot of software in the market that does not use the power offered by the available computers. We can improve the performance of this software.

One of the techniques for improving the processing capacity consists of increasing the microprocessors' working frequency (over-clocking), which raises the number of instructions capable of processing in the same period. This technique has been used for many years and evolved from the legendary 8086/8088 with its poor 4.77 MHz (Megahertz) to the many GHz (Gigahertz) of modern microprocessors.

Nevertheless, microprocessor manufacturers are increasingly facing difficulties in raising the frequencies because the manufacturing process becomes more complex and the generated heat is difficult to dissipate in an efficient and inexpensive way.

Consider our robot instance again. You want to buy a single robot, but want him to clean the oven in 5 seconds. That is possible, but he needs plutonium as an energy source because he must move his arms and legs at a very high speed. Besides, he needs an ambient temperature of 5^oF (5 degrees Fahrenheit) or -15^oC (-15 degrees Celsius). Why? Because metals moving at very high speeds generate heat. You do not want a burnt robot. Moreover, plutonium is very expensive. Something similar happens with modern microprocessors.

Therefore, the other alternative is to develop new micro-architectures; incorporating first duplicated, and then quadruplicated processing structures, and so on. In this way, there are many sub-processors in one single microprocessor's package. These sub-processors are known as execution cores or processing cores.

Microprocessors with multiple execution cores, also known as multi-core, offer many complete execution cores that are interconnected in a single package. Their physical look is very similar to a conventional single core microprocessor. Nevertheless, they are equivalent to something like two or more microprocessors inside a single piece of silicon, as well as many pieces of silicon interconnected between themselves under the same physical package. Of course, we are avoiding deep technical issues.

Ever since the rise of multiple execution cores, the possibilities of combining the communication architectures, the different cores owned, and shared resources have been multiplying. As with everything in life, in each possibility, there is a trade-off between manufacturing costs and performance. For this reason, a new land appeared in the microprocessors' world.

In some cases, each execution core includes L1 and L2 cache memories, as shown in the following image:

In other cases, L2 cache memories are shared between two or more cores. Therefore, each core will have access to the whole L2 cache, as shown in this image:

The greater the number of resources included in each core and the fewer the resources shared with the others, the greater the processing speed achieved by each core. On the other hand, sharing resources between cores benefits applications not optimized for multiprocessing because they use a single execution core.

The robots' communications interface must be as efficient as possible, since you want the robots to do many different tasks.

Achieving efficient external memory accesses is one of the most important matters with these micro-architectures. The communication with external memory has a great overhead with respect to time, as compared to the internal cores' speed. When we design the most critical algorithms for our applications, we must minimize the external memory accessed in order to achieve the best performance. It is one of the main subjects to consider in designing applications that will be developed with parallelism in mind.

There are two options available to speed up the tasks done by the robots, taking into account the washing up example:

The internal bus is very important, because it transports data between the different execution cores. Many microprocessors use one or more dedicated buses for that task with very high working speeds, while others establish those communications through the FSB (a less efficient way). When the microprocessor has more than two cores, the architecture could be any possible merger between the known architectures for single and dual core microprocessors. There are some microprocessors built by two pair of cores, each one using a dedicated bus for the data interchange between both the cores, but with both pairs talking through the FSB. The following diagram illustrates this micro-architecture:

When we are optimizing applications to take full advantage of these micro-architectures, one of the things that we should minimize is the information going through the FSB. Besides, we must consider this in evaluating the optimized applications efficiency. If we don't, we will probably draft wrong conclusions about them and we will try to optimize already maximized performances (according to the underlying hardware architecture).

A system with asymmetric multiprocessing based in many independent physical processors has many FSBs to access external memory, one for each physical processor. However, a system with a microprocessor having multiple cores has to share the FSB which acts as a great single door to the outside world and to the external memory. Therefore, the tasks for coordinating the activities in the execution cores require additional time to avoid conflicts in the shared FSB. This is an important difference between multiprocessing using independent physical microprocessors and multiple cores in one physical microprocessor.

The probability that a FSB will become a bottleneck is very high when applications are not optimized to take full advantage of cache memories included in each core. Therefore, when the software is running over these micro-architectures, it should avoid frequent accesses to the main memory.

Besides, many asymmetric multiprocessing systems use duplicated communication channels with the main memory. This feature is not available in many multi-core microprocessors. It makes it nearly impossible to predict the performance of applications in completely different systems architectures. However, designing them with parallelism in mind will take full advantage of any feature present in the system.

Nowadays, microprocessors with multiple execution cores are widespread. However, we can find many of them arranged in an 'n' way asymmetric multiprocessing system such as an eight-core system with two physical quad-core microprocessors. It is something very attractive in high range workstations and servers.

In the coming years, microprocessors are going to include more and more processing cores. Modern operating systems are already optimized to take advantage of their parallel processing capabilities. We must optimize our applications to take full advantage of them.

Analyzing the micro-architectures used in modern microprocessors is a topic for an entire book. However, we needed some knowledge about them in order to understand the parallel processing capabilities that are useful for our goals.

Do not expect plutonium robots! They will still be very expensive to maintain.

Look at the computers that you will be able to use with the researchers. How many microprocessors do they have? How many cores does each microprocessor have? You can use CPU-Z in order to find this information. You can download this free software from http://www.cpuid.com/.

In the following image, you can see the information provided by CPU-Z for a computer with a microprocessor providing four cores:

At the bottom of the window, the Cores item displays number 4.

Talking about information systems, scalability is an application's ability to upgrade its performance, its response time, or the number of users to which it can provide services. We can measure scalability from many points of views:

We will analyze scalability taking into account the processing capacity, leaving behind other hardware resources, which could break down scalability, as the ones mentioned earlier.

We have two possible ways to scale applications designed for mono-processing. One way is to use a microprocessor with a more efficient micro-architecture, capable of executing many more instructions per clock cycle. The other way is to use a microprocessor with the same micro-architecture, but with a higher frequency and hence more clock cycles. So, if we cannot find a microprocessor with one of the explained features offering a better performance than the one installed in the computer dedicated to run the application, we will not be able to improve its performance, enhance its response time, or increase the number of concurrent users.

The following table shows the complete processing power exploitation of an application designed for mono-processing with linear execution code, without considering parallelism. The table presents this situation with the increasing number of available processing cores:

The numbers are scary and demonstrate the need to tame multiprocessing quickly.

On the other hand, with applications designed to take full advantage of parallelism, multiprocessing, and multiple execution cores, scalability depends on the total number of processors or cores we can incorporate. Because applications are optimized for parallelism, each added processor will offer an immediate enhancement in performance despite the added management costs explained earlier. Hence, we will be able to achieve better performance using applications optimized for parallelism.

Mounting a computer with eight processing cores (two microprocessors with four cores in each) is more likely than asking a microprocessor manufacturer to develop an exclusive processor working at 20 GHz.

We will be able to generate applications with a dynamic scalability concept, according to the number of available or configured working processing cores.

If you have a computer with a multi-core microprocessor, you will be able to get some hands-on experience on the problems related to scalability. It is a good idea to monitor many applications you usually run and check each execution core load using Windows Task Manager. Are the applications taking full advantage of the processing power installed in your computing system? Are all the tasks using the full processing power?

The mechanism used for distributing the workload between many processors or cores that are part of a multiprocessing system is known as load balancing. Its main goal is to achieve the best possible performance by maintaining an equal workload between all the available processors or cores. It is also responsible for detecting with high accuracy the need to scale up (to increase the number of processors or cores).

If an inefficient workload distribution was applied, one of the cores would be overloaded with requests while the others would remain underutilized. For that reason, the load balancing algorithm has a critical role in the response time offered by the running application.

You do not want a robot to rebel! You want them to be happy! Therefore, you apply a carefully chosen load balancing procedure to distribute the tasks to the robots.

There are four traditional procedures for load balancing:

Take some processes and routines with poor performance results and redesign them to achieve better results with 2, 4, 8, and 16 execution cores. Which load balancing algorithm do you consider the best option for each application?

Why do we need to consider so many aspects related to hardware architecture in order to develop applications that take full advantage of parallelism? The answer is simple! Which is the most efficient way to reach Rome from our position? We need a map showing all the different possible ways to develop an accurate answer. The same happens when developing applications for multiprocessing. We need to know certain details about the way multiprocessing works in order to develop efficient methods to take full advantage of it. If we do not do this, any efforts made to take advantage of parallelism will not increase the application's performance.

There are great differences between parallelism and linear mono-processing programming. Hence, before starting with the necessary paradigm shift necessary for achieving the best results, it is very important to understand the underlying hardware architectures in which our optimized applications will be running.

We are talking about hardware, which is supposed to be a responsibility of modern operating systems. In some cases, as with the Java programming language, it is the responsibility of a virtual machine, which provides an even more isolated environment.

C# is a high-level, object-oriented language that provides a very good level of isolation from hardware. Nevertheless, when we are optimizing the applications for parallelism, we must obtain certain information about the hardware in order to adapt some aspects of the application to take full advantage of available resources.

This is the point where the first complications appear. The operating system isolates the applications from the underlying hardware, generating an intermediate manager. Its kernel is the lowest level layer above the hardware and manages memory, processors, and many other hardware, in a low-level way. Therefore, it is in charge of administering the different available processors or cores and distributing the workload in the most efficient way. It reduces our possibility to control certain variables and make them dependent on the operating system with which we are working.

The results achieved by the same applications in different Windows versions such as Vista, 2008, 2003, and XP can be quite different. It happens because each new release tends to be more optimized for the most recent hardware and hence for providing more efficient multiprocessing with multi-core microprocessors. For this reason, the operating system is also responsible for the performance results.

Multiprocessing is not new. It has been available for many years, although always limited to powerful servers and sophisticated specific workstations because of its expensive and difficult-to-find components.

Nowadays, microprocessor manufacturers tend to add more execution cores to every new microprocessor instead of increasing their working frequency. They also improve the instructions per clock cycle ratio. But the focus is on the parallelism capabilities. They are continuously improving the execution of instructions in parallel. Therefore, the multi-core revolution is here and hence, we will see more and more cores in one physical microprocessor. For this reason, the time has arrived to take full advantage of parallel architectures available in modern computers. Linear code has reached its end.

We will go through lots of samples covering the most useful situations in which we will need to take full advantage of the available processing cores with C# 2.0 and 3.0.

Exercise your mind to prepare for parallelism. This group of researchers seems to have many complex algorithms in mind: