PyTorch’s computation graphs
PyTorch performs its computations based on a directed acyclic graph (DAG). In this section, we will see how these graphs can be defined for a simple arithmetic computation. Then, we will see the dynamic graph paradigm, as well as how the graph is created on the fly in PyTorch.
Understanding computation graphs
PyTorch relies on building a computation graph at its core, and it uses this computation graph to derive relationships between tensors from the input all the way to the output. Let’s say that we have rank 0 (scalar) tensors a, b, and c and we want to evaluate z = 2 × (a – b) + c.
This evaluation can be represented as a computation graph, as shown in Figure 13.1:
Figure 13.1: How a computation graph works
As you can see, the computation graph is simply a network of nodes. Each node resembles an operation, which applies a function to its input tensor or tensors...
