This chapter serves as an introduction to the blockchain and its underlying technology. It requires minimal technical knowledge. It summarizes the content that you can expect to encounter in the upcoming chapters that go into greater depth. In this chapter, I will explain the blockchainÂ technology is on a more architectural level, rather than describing every detail of each component or layer.
When you ask 10 people what they think the term blockchain means, you might not be surprised to get 10 different and conflicting answers, as use ofÂ the term blockchainÂ can be a bit confusing. A lot of people connect it to Bitcoin (https://bitcoin.org/) or other virtual (crypto) currencies. Some people talk about the Ethereum Application Platform (https://ethereum.org), some about smart contracts (executable code), but most of the time,Â blockchain is explained as a distributed, shared ledger.
In essence, a blockchain is a system of maintaining digitally-distributed ledgers in a way that allows individualsÂ who do not fully trust each other to agree on updates to the shared ledger. Blockchains use peer-to-peer protocols rather than a central authority or third party to distribute and verify transactions between entities. There isÂ no single point of failure, so entities can appear, disappear, or malfunction without affecting the group.
In this chapter, you will learn about the following topics:
- What is a blockchain and how does it work?
- What is the architecture behind theÂ blockchainÂ ecosystem?
- What are the differences between public and private blockchains?
- How secure is a blockchain, and what about anÂ individual'sÂ privacy?
- What kind of applications can run on a blockchain?
When talking about blockchains, we always refer to it as a distributed ledger technology (DLT)Â that established the underlying, open source technology behind Bitcoin.Â A blockchain is a digital system of recording transactions of assets in a list that is replicated across available nodes in a network, rather than being stored in a central data store, as is the case with traditional databases.
In a distributed ledger such asÂ a blockchain, the data is distributed to all nodes in a trustless manner (meaning without a trusted third party such as VISA, MasterCard, or your bank) using a peer-to-peer protocol in near real time. Each node individually processes and verifies every transaction redundantly, bundles the verified transactions into a block, and broadcasts them to all other nodes in the network. Through a consensus mechanism, the block of transactions is validated by other nodes in which the majority has to approve the block before it becomes final and is added to the blockchain. TheÂ blockchain uses a combination of digital signatures and cryptography to prove your identity and authenticity and to enforce read/write and execute permissions (access rights). This makes it possible to permit write access for certain participants and read access to other participants, or even to a wider audience; that is, everybody.
If you loosely compare a blockchain to a traditional database, a blockchain is a system that contains an ordinary database and some extra software that corroborates that submitted records conform to previously agreed-upon rules before adding the new records to the database. This extra software listens and broadcasts new records to all nodes, or peers, participating in the network, ensuring that each peer has the same data in its database. The following diagram is an overview of the capabilities that make up blockchain technology:
The major technologies used by the blockchain are divided into five groups, each representing a different layer in the architecture
Technically, a blockchain is a new method of data storage. It is actually just a file with a predefined data structure (that is, how the data is logically put together). It can be compared with other data structures, such as relational databases (tables, columns, and rows), XML files, comma-separated valuesÂ (csv), ExcelÂ databaseÂ files, and binary files (images and videos). An analogy that I often use is that blocks in a chain are the same as pages in a book. Each page in a book, just like this one, has a bunch of text structured in paragraphs, and information about its contextÂ (also called metadata), such as theÂ chapter number, chapterÂ title, and page number.Â Similarly, in a blockchain, each block consists of aÂ collection of content, for example, the list of transactions, and aÂ header, which contains technical information about the block, a reference to the previous block, and a digital signature (hash) of the data contained in the block.
A blockchain, where blocks are linked to each other to make a chain, is analogous to pages in a book. Pages use sequential numbering that makes it easy to know their order. If pages were to be pulled out of the book and thrown into a pile, it would be easy to put them back in order. A blockchain, though, is cleverer. The following diagram shows that each block links back to the previous block via the block's fingerprint. The fingerprint is determined by the individual block's content and the fingerprint of the previous block, as demonstrated in the following diagram:
Each block in the chain links back to its previous block, like page numbers in a book
In a book, the ordering of pages is implicitly built on a page whose number is one less; that is, page 13 follows page 12 (13-1), whereas blocks are represented by fingerprints or hashes that are built upon each other. For example, block 3 with hash 8ec6cc0 is determined by hashing its data together with hash 9a59c5f of the previous block. By using a fingerprint that is determined based on the previous one, it can be used for validating the internal consistency of the data.
This scenario is shown in the following screenshot:
Two blocks are linked together by their hashed fingerprints. TheÂ fingerprint of block #2Â is based on its data and the fingerprintÂ of block #1.
You can check whether the data is consistent within a block by generating the fingerprint yourself and comparing it to the one that is part of the block's header. If someone wants to change the information stored in one of the earlier blocks, they need to regenerate all of the fingerprints from that point until the end of the chain. However, the blockchain will appear to be altered, and it is instantly noticeable by others. Depending on the consensus method used, the creation of these fingerprints can be a very difficult and slow process, which makes it very problematic to rewrite the blockchain.Â Furthermore, the number of blocks already present in the blockchain can be huge, for example, for Bitcoin (June 3, 2018: 512253 blocks with a size of 156 GB). The following screenshot shows that when changing the data, the hash is also changed and the block becomes invalid:
The data in block #2 is changed, and it generates a different fingerprint than before
The technology (that is, the architecture, mechanisms, security, and so forth) behind the blockchain can be seen both as the backbone for future accounting and as an engine for a modern message transport system. So how does a blockchain actually work during runtime?
The following diagram shows howÂ blockchain systems must follow a specific flow of actions in order to consistentlyÂ maintain a distributed ledger of facts (a key-value database with the current state of assets) and a separate history of their updates (distributed transaction log). The transactions that you submit are stored and verified without the involvement of a governing central authority using advanced mathematics and computer science, that is, cryptographic hash functions.
The blockchain not only secures these transactions but also protects their integrity (and anonymity). This demonstrated in the following diagram:
The flow that a transaction follows from submission to being verified by the majority of networks
A typical blockchain providesÂ clientÂ APIs to interact with theÂ transaction, but generallyÂ you would use a web application thatÂ callsÂ theÂ APIÂ with which you would submit your transaction to one of the active nodes (peers) in the network.
Depending on the network's capabilities, the transaction is either locally validated or directlyÂ broadcastÂ to all active nodes in the networkÂ using theÂ peer-to-peer network (P2P). When locally validated, your transaction is only broadcast after validation,Â based on a set of network rules. Each node that receives your transaction will, depending on network rules, either verify it instantly or transcribe it into a secure record and place it in a queue of pending transactions.Â In this case, nodes, in other words, the participating computers or servers in the network,Â verifyÂ whether your transaction is valid based on a set of rules to which the network has agreed.
For example, your transaction can trigger participating nodes to execute business logic (such as a smart contract) and follow the consensus protocol to verify the results. Some consensus protocols require you to pay a transaction fee to get your transaction verified. The amount you pay is part of your transaction, and it determines the time it takes to get verified. Some nodes that create blocks only verify transactions that have a higher reward.
Depending on the rules definedÂ by the network's protocol, validating nodes combine transactionsÂ into a block and digitally fingerprint the result so that they can be validated by other nodes in the network.Â Just like transactions, blocks are broadcast to all active nodes using the peer-to-peer network. WhenÂ consensusÂ is reached, depending on the network' rules, participating nodes either batch the transactions and store the results as a cryptographically secured, immutable data block, or they take the approved block and append it to their ledger. Consensus protocols may include a reward system for nodes that create a block or require you to pay a transaction fee to distribute the currency to the rightful recipient.
Following the consensus mechanisms and network rules, it may take several blocks until you can be certain that your transaction is verified and not part of an orphaned chain. It is possible for multiple peers to create a block at the same time or just a few seconds apart from each other. This can produce aÂ forkÂ in the chain. Both blocks are initially accepted by the majority of the network, creating two chains. For any block of transactions, however, there is only one way to the genesis blockÂ (that is, first created in the chain), and blocks that are part of the smaller chain are eventually rejected.
In the following diagram, you can see that the longest chain survives and that smaller chains are rejected:
The genesis block is shown in green. The main chainÂ is shown in black and the orphaned/rejected chain appears in red.
Those transactions that are part of the rejected block(s) are added back to the pool of queued transactions and will eventually be included in another block. With blockchains that use an incentive model, the reward that someone receives for solving an orphaned block is practically lost.
As I mentioned earlier, the blockchain is not just a distributed database; it includes advanced software and security techniques to create a network of nodes (peers) that are always in sync, where each node validates and verifies transactions and blocks redundantly in order to reach consensus, and it provides a platform to run decentralized applications. To achieve this, the blockchain or digital ledger technology is built upon a layered architecture. In most cases, thisÂ contains four or five layers, namely the data layer, network layer, consensus layer, incentive layer, and application layer.
The following diagram shows the common capabilities that are part of this layer:
The capabilities of the data layer, such as cryptographic protocols, hash functions, digital signatures, and others
The data layer includes capabilities that describe the transaction and data model, the type of participants that can connect and use the blockchain (that is, the permissions model), the structure of the chain and its blocks, and the cryptography and hash functions to secure data and generate digital signatures.
The first two capabilities, data model and transactions, are closely related to each other. This is where the data model describes the type of assets available on the network and its data structure. Transactions trigger actions to modify asset data and transact assets between participants. The data model can be very simple and contain just one asset, such as a cryptocurrency like Bitcoin, or a more complex model with multiple assets that can even have relationships between them. The same applies to the transactions available on the network.
An asset(s) can be created or referencedÂ in a transaction, which in essence transfers the asset(s) between two parties who wish to exchange the data, for example, processing a payment between two parties, placing an order on an online store, registering an automobile, tracking diamonds around the world, or sharing your digital identity. More background will be provided on these two capabilities in Chapter 3,Â Blockchain 101 - Assets, Transactions, and Hashes.
The chain structure is also related to transaction data. It describes the data structure in which individual transactions are combined into a block and how these blocks are chained to each other. The structure is usually different across blockchain platforms, since it is used to run on a specific platform. When comparing blockchains, you will notice that more advanced platforms have more complex structures; that is, they contain more information. For reference, the minimum size of a block on the Bitcoin blockchain is around 100 bytes, whereasÂ the minimum size of a block on the Ethereum blockchain is around 525 bytes. The chain structure also describes the maximum size of a block, how blocks are chained together, and how incentives are paid out to participants. More information will be providedÂ about this capability in Chapter 4, Blockchain 101 - Blocks, Chains, and Consensus.
These three capabilitiesâcryptographic protocols, hash functions, and digital signaturesâsecure and digitally sign the data in the blockchain. A cryptographic protocol performs a security-related function by applying cryptographic methods, and it describes the algorithms such as hash functions should be used. The protocol used by a blockchain usually incorporates at least these aspects:
- Entity authentication
- Public/private key cryptography
- Secured application-level data transport
- Fingerprint/signature generation
- Data encryption methods
To prove that the data on the blockchain has not been tampered with, transactions, blocks, and participating nodes are signed by generating digital signatures (or fingerprints) on the content of the data using hash functions. A hash function takes any input dataÂ and produces an output, which, based on the algorithm in use, has a different fixed length. The output of a hash function is always a string, for example, an SHA256 hash is a 256-bit (32 byte) string of the input data. However, the input could be a list of transactions, and the output would be the digital signature that is placed on the block so that the content of the block (transactions) can be easily validated by others.Â In the given example,Â SHA stands forÂ Secure Hash Algorithms,Â whichÂ are a family ofÂ cryptographic hash functionsÂ published by theÂ National Institute of Standards and Technology.Â
In the cryptography used in a blockchain, there is a fourth capability that ensures that the data is still valid, and that is a Merkle tree. The Merkle tree, or hash tree, is a tree of data blocks in which every leaf node is labeled by a hash or digital signature, and every non-leaf node (for example, the top of the tree) is labelled with the cryptographic hash of all of the digital signatures of its child nodes as its input. The hash tree is used by the blockchain to verify all data that is stored, processed, and transferred between nodes in the network. It can ensure that the blocks received from other nodes (peers) in the network are undamaged and unaltered, and it can verify that other peers do not send malicious data blocks and lie about their content. More details about these capabilities will be presented in Chapter 3, Blockchain 101 - Assets, Transactions, and Hashes.
A blockchain network is nothing without its participants. A public blockchain can have an infinite number of participants, since anyone can join the network. On the other hand, a blockchain network for business is a collectively-owned, peer-to-peer network operated by a group of identifiable participants. With either network, participants can be individuals or legal entities, such as a business, university, or hospital. The data layer describes the rules for joining the network and the permissions model for accessing and writing the data.
As the blockchain will become more relevant, powerful, and useful in direct proportion to the number of participants, this paradoxically raises one of the biggest concerns, which is the processing power needed to maintain and operate a huge blockchain over time. Solving this challenge is paramount from an architectural perspective and probably key for blockchain applications to become 100% viable in relation to a ton of use cases and eventually positioning themselves as mainstream technology.
The second layer up on the stack, just above the data layer, is the network layer. This layer deals with the propagation or broadcast of transactionsÂ and block dataÂ among available peers in the network,Â the reliability of the network, and local validation of data. The following diagram shows the common capabilities that are part of this layer:
The capabilities of the network layer include a peer-to-peer network to broadcast transactions
You may have heard of BitTorrent, a peer-to-peer network where users share files among each other without a central server having control over the data. The network layer of a blockchain is similar to this, and it is also managed by a peer-to-peer network, which is an architecture for distributing data in a network. In the case of blockchain, it is a network in which nodes (peers) are interconnected and share data or tasks (resources) among each other. This is different to a traditional client-server model, where the centralized server holds 100% of the data and the client needs to trust that the data is legitimate.
A peer-to-peer network runs without the use of a centralized administration system to coordinate transactions. Rather, it sends or broadcasts transactions to each connected party in the network. Nodes can join or leave when they want. When a node joins or rejoinsÂ the network, all active nodes will then share all of the updates required to be in sync. These peers commonly find each other through aÂ central index server, or by seeking other participants that use the same software through the internet. When a transaction is published to one of the peers in the network, that peer broadcasts the transaction and its data to all connected nodes in order to ensure that everyone is in sync with each other. Besides transactions, peers periodically generate blocks of verified transactions, which is part of the consensus mechanism, and they are broadcast in the same way.
A peer-to-peer network is, in some ways, less efficient than the client-server model, as the data is distributed and redundantly processed. However, each peer can operate independently, which makes theÂ peer-to-peer network more scalable and robust. Since no central server controls the flow of data, it is harder to close down the network. Additional detail regarding this capability is presented in Chapter 3,Â Blockchain 101 â Assets, Transactions, and Hashes.
When running a big-scale blockchain across multiple continents, it is sometimes required, due to legal or performance reasons,Â for example, that these continents' systems can run separately for each other. Each continent can have its own network of nodes running independently, but they are all connected to each other through trusted node(s) that relay validated blocks of transactions to and from each continent's network. More information about this capability is presented in Chapter 4, Blockchain 101 - Blocks, Chains, and Consensus.
Platforms that support smart contracts often support local validation. AsÂ the name suggests, this capability allows a peer to validate a transactionÂ locallyÂ before it broadcasts the transaction to the rest of the network. The peer to which the transaction is submitted will validate and execute the transaction and smart contract rules. Only if no exceptions occur will the transaction be broadcast, else the transaction is canceled. More information regarding this capability is presented inÂ Chapter 6,Â Understanding the Blockchain Data Flow.
The third layer of the stack is theÂ consensus layer. This layer deals with the enforcement of network rules that describe what nodes within the network should do to reach consensus about the broadcasted transactions. It also deals with the generation and verification of blocks. The following diagram shows the common capabilities that are part of this layer:
The capabilities of the consensus layer include the different consensus mechanisms that are available
The consensus layer includes capabilities that describe the rules for reaching consensus. The rules that need to be enforced depend on the consensus mechanism that is chosen when the network is initially set up.
When we speak about consensus, we mean the collaborative process that participating nodes of the network use to agree that a transaction is valid and to keep the distributed ledger synchronized at all times. These consensus mechanisms lower the risk of malicious (or fraudulent) transactions because they would have to occur (or be executed) across many locations at the same time, or else the tampering will be noticed almost immediately by other nodes. To reach consensus, the majority of the participants need to agree that the transaction is valid before it is permanently recorded in the ledger.
Once a transaction is permanent, no one, not even a system administrator, can delete the transaction from the ledger. The cost and time needed to reach consensus depends on the mechanism in place and the number of nodes participating in the consensus. A permissionless, or public, blockchain has relatively higher costs as compared to a trusted network of participants (permissioned or private blockchain). A wide variety of consensus mechanisms exist and are available to choose from in order to run an enterprise blockchain. When trust is high between nodes, a simple consensus mechanism, such as a majority vote, may be all that is needed. Alternatively, the network may choose to use a more hardened method.
The following example mechanisms and capabilities demonstrate how a network can reach consensus.Â The available consensus mechanisms can be categorized in to two groups: sophisticated and lightweight consensus mechanisms.Â Where the trust between participants is limited or non-existent, you will usually see one of the following more advanced consensus methods being used.
In the world of Bitcoins and Altcoins (cryptocurrencies based on the blockchain developed by the Bitcoin core team), the Proof of Work (PoW) mechanism is used for consensus.Â PoW was originally a protocol developed with the primary goal of preventing cyber attacks, such as a DDOS attack. The idea behind PoW was first published in 1993 by Cynthia Dwork and Moni Naor, and used in the Bitcoin white paper as it allows for trustless and distributed consensus. This protocol requires participating nodes to perform an intensive form of calculations (also called mining) in order to create a new group (or block) of trustless transactions on the blockchain. The mining of transactions is necessary for two reasons:
- Verifying the legitimacy of transactions
- Creating new digital currency to reward miners for executing the first reason
To verify these transactions, the miners need to solve a mathematical problem (or puzzle). The first miner that solves this puzzle gets the reward (in the form of new cryptocurrency) and a transaction fee amount supplied by the transaction owners. Verified blocks of transactions are permanently added to the public blockchain ledger, and with every new block, the puzzle gets a bit more difficult. This requires miners to work more efficiently over time. Miners who can deliver more computing power are usually the ones that solve the puzzle the quickest.
Luckily, there are other ways to verify transactions. A mechanism known as Proof of Stake (PoS) is an algorithm with the same end goal as PoW. However, the way it achieves the objective is different. The main difference between PoW and PoS is that with the latter, participation is restricted to the participants that have a legitimate stake (wealth) in the blockchain. Instead of all participants (or stakeholders) trying to confirm the validity of the information submitted, this consensus method chooses an individual to approve it by running a type a lottery.Â The chance of your being chosen is calculated based on your proportional stake (wealth) in the network. For each X amount of stake a participant holds, they get a lottery ticket. When it is time to verify and create a new block of transactions, the network chooses a lucky winner to announce their conclusions. Where a PoW-based blockchain rewards a miner for solving (mining) the mathematical puzzle to create a new block, a PoS-based blockchain does not reward an individual for creating a new block. Rather, the individual receives compensation (in the form of collected transaction fees). Thus, the term mining is replaced with the term forging, where a block is forgedÂ rather than mined.
This is demonstrated in the following diagram:
PoW requires expensive computer calculations to create a new block of transactions, which can be done by anyone.Â With PoS, the creator of a new blockÂ of transactionsÂ is chosen based on their stake (wealth).
The preceding diagram shows the main difference between the two consensus methods. The PoS consensus method has advantages over the PoW method, as it does not perform useless calculations in order to create a block. This prevents a lot of energy from being wasted and is more cost efficient. Also, the PoW protocol is sensitive to a 51% attack, where an individual controls a minimum of 51 percent of the total computational power available in the entire network. With the PoS protocol in place, it is much harder to execute this type of attack because an individual needs to own 51 percent of the total amount of wealth (assets/coins), which is very unlikely.
The PoS protocol also has a variant called the delegated proof of stake (DPoS). The main difference between the two is analogous to the difference between a direct democracy, where you vote for a specific person, and aÂ representativeÂ democracy, where you select a group of voters who then collectively vote on a specific person. With a DPoS, the participants choose an entity to represent their collective stake in the blockchain. Thus, you decide which entity, also called a delegate node, will represent your stake in the blockchain. This allows you to join a team in order to magnify your stake. This helps balance out the power of large stakeholders. In Chapter 4,Â Blockchain 101 - Blocks, Chains, and Consensus, I will address these protocols in more detail.
In a business environment, where trust between participants is high or at least partially present, you can come across one of the following more lightweight consensus methods.Â One of the most well-known consensus mechanisms, besides PoW and (delegated) PoS, is the practical byzantine fault tolerance (PBFT) method. It is used by many enterprise blockchain providers.
The difference between the previously-discussed sophisticated methods and PBFT is that this protocol is much more lightweight, since it does not require nodes to perform computations in order to create and verify blocks of transactions. With PBFT, every peer in the network maintains its own internal state, or their view on the current plan of actions. Transactions that are submitted to the network reach the validating peers at different times, so the order of transactions received doesn't have to be the same. In a given time period, peers that are fully in sync vote for a validating leader who chooses the sequence of transactions. The other peers use this sequence, in conjunction with their internal state, to perform a computation until they have the same sequence and consensus.
The following diagram shows you visually how this consensus mechanism works:
With PBFT, each peer might receive the same transactions in a different order, but broadcast their sequence to all other peers in order for validating peers to vote regarding the correct order
With PBFT, a consensus is reached based on the total number of decisions submitted by all peers.Â The consensus does not operate on the principle requiring that the majority of peers agree on a single decision submitted by a validating peer, but every decision that is sent.Â For every block, one of the validating pears is chosen as the leader, which will ultimately decide the final order of the transactions. So, if participant A sends one result sequence to participant B, and a different result sequence toÂ participantÂ C, thenÂ participantÂ C would send this different result toÂ participantÂ B, who concludes thatÂ participantÂ A has sent two different sequences.Â I will discuss the PBFTÂ in more detail in Chapter 4, Blockchain 101 - Blocks, Chains, and Consensus.
Another similar consensus mechanism is the federated Byzantine agreement (FBA). It assumes that participants in a network know each other, and they can distinguish which participants are important to them and which are not. In contrast to PBFT,Â instead of listening to all votes coming in for a certain sequence, a validating peer waits for the majority of the nodes it considers to be important to agree on a transaction before it agrees to the transaction. The same goes for the other validating peers in the network. A transaction is considered verified once enough peers considered important by enough nodes have agreed on its legality.
Another up-and-coming consensus mechanism is called the Tangle protocol. It works a bit differently than the ones I have explained up until now, and it is not considered a blockchain. When you submit a transaction withÂ theÂ TangleÂ protocol, it is confirmed by the network if and when two other peers in the network have proofed your transaction. Therefore, it is different from the others since, at any given time, no single node helps maintain the entire ledger. Each node helps by adding or editing two transactions at a time. The protocol also stores the transactions on a directed acyclic graph (DAG) (https://en.wikipedia.org/wiki/Directed_acyclic_graph) rather than on a linked list. In short, a DAG is a graph without a directed cycle, such that there is no way to start at any vertexÂ vÂ and follow a consistently-directed sequence of edges that eventually loops back toÂ vÂ again, as shown in the following diagram:
Transactions connected into a DAG. Arrows are drawn from the child to the parent. G is the genesis transaction.
The Tangle protocol does not know the concept of miners either. Since each edge holds only one transaction, other users can easily perform validation and PoW. Thus, there are no fees or rewards for confirming transactions in the ledger, but in order to submit your own transaction, you need to proof two other transactions first.
It might be sufficient to work with a permissions-only consensus. This capability describes a consensus method in which there is no PoW to be done to verify transactions. Rather, it is based on the authorization and granted permissions that a user has on the data. If a user has write privileges on a certain entity, then they are allowed to modify that entity's data. Some users might have all permissions on a system, whereas others only have read permissions on a specific type of entity. This consensus model is more in line with permission models used by a traditional database or web application.
A number of blockchain platforms currently available support a capability called sharding consensus. Sharding is a type of data partitioning that separates large databases into smaller, faster, and more manageable parts called shards. Some blockchains try to use sharding consensus to make the blockchain faster and more efficient by not having every validating peer validate the same data blocks. With sharding consensus, a validating peer is assigned to one of the partitions and is only responsible for validatingÂ transactions that are part of that partition. Other blockchains implement this capability by allowing participants to create private channels or subnets between specific peers in the network for the purpose of conducting private and confidential transactions,Â meaning peers can exchange transactions on a global ledger and exchange private transactions with other peers on a private channel. Participants who want to form a channel must be explicitly authenticated and authorized on that channel to transact and share updates on the ledger. Each channel has its own shared digital ledger and transactions log, and it must coexist in the same blockchain.
The fourth layer of the stack is the optionalÂ incentive layer. This layer deals with the distribution of rewards that are earned by nodes in the network for the work they do to reach consensus. Whether this layer is implemented or notÂ depends on the consensus mechanism in use. The following diagram shows the common capabilities that are part of this layer:
The capabilities of the incentive layer, including the distribution of rewards and transaction fees
The incentive layer include capabilities that describe what kinds of incentives are given by the network, when and how incentives can be earned by nodes, and the minimum amount of transaction fees needed to perform actions on the blockchain.
To run a successful public blockchain, there needs to be some kind of incentive program for individuals to join the network and to participate in the validation of transactions. As explained in the section on the consensus layer, what kind of work needs to be performed to reach consensusÂ depends on the mechanism chosen by the network originator: the more difficult the PoW, the more elaborate the rewards distribution system.
A blockchain that uses PoWÂ rewards the node (or miner) who solves the mathematical puzzle first when creating a new block. The node receives an amount of cryptocurrency in return. For example, Bitcoin currently has a block rewards of 12.5 coins, which halves every 210,000 blocks. With 144 blocks mined each day, it halves on average every four years. The miner that mines the block also receives all of the transaction fees. The total amount of transaction fees that the miner receives depends on the kind of transactions included in the block. The individual fee of a transaction is based on its size (in bytes), the age of its inputs (how long ago the coins spent were received), and the speed at which you want your transaction to be validated and verified. Thus, to submit a transactionÂ successfully,Â you need to calculate the amount of the fee you need to pay in order for your transaction to be included by a miner in one of the next blocks. TheÂ higher the fee you must pay, the quicker your transaction is validated and verified. For example, in Bitcoin, the transaction fee for the next block can range between $5 and $35 USD.
A blockchain that uses PoS does not have a reward system for mining, or, in this case, forging a block of transactions. All of the digital currency is created in the beginning and can, for example, be bought and sold through exchanges and may also distributed as transaction fees. The node that does the proofing of the transactions will only receive the transaction fees included by the original submitters of the transactions. The amount of transaction fee the forger receives depends on the complexity of each individual transaction and the fuel needed to execute it. So, to submit a transactionÂ successfully, once againÂ you need to calculate the amount of the fee you need to pay in order for your transaction to be validated and verified in one of the upcoming blocks. When the amount is too low, it is possible that your transaction will remain in a pending status for a long time. You can then choose whether to resend it with a higher fee. Some blockchains, such as Ethereum, in additionÂ to the transaction fee, also useÂ a gas feeÂ for the execution of the transaction, and when this is too low, your transaction will fail and you will lose the gas fee to the forger.
There are also consensus mechanisms that combine the two systems, such as Proof of Activity (PoA). With PoA, miners first have to solve a cryptographic puzzle to create a block. The winner then receives a block reward. This block, however, does not contain transactionsâit only contains the address of the winner. Next, a group of validating nodes (forgers) is chosen to verify the transaction that will be added, and they receive a reward for validating transactions.
The fifth and final (top) layer of the stack is called theÂ application layer. This layer deals with providing the interfaces to access, program, and use the blockchain. The following diagram shows the common capabilities that are part of this layer:
The capabilities of the application layer, including the programmable smart contracts and APIs
The application layer includes capabilities that provide application interfaces on top of the blockchain, both out-of-the-box functionality and custom implementations. The capabilities describe how the digital ledger is implemented and exposed to the world, how smart contracts can be built and run on the blockchain, and how third-party applications can interact with the digital ledger and smart contracts.
One of the core capabilities of the blockchain is the digital ledger, a type of database or system of records, which is distributed (shared, replicated, and synchronized) by the network layer among the participants in the network. The digital ledger records the transactions, such as the transfer of assets or data, from one participant to another, or among multiple participants in the network. Commonly, a key/value data structure is used to record these transactions and its assets. More advanced digital ledgers, such as the one used by Ethereum, NEO, and Hyperledger, also record smart contract code in the ledger as its own asset. SomeÂ digital ledgers have the ability to save the current state separately from the transactions log, which allows third-party applications to query the dataÂ easilyÂ without needing to replay the entire transaction log.
Another capability that isÂ indispensable in a modern blockchain is smart contracts.Â A smart contract, also known as a cryptographic contract, governs interactions with the digital ledger, and it allows agreements between network participants to beÂ executedÂ automatically.Â Smart contracts can act as a complement, or substitute, for legal agreements. It is computer code that directly controls certain aspects of transactions under certain conditions. A smart contract not only defines the terms and conditions (rules and penalties) of an agreement, but it is also capable of automatically facilitating, executing, and enforcing the negotiation or performance of an agreement. A smart contract does this by taking the input, putting that input through the rules set out in the smart contract, and executing the required actions defined by those contractual clauses. For example, a smart contract could stipulate the pay-out on a shipping of perishables depending on when the shipment arrives. Shipments that arrive later than agreed to by both parties are free; otherwise, the importer automatically pays the grower the unit price Ã the number of units in the shipment.
For external clients and applications to interact with the blockchain, its ledger, and smart contracts, most platforms offer both a CLI (command-line interface) and a RESTful API (application programming interface). Such interfaces define a set of functions that developers can use to perform certain actions on the blockchain and receive responses via the HTTP protocol using the
POST method. The twoÂ interfacesÂ differÂ in terms of functions supported. For example, you can use the command-line interface to control the settings of your node, whereas RESTful APIs can be used to invoke and query data on the blockchain. More advanced blockchain platforms also offer SDKs (Software Development Kits) for specific programming languages that are an abstraction layer on top of these APIs.
A capability that is still a very new concept is a decentralized application. A decentralized application (dApp) is a blockchain-enabled website that runs independently on every node of the peer-to-peer network, rather than on a single serve. They are comprised of both a frontend (web) application and a backend application, where the smart contract (backend application) allows it to connect to the blockchain. For example, a decentralized application includes the data model it uses (participants, assets, and transactions), an authorization and permissions model, smart contracts (backend), and a frontend web application. One public blockchain platform that supports this capability is Blockstack (http://blockstack.org).
When starting to develop applications to be run on the blockchain, the technologies that you will need depend on whether you will allow anyone to join and write data on your blockchain, or whether only known, (partly) trusted entities will be allowed toÂ join and write data. Generally speaking, there are two kinds of blockchains: public, or permissionless, andÂ private, or permissioned. A public blockchain is not specifically owned by anyone, whereas a private blockchain can be owned by a single entity or by a consortium (group of entities). As explained throughout this chapter, both public and private blockchains use the same technologies, but this is where the similarities end.
- Public blockchains: When we talk about public blockchains, we generally mean that the distributed ledger is public, and virtually anyone, without having any permissions granted by a central authority, can write and/or read data to the ledger. Since a public blockchain is designed to be trustless (anyone can write to it), and participants don't need anyone's approval to add data to the ledger, it requires advanced mechanisms for arbitrating discrepancies and ways to defend itself against attacks, as there is no authority to decide what happens when someone misbehaves. To secure aÂ public blockchain, anyone can choose to participate in the consensusÂ (validation) process, assisting in validating transactions by determining which ones get added and by verifying the current form and status of the blockchain. Another advantage stemming from the factÂ that anyone can joinÂ aÂ public blockchain (trustless) is that most are secured by crypto-economics, where participants receive economic incentives for the work they do to validate transactions. This makes it more interesting to join, and it creates a fully decentralized network. The downside ofÂ aÂ public blockchainÂ is that these mechanisms create more complexity and raise the cost of running this type of blockchain.
- Private blockchains: Conversely, in a private blockchain, the distributed ledger is only accessible to participants who are known and trusted. The control over who can read verified transactions, who can submit transactions, and who can verify transactions, is done by a preselected set of nodes. Participants can only join after obtaining an invitation or gaining permission. An invitation can be sent by an existing participant, a certificate authority, or by a decision of the entire consortium.
The private blockchain is mandated when a consortium of parties wish to participate in trading, but sometimes do not fully trust one another, or when some information should only be accessible to some of the trading partners. Many of the mechanisms that a public blockchain needs to keep the data tamper-proof are not needed on aÂ private blockchain,but are regulated by legal contracts. ThisÂ dramaticallyÂ impacts the technical decisions and changes the building blocks and mechanisms required to run the blockchain. It leads to lower costs and the faster throughput of data, since there are fewer nodes that need to reach consensus. Due to its controlled-access aspect,Â private blockchainsdeliver increased privacyÂ so that mission-critical applications can also run on the blockchain. The downside of a private blockchain is that you have to decide which participants have the power of granting permissions.
I will discuss the differences between public and private blockchains and their advantages and challenges in more detail in Chapter 7, Public Versus Permissioned Blockchains and Their Providers.
Both public and private blockchains provide a certain level of out-of-the-box security for your data. The consensus mechanism is the main driver behind the security and correctness of the blockchain. With a public blockchain, all users need to abide by the consensus algorithm that verifies all transactions, and when doing so they need to prove that they made a sufficient amount of effort by solving a mathematical problem. In many cases, the first user to solve the problem, or who is chosen to solve the problem, gets rewarded. Each new solution then forms the basis for the next block of transactions to be solved. It becomes almost impossible to manipulate data that is confirmed in an earlier block, since it directly affects the blocks that were created after that block. A private blockchain is even more secure, since you need to have secured permission to participate on the network. Since read and write access needs to be grantedÂ explicitly to a participant, it is likely that you know and trust them. The changes on the ledger can be tracked back to an actual person, whereas with a public blockchain, it is only tied to a network address that can be owned by anyone.
Nonetheless, there are still some security risks that the software and the network rules cannot fix for you. Public blockchains, for example, that use cryptocurrencies to fuel their network have also led to black market trading. Since transactions are bound to an address and not a personal identity, it is hard to figure out who is actually trading. Because of this, public blockchains increasingly draw the attention of cybercriminals who steal cryptocurrencies or other available assets. Another security issue relates to the method of reaching consensus.
The most commonly-used meansÂ of achieving consensus consume a lot of energy. This may lead to centralization or the possibility of collusion, because the majority of the network nodes are run in countries with cheap electricity, or even within a single country.
This does not meanÂ there are no security concernsÂ when using a private blockchain. With a private blockchain, operators can control who is allowed to connect to the network and operate a node. Some concerns include the fact that a node can restrict the transmission of information or transmit incorrect information. Such nodes must be identified and bypassed in order to maintain the integrity of the system.
Besides security, you need to think about the participant's privacy on the blockchain. Privacy is much more nuanced, and addressing this issue can lead to uncomfortable questions. What needs to be kept private? Why? From whom? When looking into maintaining privacy, there are solutions that can be easily implementedÂ in some cases, while in others it may be much harder. Many of these solutions are compatible with the currently existing blockchains, but it depends on what you want to achieve as to whether they will prove satisfactory. Technologies that allow users to do absolutely everything on blockchain without the possibility of being tracked are more difficult to create.
To secure private data on a public blockchain, you can look into the following methods:
- Cryptographic obfuscation
- Secure multi-party computation
- Zero-knowledge proofs
In short, cryptographicÂ obfuscation is a way of turning your application into a black box version (or its equivalent). The application still has the same underlying logic, and it also returns the same outputs for the given inputs. However, because the data is encrypted along the way, it's impossible to determine any details of how it works.
Secure multi-party computationÂ is a type of cryptographic system where parties jointly compute a function over their inputs while keeping thoseÂ inputs private. Each party initially receives access to a share of the input by the sender and computes a function over that share. The outputs are returned to the sender, who can assemble the final output without any party knowing more than their initial share.
Another powerful technology is zero-knowledge proofs. This allows you to construct a mathematical proof that, when executing a given program on some input by the user, returns a particular output without revealing any other information. OneÂ zero-knowledge proofÂ that can easily be implemented uses a digital signature showing that you know the value of a private key, which, when processed by the smart contract, can be converted into a particular public output.
When using a private or permissioned blockchain, it is much easier to secure the privacy of the data in the blockchain because participants and operators can limit read and write permissions on shared data (assets) and the execution ofÂ transactions. Even though you might think you're losing the decentralization of the network, there is still some degree ofÂ decentralization maintained in their structure. This is done by allowing participants to grant read/write permissions to other participants, which leads to a partially decentralized design.
I will discuss the security risks and the privacy methods of both public and private blockchains in more detail in Chapter 5, Blockchain 101 - Security,Â Privacy, and Smart Contracts.
I want to conclude this chapter by going over some of the applications that you can imagine running on some of the public or private blockchains currently available. The most widely-used or anticipated public blockchain platforms on which to run your own decentralized application are Ethereum and Blockstack.
- Ethereum: One of the longestÂ standing platforms in existence is Ethereum, an open platform that lets you build, run, and use decentralized applications using the blockchain technology. This decentralized platform runs applications in the form of smart contracts; that is, applications that run exactly as programmed without censorship, fraud, or interference by a third party. These smart contracts consist of computer code that can facilitate the exchange of anything of value, such as currency, content, property, or shares. Because of the decentralized nature of the platform, there is no possibility of downtime.
- Blockstack: One upcoming public platform is Blockstack, which aims to be the newÂ internet of decentralized applications. WithÂ Blockstack,Â users own their data and maintain theirÂ privacy, security, and freedom, instead of the application provider. In addition to the platform, Blockstack also has its own browser in which you can access the available apps. It uses the already existing lower level of the public internet, but applications are serverless and decentralized. Blockstack also provides a decentralized domain name system (DNS), decentralized public key distribution system, and registry for apps and user identities.
Either of these platforms lets you build applications. Ethereum is more basic and broadly accepted, but it only supports smart contracts (backend code). Blockstack, on the other hand, is a platform for decentralized applications that contains both frontend (web application) and backend (smart contracts).
Storj (pronounced: storage) is a decentralized cloud storage platform that uses the blockchain technology and end-to-end cryptography to secure your files in a decentralized manner. Storj protects your data by encrypting your files client-side, shredding them into little chunks called shards, and storing these pieces in a decentralized network of computers. Because the files are shredded into little pieces, nobody has a complete copy of your encrypted files. Beyond serving as a platform, it is also a cryptocurrency and a suite of decentralized applications. The applications can be used to store your files on the network or rent out your hard drive space. The cryptocurrency calledÂ Storj Token is given in return to nodes that share their hard drive space. More information about the platform can be found on their website atÂ https://storj.io/.
Personally, I like to listen to great music and always hope that the artists I listen to get paid for their work from theÂ monthlyÂ subscription fees that I pay. One platform that tries to address the issue of ownership (creator's identity) and music licensing is UjoMusic. The platform provides a portal where the artist can own their creative works, and the use of those works always remains in the control of the artist. It uses Ethereum, so it is no longer necessary to register a copyright and sign with a publisher in order to ensure that an artist gets paid when somebody uses or listens to their creation.Â UjoMusic uses a decentralized and distributed file storage system (Swarm, the decentralized storage branch of Ethereum) to enable redundant copies of the data that are fault-tolerant, resilient, censorship-resistant, and self-sustaining due to the built-in incentive systems. UjoMusic also uses COALA IP (https://www.coalaip.org/), which is a blockchain-ready intellectual property and licensing protocol so that they can build an open, global database of rights, holders and their works. More information about theÂ UjoMusicÂ platform can be found on their website atÂ https://ujomusic.com/.
Who hasn't bought something online from Amazon, eBay, or Alibaba? These companies offer a marketplace where you can buy goods from sellers worldwide. Instead of directly interacting with the seller to conduct a transaction, the data is owned by the online service, and the payment goes through a provider such as VISA or MasterCard.
The blockchain project OpenBazaar aims to cut out the middleman. It still provides a platform for e-commerce, but it uses a different approach. It uses the blockchain technology to put the power back into the customer's hands. Instead of buyers and sellers needing to go through a centralized service like Amazon and using a credit card to make a purchase, OpenBazaar connectsÂ buyers and sellersÂ directly. To use OpenBazaar, you download the client application. With this application, a seller can create a new product listing, including the details that you would normally see on an e-commerce website. When you publish the listing, it becomes accessible and is distributed over a peer-to-peer network to other users. Anyone can search for the item based on the keywords you applied. If someone buys your item, they pay with a cryptocurrency, like Bitcoin, and, when purchased, the client application creates a contract between the buyer and the seller with both digital signatures. Payments are sent into an escrow account for holding. Once the seller has sent the item to the buyer and they are satisfied with it, the buyer releases the funds and the seller receives the cryptocurrency (for example, Bitcoins). Surely it can all go wrong, just like any other marketplace? A buyer can receive something totally different from what they ordered or receive nothing at all. In this case,Â OpenBazaarÂ offers the use of moderators (also users of the network), who both the buyer and seller trust, to resolve the issue, and they only release the funds if they sign the transaction. More information about this platform can be found at https://www.openbazaar.org/.
The last three examples discussed were of public applications that could be used by anyone. In some cases, however, you don't want to build and run a decentralized application that is accessible to everyone in the world. Certainly, with enterprise or mission-critical applications, you want to control access and secure the privacy of the data on the blockchain. If you want to run these kinds of decentralized applications, it is better to look at private or permissioned blockchain platforms, such as Hyperledger (Fabric) and R3/Corda.
- Hyperledger Fabric:Â Hyperledger, and specifically the Hyperledger Fabric project, is one of the most commonly-used permissioned blockchains, and it is endorsed by several large IT vendors, including IBM, Oracle, and Microsoft. It is a blockchain framework implementation, a combination of a digital ledger technology (DLT) and a smart contract engine. Hyperledger Fabric is a platform with a modular architecture that can be used for all kinds of distributed ledger solutions. It aims to deliver a high degree of confidentiality, resiliency, flexibility, and scalability. The architecture is designed so that it supports pluggable implementations of different components and accommodates the changes in the blockchain/digital ledger ecosystem.
- R3/Corda: Some private or consortium blockchains focus on specific markers. Corda is one of these, and it is a distributed/shared ledger platform specifically designed for recording and processing financial and legal agreements by regulated financial institutions. The platform supports smart contracts, which, in Corda's case, is an agreement whose execution can both be automated through computer code along with human input and control, and whose rights and obligations, as expressed in legal prose, are legally enforceable. Corda links the business logic and data of the smart contract with associated legal prose in order to ensure that the financial agreement(s) follow the law and can be enforced.
You can build applications on both platforms. Whereas R3/Corda is specific to financial and legal applications on which you can execute and enforce smart contracts linked to legal contracts, Hyperledger Fabric is a platform that lets you create and control your own blockchain and the applications running on its network. One example of a permissioned, decentralized application that uses the Hyperledger Fabric project is Medicalchain.
This is one blockchain that I currently follow with much interest. It uses the blockchain technology to store patients, medical/health recordsÂ securely. Instead of different organizations having their own copy of these records, it maintains a single version of the truth. If organizations such as doctors, hospitals, laboratories, pharmacists, and health insurers want access to a patient's record, they can request permission for this purpose and record additional transactions on the distributed ledger. The platform creates a user-focused electronic health record, and it stores and shares this record securely in the blockchain. Medicalchain empowers and enables patients and users to give healthcare professionals explicit access to their data, and it records all interactions with the data in an auditable, transparent, and secure way. The platform offers users two applications: a doctor-to-patient telemedicine application, and a health data marketplace application. The telemedicine application enables users to consult with a doctor remotely (using their mobile phone, for example) for a small fee that can be paid directly to the doctor within the application. The platform also offers organizations the ability to build custom, decentralized applications to improve the (localized) user experience and put theseÂ applications on the marketplace. Users are able to use their medical data to power these applications and other services running on the platform. Besides Hyperledger Fabric, they can also use the public Ethereum blockchain for its cryptocurrency, which can then be used to pay for services on the platform. More information about this platform can be found athttps://medicalchain.com,Â or by reading their white paper athttps://medicalchain.com/Medicalchain-Whitepaper-EN.pdf.
Another similar example in which a private or permissioned blockchain such as Hyperledger Fabric can be used is for vehicle insurance. Imagine a consortium of drivers, insurers, emergency services, and vehicle repair shops.
Together, they operate a blockchain and run a decentralized application that can record the insurance policy, driver reports, and driving records, allowing an Internet of Things (IoT)-equipped vehicle to execute a claimÂ automaticallyÂ when it is involved in an accident. The application could automate claim processing, verification, fraud detection, and payment. Such an application would eliminate duplicate reports and enable the sharing of all data in a transparent way so that repair shops wouldn't have to contact the insurance company first before repairing the car, as they have already been informed that they are allowed to do the repairs.
I will discuss some of the available blockchain providers and how you work with them to develop your own applications in greater detail in Chapter 7, Public Versus Permissioned Blockchains and their Providers.
This chapter served as an introduction to the technology behind blockchain. The chapter started by explaining that a blockchain is a digital system of recording transactions of assets in a list that is replicated across available nodes in the network. A block in the chain is somewhat analogous to pages in a book.
The chapterÂ continued by answering the question of how a blockchain works, and it described the flow of data in a blockchain once a transaction is submitted, from the broadcast of transactions to all other nodes using a peer-to-peer network to the verification steps in the consensus protocol. I reviewed each step in the flow of data, its purpose, and the changes that occur to the data.
In the next section, you learned what the architecture behind the blockchain technology looks like, as well as the architectural layers and capabilities of a typical blockchain. I described the purpose of each layer, namely the data layer, network layer, consensus layer, incentive layer, and application layer, and, for each capability, additional details about what you can do with it were provided.
After explaining the technology in more detail, you learned about the similarities and differences between public and private blockchains, how each secures the data and ensures your privacy, and what kinds of applications can run on public and private blockchains.
In the next chapter, How Blockchain Will Disrupt Your Clients and Customers, I will talk about Oracle's vision for using the blockchain in everyday applications, how it will affect Oracle developers, and the possibilities for end users/customers.