It is a common belief that the bankruptcy filed by Lehman Brothers, a Wall Street banking giant, on September 15, 2008, triggered the global financial crisis in 2008-2009. Excessive risk exposure in subprime mortgage and financial derivatives by large banks almost brought down global financial systems. The crisis was the ultimate consequence of a fundamental breakdown of trust in the relationship between customers and the financial institutions that should have been serving them.
Shortly after that, Satoshi Nakamoto, a mysterious and anonymous entity, published a whitepaper on October 31, 2008, called Bitcoin: A Peer-to-Peer Electronic Cash System, which is considered the origin of Bitcoin and all cryptocurrencies. Satoshi proposed a completely decentralized approach for Peer-to-Peer (P2P) payment without central banks or intermediaries. He outlined the principles and functions of what would be developed and introduced as Bitcoin in the following year.
The technologies behind his invention are called blockchain and have since evolved well beyond Bitcoin and digital payment. It is now a suite of technologies, forming the foundation of distributed ledgers and cryptocurrency. No one knows who or what Satoshi is, if it is one individual or a group, but its paper is profoundly changing money, digital- and cryptocurrencies, business, and the world.
The purpose of this book is to help you to understand blockchain technologies, introduce you to the tools and technologies in the Ethereum ecosystem, and get you started with developing smart contracts and end-to-end decentralized applications. We will start with basic concepts in Bitcoin, Ethereum, cryptocurrency, and blockchain. In this chapter, we will cover the following topics:
- Introducing blockchain technology
- Rehashing cryptography
- Anatomizing the blockchain consensus mechanism
- Understanding Bitcoin and cryptocurrency
- Ushering in the world of Ethereum
For all the source code of this book, please refer the following GitHub link: github.com/Packt-Publishing/Learn-Ethereum.
You might have heard the parable of the blind men and an elephant. It is a folk tale about each of six blind men's individual descriptions of the same elephant based on their own touch and feel of the animal. It highlights the fact that different perspectives may lead to distinct viewpoints, emphasizing the limits of perception and the importance of a complete context.
When Satoshi invented Bitcoin, the fundamental concept in its vision was to build a blockchain, a shared public ledger (longest proof-of-work (PoW) chain), that verifies and records immutably all transactions through a decentralized computer network (P2P network) and a consensus mechanism with computational proof. Satoshi came up with an elegant solution solving the double-spend problem of electronic monies. A double-spend is an attack when someone tries to spend money through a transaction that isn't actually available anymore as the money was already spent before.
Blockchain is a new elephant in the digital world. To most of the public, blockchain is nothing but an obscure pseudonym for all cryptocurrencies, including Bitcoin, Ethereum, and more. So, what is blockchain? What does a blockchain look like? How does it work? Where can we use blockchain? Do you need a blockchain? Although there are many ways to describe a blockchain, mainly from different perspectives, there is no universal definition of a blockchain.
On the contrary, there are prevalent debates over the essential attributes or qualities of a blockchain. It is perceived as a new architecture with existing technologies, the next generation of the internet and web, a future database and distributed shared ledger, the new Napster (a P2P file-sharing system used in the 90s) with a pure decentralized P2P network, a cryptocurrency, or a trustless secure transaction system, and so on. It is all of them. Only by combining all of them can we understand the whole picture of blockchain technologies and get a sense of the true potential of blockchain.
The following diagram illustrates different viewpoints of blockchain technology:
So, what is a blockchain anyway? Think of blockchain as a new architecture paradigm and a new trust protocol. It is a computer science primitive forming the foundation of most cryptocurrencies and decentralized applications. It is a P2P transaction model that can enable two parties to transact in a way that is tamper-resistant and cryptographically proven. As the technology behind Bitcoin and other cryptocurrencies, blockchain is an open, distributed ledger that can be simultaneously used and shared within a large decentralized, publicly accessible network.
In essence, blockchain is a distributed shared ledger technology supported by three pillars, as shown in the following screenshot; these are P2P networks, cryptography, and a consensus mechanism:
To understand how blockchain works, let's start with the fundamental concepts and key building blocks of blockchain technologies. Then, we'll discuss the key differences between centralized, distributed, and decentralized systems. We will then dive into the blockchain data structure and discuss how transactions, blocks, and chains are maintained and how the network reaches a consensus on the state of the chain, as well as how to secure the blockchain with cryptographic technologies.
Following is a list of the key building blocks of blockchain technologies:
- Transactions: A transaction is a value transfer between two parties. It could be a transfer of money, tangible assets, or cryptocurrency. Transactions are broadcasted to the blockchain network. They are validated and verified by all nodes and collected into blocks. Once the block reaches a certain depth—in Bitcoin, this is 6 blocks—those transactions in the block can be considered irreversible.
- Block: All verified transaction records are collected into a data structure called a block. It has a header and body part, where the header contains a cryptographic hash of the previous block, a timestamp, and a Merkle tree root hash of all transactions in the block. The body is the container of transaction data.
- The chain of block (blockchain): A blockchain is a linked list of a chain of blocks. Blocks are linked together using a cryptography hash as the pointer to the previous block.
- Decentralized P2P network: It is a P2P network in which interconnected nodes share resources amongst each other without the use of a central authority or some sort of intermediary.
- Consensus protocol: The consensus protocol in blockchain is a set of rules that all network nodes will enforce when considering the validity of a block and its transactions. The consensus mechanism is the process used by the network nodes to achieve agreement on the network state. It is a fault-tolerant mechanism to ensure the reliability and integrity of the network.
- Mining: Mining is the process by which network nodes in blockchain systems add new blocks to the blockchain and get rewarded with crypto-incentives.
To explain how blockchain works, let's look at what steps are involved with the existing business model for completing a simple payment transaction.
A customer, Alice, needs to pay $10 to Bob, who happens to have an account in the same bank as Alice. She can make the payment either by visiting a bank branch or using the web. Let's say she tries to do that online through the bank's web portal. She will need to authenticate herself using her username and password and then put the transfer order in and wait for the bank system to confirm whether the transaction is completed.
As shown in the following diagram, in order to support such online banking activities in the traditional world, the bank has to establish an identity and access management system and authenticate Alice's login credentials. Behind the scenes, the bank needs to develop a bank web portal and a backend system to verify whether Alice has the right account with the bank and has enough money to pay Bob to transfer $10 out of Alice's account, and put $10 in Bob's account. The bank has to maintain a ledger to record the details of the transaction in a database and show the balance each person has.
The following diagram shows a centralized bank system model:
As the business grows, customers' needs change too. The traditional brick and mortar business model is being replaced by the digital banking and commerce model. This requires technology changes in the bank system too. Banks nowadays deploy a distributed system model to serve the ever-growing needs of their customers.
The following diagram shows the distributed bank system model:
The fundamental issue with the preceding centralized or distributed system model is a single point of failure. Failure could come from malicious network attacks, system failures, or security and privacy breaches; it could come from business failures in the bank itself, which can cause millions of people to lose their homes due to the bankruptcy of big banks during a global financial crisis. It could happen due to currency failure itself, such as the currency collapse in Venezuela, where the lifetime savings of average citizens suddenly became worthless overnight. Also, the payment may be blocked due to government censorship.
Satoshi Nakamoto believed that the root problem with traditional fiat currency is all the trust required to make it work. Citizens have to trust the central bank not to devalue the currency. Consumers have to trust the bank to manage their money. But history has shown again and again that trust is often breached.
Satoshi designed an elegant decentralized P2P electronic cash system, and the technology behind that, blockchain, is the solution, where transactions are maintained in a distributed shared ledger and replicated across a global P2P network; security and privacy are ensured with cryptographic technologies, and transaction integrity is achieved through a consensus mechanism.
The following diagram shows a decentralized bank system model:
As new transactions are made, they are broadcasted to all network nodes, and over time all transactions that have occurred are sequenced together in the public ledger and made available in all replicated network nodes, as shown in the following diagram:
Now that we understand the between centralized and decentralized models, let's how blockchain works.
Using the previous example, as shown in the following diagram, let's assume Alice wants to buy something from Bob and she agrees to pay Bob $10 or 10 Bitcoins (BTC):
Let's walk through the high-level processes step-by-step, demonstrating how the blockchain works:
- Create blockchain transactions: A transaction is a value transfer between two parties. When Alice sends $10 or 10 BTC to Bob, it will create a transaction with one or more inputs and two or more outputs, where the inputs reflect Alice's account, and the outputs reflect which account(s) Alice intends to transfer to. The transaction is then digitally signed with Alice's private key and broadcasted to the P2P network. The receiver will use the digital signature to verify the ownership of Alice's funds. We will discuss digital signatures and cryptographic hash functions in detail in later sections.
- Validate the transactions and add to the transaction pool: Once the transaction is submitted to the blockchain network, the bookkeeper node (usually a full node in a P2P network that receives the transactions) will validate it according to the business and technical rules defined by the blockchain network. If the transaction is valid, the bookkeeper will add it to the transaction pool and relay the transaction to the peers in the network.
- Create the candidate blocks: Transactions in the transaction pool are collected into the block periodically. In a Bitcoin network, every 10 minutes, a subset of network nodes, called mining nodes or miners, will collect all valid transactions from the transaction pool and create the candidate blocks. The following diagram shows the structure of a candidate block:
As illustrated in the preceding diagram, the high-level processes are as follows:
- The candidate block packages recent valid transactions into the block structure based on block specifications.
- For each transaction in the package, it creates a cryptographic hash of the transaction data, recursively calculates the hash out of existing hashes, and creates a Merkle root of all transactions, as depicted in the following diagram:
The miner node looks for the latest block on the blockchain and adds its hash to the block header of the candidate block as the reference from the candidate block it intends to link to.
- Mine the new block: Once the candidate block is created, the race starts for the chance to add new blocks and win the rewards. The process for such a race is called mining. The winning of the race is determined by the consensus mechanism. We will discuss different consensus mechanisms in later sections. In blockchain systems such as Bitcoin or Ethereum, the PoW consensus mechanism is applied to mining. Miners will keep trying to find a random number, the nonce in the block header structure, until the hash meets certain challenging conditions. For example, one such challenging condition is, the resulting block hash is smaller than a target number, or in some cases, the hash has a few leading zeros. In practice, every random number has the same chance to win the race, so practically, you can just start a loop through from 1 to 232 until it finds such a nonce. It requires huge CPU hashing power to find such a nonce. The challenging condition, called difficulty, can be adjusted based on the target number or bits in the block header structure. The difficulty in winning the race grows exponentially the smaller the target number is or the fewer bits are in the block header structure.
- Add a new block to the blockchain: The first winning node will announce the new block to the rest of the network for verification. Once the block is verified and approved by the majority of the network miners, it will be accepted and becomes the new tip of the chain. Since all blocks are chained together by linking the hash to the previous block, any tampering with the ledger becomes impossible since it will require PoW on all previous transitions.
All miners have the chance to solve the puzzle, but only the winning miner has the authority to add the block to the chains and claim the bounty. Once the new block is added to the blockchain, all in-progress miners will stop their mining efforts on the newly added block and start the race again on a new block.
The following diagram summarizes the step-by-step process when new transactions are submitted to the blockchain network:
Now you know how works. Cryptography plays a critical role in maintaining the transaction state in the blockchain and ensuring immutability. Cryptography is not new. In the next section, we will go over some key concepts in cryptography.
Cryptography is the study of secure communication techniques that prevent third parties or the public from reading private messages and allow only the intended recipient of a message to view its contents. It is the cornerstone of information security, which serves as the basis for delivering secure business applications and services. Modern cryptography concerns itself with the following five objectives of information security:
- Confidentiality: This is the concept of preventing sensitive data from being accessible by any unauthorized entities.
- Integrity: This means protecting sensitive data from unauthorized changes during transit from one party to another party.
- Authentication: This is the process of ensuring that user identity is truly what the user claims it to be, whether the user is human or a system.
- Authorization: This is the concept of determining what actions an authenticated user is allowed to perform.
- Non-repudiation: When a user performs an action on data, the action must be bound with the user so that it can't deny performing such actions.
Cryptography deals with the design of algorithms for encryption and decryption, which are intended to ensure the secrecy and the authenticity of the messages or transactions. Let's start with some key elements in modern cryptography:
- Encryption: This is the process of converting plain text or data into an unintelligent form, typically using a mathematical algorithm.
- Decryption: This is the process of reversing encryption—converting an encrypted message back into its original text and data.
- Hash: This is the process of converting any data block (arbitrary size or message) into a fixed-length hash code. A cryptographic hash function is a deterministic mathematical function performing such a conversion using cryptography, and it always maps to the same result for a given data block.
Cryptography is a key cornerstone of blockchain technology, along with the consensus mechanism and decentralization. It is used in many different forms, including, for example, wallets (for proof of cryptocurrency ownership), transactions (for PoW consensus), and P2P communication. In this section, we will go over key blockchain-related cryptography topics, including public-key cryptography, digital signatures, cryptographic hashing, and Merkle trees.
Public-key cryptography is a form of cryptographic function in which encryption and decryption are performed using two different keys—one public and one private key. They are generated in pairs. It is also called asymmetric cryptography. The public key can be shared with the public, but the private key is meant to be a secret code only known by its owner.
The keys are used in tandem too. Either of the two keys can be used in encryption, with the other one used for decryption. It is computationally improbable to determine the private key given only knowledge of the cryptographic algorithm and the public key.
Public key cryptography is most used in three ways, to:
- Secure the message transmission between two parties and ensure the confidentiality of messages or data.
- Authenticate the sender and ensure the message is indeed sent from the sender.
- Combine with the cryptographic hashing function and provide a digital signature on a document before sending it to the receiver.
We will go over the first two here and discuss digital signatures in the following section:
- Public key cryptography for confidentiality: In this case, as depicted in the following diagram, the digital signature is used to encipher messages between two parties during transmission. The sender (Alice) uses the receiver's public key to encrypt a message, and the receiver (Bob), who holds their own private key in secrecy, can decrypt the messages using their private key:
- Public key cryptography for authentication: In this case, as shown in the following diagram, the signature is used to authenticate the sender's message. The sender uses its own private key to encrypt a message before sending it to the intended parties. The receiver can use the sender's public key to confirm the message's authenticity and decrypt it. The combination of this approach with the message's cryptographic hashing function provides a digital signature, which we will discuss in the next section:
Public key cryptography is an essential technology underpinning wallets and transactions in the blockchain. We will discuss the Bitcoin wallet in the Understanding Bitcoin and cryptocurrency section.
A cryptographic hash function is an algorithm used to randomly convert a string of binary data into a condensed representation of a message—a message digest. Its output is called a hash value, digital fingerprint, digest, or checksum. It is deterministic and always results in the same hash value for a given message. It is capable of taking any size of data block and producing a fixed-size hash value that uniquely identifies the original data block. It is a one-way, irreversible function; the only way to recreate the input data is to try a brute-force approach with all possible values to see whether there is a match, which is almost computationally infeasible.
Cryptographic functions have been widely used in blockchain technology, including the following:
- Merkle trees: As we showed earlier, when a miner node pulls transactions from the transaction pool, it packages them in a block, where the block header has a field referencing the Merkle root of all transactions.
- Block chaining: Blocks in the blockchain are chained together with a reference to the previous block using a cryptographic hash.
- PoW: The PoW consensus algorithm itself is a game in solving a cryptographic hash function. We will discuss it in more detail in the Understanding the blockchain consensus mechanism section.
A digital signature is a set of algorithms for determining the authenticity and integrity of digital messages or documents. It assures the recipient that the message was indeed created by the expected sender and that the message was not altered during transmission. The sender cannot deny having sent the message.
When Alice sends a document to Bob, she will follow certain steps to digitally sign the document, as shown in the following diagram:
These steps are as follows:
- Calculate the message digest of the document Alice wants to send to Bob with a cryptographic hash function, usually MD5 or any SHA algorithm.
- Encrypt the message digest with Alice's private key, append the encrypted message digest to the original document, and send the combined message out.
- Once Bob receives the combined message from Alice, he will separate the encrypted message digest from the document itself. Bob will use Alice's public key to decrypt the encrypted message digest.
- At the same time, Bob will calculate the message digest of the received document and compare the resulting message digest with the decrypted message digest to see whether there is a match. If yes, Bob is assured that the document originated from Alice without any tampering.
In blockchain, a digital signature is a way to prove ownership of the underlying cryptocurrency or electronic coin. When Alice needs to pay Bob 10 BTC, she will digitally sign a hash of the previous transaction, which can prove that Alice has ownership of the 10 BTC.
In summary, cryptography is one of three foundational pillars in blockchain technology. Public key cryptography is the basis for blockchain wallets and transactions, and the cryptographic hash function is a key element underpinning the PoW consensus mechanism. A digital signature is used as proof of ownership of the underline electronic coins or cryptocurrency.
In the next section, we will introduce and look at a blockchain consensus mechanism in detail and discuss how cryptography technologies are leveraged to reach consensus among decentralized parties.
A fundamental problem in large-scale distributed systems is how to achieve overall system reliability in the presence of failures. Systems need to be fault-tolerant. This requires a process for distributed, often heterogeneous systems to reach a consensus and agree on the network state, whether it is a database commit or an action to take. In this section, we will discuss two types of consensus algorithms, PoW and proof-of-stake (PoS).
Consensus in a blockchain is the process by which a network of mutually distrusted nodes reaches an agreement on the global state of the chain of blocks. In blockchain, transactions or data are shared and distributed across the network. Every node has the same copy of the blockchain data. Consensus allows all of the network nodes to follow the same rules to validate transactions and add new blocks to the chain, and therefore allows it to maintain uniformity in all of the copies of a blockchain.
Sometimes, it is also called a consensus mechanism or consensus algorithm. A consensus mechanism focuses on the process and approach of how to reach an agreement. A consensus algorithm is a formal procedure or computer program for solving a consensus problem, based on conducting a sequence of specified actions. It is designed to achieve reliability in a network involving multiple nodes. Consensus algorithms ensure that the next block in a blockchain is fully validated and secured. Multiple kinds of consensus algorithms currently exist, each with different fundamental processes. Different blockchain platforms may implement different consensus mechanism. In this section, we will focus on the following two popular algorithms, show how they work, and discuss the pros and cons of each mechanism:
- PoW: This consensus algorithm was introduced by Satoshi and commonly adopted by many other blockchains, including Ethereum. The PoW is the mining process with the purpose of finding an answer to a cryptographic hashing problem. To do so, the miner has to follow the block selection rules to locate the previous block and use the hash from the previous block header, together with the Merkle root of current transactions in the new block, to solve the hashing problem. It requires considerable computations and hashing power. In Bitcoin, block selection rules that specify the longest chain wins.
- PoS: This consensus algorithm aims to select miners based on the various combinations of random selection based on the miners' wealth or age (the stake). Instead of miners competing to solve energy consuming cryptographic hash functions, the network will instead use a pool of validators. Validators are nodes that are willing to stake their cryptocurrency on the blocks of transactions that they claim should be added to the public blockchain.
Proof-of-work, also referred to as PoW, is the most popular consensus algorithm used by blockchain and cryptocurrencies such as Bitcoin and Ethereum, each one with its own differences. We will talk about the specific implementation of PoW in Bitcoin and Ethereum in later sections.
In general, PoW is like a race between miners to solve a cryptographic puzzle ; upon solving the puzzle, they win the chance to add the block to the chain and get rewarded. As shown in the following screenshot, miners collect all pending transactions from the decentralized network and compete with each other to solve the puzzle. Whoever solves the puzzle will generate a block and push that block into the network for verification from other nodes, after which, the other nodes can add that block to their own copy of the blockchain:
The cryptographic puzzle that miners race to solve is identifying the value of the nonce. A nonce is an attribute in the block header structure. In the beginning, each miner guesses a number to start with, checking whether the resulting hash value is less than the blockchain specific target. Bitcoin uses the SHA-256 algorithm for this. SHA-256 outputs a fixed-length number. Every number between 0 to 232 has the same chance to solve the puzzle, therefore a practical approach is to loop through from 0 to 232 until a number can meet the criteria, as shown in the following diagram:
Once a miner finds the nonce, the results, including the previous block's hash value; the collection of transactions; the Merkle root of all transactions in the block; and the nonce are broadcasted to the network for verification. Upon being notified, the other nodes from the network automatically check whether the results are valid. If the results are valid, they add the block to their copies of the blockchain, stop the mining work in hand, and move on to the next block.
A target is a blockchain-specific 256-bit number that the network sets up for all miners. The SHA-256 hash of a block's header—the nonce plus the rest of the block header—must be lower than or equal to the current target for the block to be accepted by the network.
The difficulty of a cryptographic puzzle depends on the number of leading zeros in the target. The lower the target, the more difficult it is to generate a block. Adding leading zeros in the target number will increase the difficulty of finding such a nonce exponentially. As you can imagine, the higher the difficulty setting, the more difficult it will be to evaluate the nonce. Adding one leading zero in the target will reduce by 50% the chance of finding the nonce. The difficulty is decided by the blockchain network itself. The basic rule of thumb is to set the difficulty proportionally to the total effort on the network. If the number of miner nodes doubles, the difficulty will also double. The difficulty is periodically adjusted to keep the block time around the target time. In Bitcoin, it is 10 minutes.
The winner of the cryptographic puzzle needs to expend huge energy and crucial CPU time to find the nonce and win the chance to create new blocks in the blockchain. The reward for such actions depends on the blockchain itself. In a Bitcoin blockchain, the winner will be rewarded with Bitcoin, the cryptocurrency in the Bitcoin blockchain.
The PoW consensus is a simple and yet reliable mechanism to maintain the state of the blockchain. It is simple to implement. It is a democratic lottery-based system that lets you participate in the game of mining and get the rewards, where every node can join and higher CPU power may not translate into higher rewards. Currently, the winning miner is rewarded with 12.5 BTC for each block created in the Bitcoin blockchain.
Satoshi's original intention in using a PoW mechanism is to solve double-spend issues and ensure the integrity of the global state of the Bitcoin blockchain network. Let's say Alice sends 10 BTC to Bob, and at the same time or later on she pays Catherine the same 10 BTC. We could end up with the following three situations:
- The first transaction goes through the PoW and is added to the blockchain when the second transaction is submitted. In this case, the second one will be rejected when miners pull it from the transaction pool and validate it against all parent blocks.
- Both transactions are submitted simultaneously and both go into the unconfirmed pool of transactions. In this case, only the first transaction gets a confirmation and will be added in the next block. Her second transaction will not be confirmed as per validation rules.
- Both get confirmed and are added into competing blocks. This happens when miners take both transactions from the pool and put them into competing blocks. The competing blocks form a temporary fork on the blockchain. Whichever transaction gets into the longest chain will be considered valid and spent, and the other one within the block on the short chain will be recycled. When it is reprocessed, it will be rejected since it is already spent. In this case, it may take a few blocks to get the other one recognized as the double-spent one.
However, there are a few drawbacks with the PoW algorithm, thanks to the economic cost of maintaining the blockchain network safety:
- Energy consumption: PoW consensus, which uses a network of powerful computers to secure the network, is extremely expensive and energy-intensive. Miners need to use specialized hardware with high computing capacity in order to perform mining and get rewards. A large amount of electricity is required to run these mining nodes continuously. Some people also claim these cryptographic hash calculations are useless as they can't produce any business value. At the end of 2018, the Bitcoin network across the Globe used more power than Denmark.
- Vulnerability: PoW consensus is vulnerable to 51% attacks, which means, in theory, dishonest miners could gain a majority of hashing power and manipulate the blockchain to their advantage.
- Centralization: Winning a mining game requires specified and expensive hardware, typically an ASIC type of machine. Expenses grow unmanageable, and mining becomes possible only for a small number of sophisticated miners. The consequence of this is a gradual increase in the centralization of the system, as it becomes a game of riches.
On the flip side, it requires huge computing power and electricity to take over the PoW-based blockchain. Therefore, PoW is perceived as an effective way to prevent Denial-of-Service (DoS) and Distributed Denial-of-Service (DDoS) attacks on the blockchain.
As opposed to PoW consensus, where miners are rewarded for solving cryptographic puzzles, in the PoS consensus algorithm, a pool of selected validators take turns proposing new blocks. The validator is chosen in a deterministic way, depending on its wealth, also defined as a stake. Anyone who deposits their coins as a stake can become a validator. The chance to participate may be proportional to the stakes they put in. Let's say, Alice, Bob, Catherine, and David put in 40 Ether, 30 Ether, 20 Ether, and 10 Ether stakes to participate respectively; they will get a 40%, 30%, 20%, and 10% chance of being selected as the block creator.
The following is how it works in the PoS consensus mechanism. As shown in the following diagram, the blockchain keeps track of a set of validators, sometimes also called block creators or forgers. At any time, whenever new blocks need to be created, the blockchain randomly selects a validator. The selected validator verifies the transactions and proposes new blocks for all validators to agree on. New blocks are then voted on by all current validators. Voting power is based on the stake the validator puts in. Whoever proposes invalid transactions or blocks or votes maliciously, which means they intentionally compromise the integrity of the chain, may lose their stakes. Upon the new blocks being accepted, the block creator can collect the transaction fee as the reward for the work of creating new blocks:
PoS is considered more energy-efficient and environment-friendly compared with the PoW mechanism. It is also perceived as more secure too. It essentially reduces the threat of a 51% attack since malicious validators would need to accumulate more than 50% of the total stakes in order to take over the blockchain network.
Similar to PoW, total decentralization may not be fully possible in the PoS-based public blockchain. This is because a few wealthy nodes can monopolize the stakes in the network. Those who put in more stakes can effectively control most of the voting. Both algorithms are subject to the social and economic issue that it makes the rich richer.
Earlier, we spoke about the temporary fork that occurs when two competing blocks are added to the blockchain. As shown in the following screenshot, this can continue until the majority of the nodes see the longest chain. Newer blocks will be appended to the longest chain. Blocks added to the shortleaf of the forked chain will be discarded, and those transactions will go back to the transaction pool and will be picked again for reprocessing. Eventually, the blockchain will comprise all conforming blocks, chained together using cryptographic hashes pointing to its ancestor:
Just like software development, forking is a common practice in blockchain. Forking takes place when a blockchain bifurcates into two separate paths. The following events, intentionally or accidentally, can trigger a blockchain fork:
- New features are added, requiring a change in blockchain protocol, such as block size, mining algorithm, and consensus rules.
- Hacking or software bugs.
- A temporary fork occurs when competing for blocks with the same block height.
A general forking scenario in a blockchain may look like the following screenshot:
Depending on the nature of such events, the actions to fix the issues could be a hard fork or a soft fork or, in the case of a temporary fork, doing nothing and allowing the network to self-heal.
A hard fork happens when radical changes in the blockchain protocol are introduced and it makes historical blocks non-conformant with new protocols or rules. Some are planned. Developers and operators agree with protocol changes and upgrades to new software. Blocks following the old protocol will be rejected, and blocks following the new protocol will become the longest chain moving forward.
But, in some cases, this is controversial and heavily debated in the blockchain community, as was the case with the Bitcoin fork on 6 August 2010 or the fork between Ethereum and Ethereum Classic. In such contentious hard fork cases, as long as there are miners maintaining the old and new software, the blocks created by the old and new software will diverge into separate blockchains.
The following screenshot illustrates both planned and contentious hard forks:
During a contentious hard fork of blockchain, a new cryptocurrency will be created to fuel the new blockchain. The owner of the existing crypto-assets may stay in the current network or move to the new network. When moving to the new network, they will receive a proportional amount of new cryptocurrency in the new network. Hard forks often create pricing volatility. The conversion rate between the old and new fork may be determined by the market. It is important to know the context and details of a hard fork, and understand the crypto-economic impacts of such a fork to both cryptocurrencies in order to take advantage of such sudden and drastic changes.
Once forked, nodes will start with separate paths moving forward. Nodes would need to decide which blockchain network they want to stay in. For example, Bitcoin Cash diverged from Bitcoin due to a disagreement within the Bitcoin community as to how to handle the scalability problem. As a result, Bitcoin Cash became its own chain and shares the transaction history from the genesis block up to the forking point. As of September 2 2019, Bitcoin Cash's market cap is around $5 billion, ranking fourth, versus Bitcoin's $215 billion.
A soft fork, by contrast, is any change of rules that is backward-compatible between two versions of the software and the blocks. It goes both ways. In the soft fork case, existing historical blocks are still considered valid blocks by the new software. At the same time, the new blocks created through new software can still be recognized as valid ones by the old software. In the decentralized network, not all nodes upgrade their software at the same time. Nodes staying with an older version of the blockchain software continue creating new blocks using the older software. Nodes upgraded to the newer version of blockchain software will create new blocks using new software. Eventually, when the majority of the network hashing capacity upgrades to a newer version of the software, in theory more blocks will be created with the newer version and make it the longest chain. Nodes with older software can still create new blocks. Since it is not in the longest chain, as illustrated in the following screenshot, similar to the temporary fork case, these blocks will soon be overtaken by the new chain:
Where more nodes are stuck on the older version, as illustrated in the following screenshot, new blocks created from an older version of blockchain software may become longer and longer; it will take a while for the new software to be effective:
So far, you have learned how PoW and PoS work. We have analyzed the advantages and disadvantages of different consensus mechanisms. In the next section, we will help you to understand what Bitcoin and cryptocurrency are and discuss how blockchain technology applies to Bitcoin.
Blockchain is the technology behind Bitcoin, which is considered the origin of all cryptocurrencies. In this section, we will introduce the basics of Bitcoin and discuss the digital payment mechanism with Bitcoin.
Bitcoin is a decentralized electronic cash system that makes peer-to-peer payment possible without going through an intermediary. The original Bitcoin software was developed by Satoshi Nakamoto, released under the MIT license in 2009, following the Bitcoin whitepaper, Bitcoin: A Peer-to-Peer Electronic Cash System. Bitcoin is the first successful implementation of a distributed cryptocurrency. Ten years after Bitcoin was born, as of February 16, 2019, it has about 17.5 million Bitcoins in circulation and it has reached about a 64 billion market cap (https://coinmarketcap.com/currencies/bitcoin/):
Like any fiat currencies or tangible assets, the price of Bitcoins can fluctuate over time and its valuation is determined by the open market. Several factors can influence the price, including supply and demand on the market, competing cryptocurrencies and altcoins, and governance and regulations. The following screenshot shows the Bitcoin market cap, daily transaction volume, and price movement since its inception up to February 2019:
In this section, we will present key concepts in Bitcoin, including the wallet, transaction and account balances, Bitcoin supply, and bootstrap. We will demonstrate how Bitcoin payment works with blockchain. We will also discuss major challenges in Bitcoin and the Bitcoin blockchain. In the end, we also briefly talk about various altcoins, a different type of cryptocurrency on the market.
Bitcoin is a cryptocurrency, digital cash, or virtual money. Unlike a fiat currency, you can't touch or feel it. You can't stash Bitcoins under your bed. So, where do you store your Bitcoins? How do you prove ownership of the Bitcoins? Technically, Bitcoins aren't stored anywhere. They don't exist in any physical form. They are a set of software objects circulating around the Bitcoin network, where ownership of the Bitcoin is proved with a cryptographic key. Payment records, with money transferred in or out, are recorded as a chain of private keys showing ownership transfer in the blockchain. If you own the private keys, you own that Bitcoin. If you lose your keys, you lose everything you have on the Bitcoin network.
A Bitcoin wallet is an application where the cryptographic keys, pairs of a public and private key, are stored. There are many forms of Bitcoin wallets in use, but broadly, they are categorized into the following four types; desktop, mobile, web, and hardware wallets. The private key is used for you to digitally sign the transaction when you spend the Bitcoin. Anyone who knows your public key can verify your signature of the payment you make to them. The public key—more accurately, a wallet address associated with your public key—is used for anyone to pay you with Bitcoin. You can have as many pairs of public and private keys as you want in your wallet.
In Bitcoin, a private key is a 256-bit-long hash and a public key is 512 bits long. They can be converted into shorter lengths in a hexadecimal representation. A Bitcoin address is generated based on the public key, using multiple rounds of mixed use of the following cryptographic hash functions: SHA-256 and RIPEMD-160. You can have as many addresses as you need, and each address can be used once for each Bitcoin transaction.
The following screenshot gives an example of a Bitcoin wallet generated from the bitaddress.org website:
The QR code on the left side is the Bitcoin address you can share with your trading partners. The secret one, the QR code on the right, is your private key when you sign your transaction.
Whenever you check your bank account, you always see a balance associated with your checking or savings accounts. Your bank keeps track of all of your transactions, and accumulates them and updates your balances following each and every transaction. A Bitcoin wallet provides you with a balance too. However, the balance in Bitcoin is not that straightforward. Instead of keeping track of every transaction, Bitcoin keeps track of unspent coins, also called UTXO.
UTXO stands for Unspent Transaction Output. In Bitcoin, a transaction is a collection of inputs and outputs transferring the ownership of Bitcoins between payer and payee. Inputs instruct the network which coin or coins the payment will draw from. Those coins in the input have to be unspent, which means they have not been used to pay someone else. Outputs provide the spendable amounts of Bitcoins that the payer agrees to pay to the payees. Once the transaction is made, the outputs become the unspent amounts to the payee; they remain unspent until the current payee pays someone else with the coin.
Taking the earlier example where Alice needs to pay Bob 10 BTCs, let's assume, prior to this transaction, that Alice has two UTXOs in her wallet, one with 5 BTCs and another with 8 BTCs. Bob has one UTXO of 30 BTCs in his wallet from other transactions. Let's also ignore the transaction fee for now. When Alice uses both UTXOs as the input to pay 10 BTCs to Bob, both will be the inputs of the transaction. One 10 BTC UTXO will be created as output to Bob, and one 3 BTC UTXO will be returned to Alice. After the transaction, Alice will have one 3 BTC UTXO in her account, and Bob will have two UTXOs in his account. They remain as UTXO until they are used to pay for other transactions:
When either Alice or Bob pays someone with the remaining UTXOs, the unspent output from the previous transaction becomes an input to the new transaction. Since all transactions are digitally signed, essentially a Bitcoin becomes a chain of digital signature on the Bitcoin blockchain network.
In Bitcoin, there is no central authority issuing the cryptocurrency and controlling the money supply. Instead, Bitcoin is created by the Bitcoin blockchain network through the discovery of new blocks. As shown in the following screenshot, the first block is also called the genesis block, or block #0, which was mined on June 3, 2009, with an output of 50 BTC. The first 50 BTC is unspendable.
The following screenshot shows the genesis block in the Bitcoin blockchain:
Bitcoin uses a Bitcoin generation algorithm to control how many coins will be minted and at what rate. It is a function of the Bitcoin block height and its block reward. It started with a block reward of 50 BTC. The block reward will be cut in half for every 210,000 blocks, or approximately, every four years. The rate of block creation is adjusted based on mining difficulty. The maximum capacity of Bitcoins in the system is 21 million, which can be reached when 6,929,999 blocks have been mined.
Take the earlier example, when Alice needs to pay Bob 10 BTC. Alice opens her Bitcoin wallet, and scans or copies Bob's transaction address and creates a transaction with a 10 BTC payment to Bob. Once the transaction is digitally signed and submitted, it is sent to the Bitcoin blockchain network:
Once the transaction is broadcasted to the Bitcoin network, the bookkeeper node, usually a full node in a P2P network that receives the transactions, will validate it according to Bitcoin protocol rules. If the transaction is valid, the bookkeeper will add it to the transaction pool and relay the transaction to the peers in the network. In a Bitcoin network, every 10 minutes, a subset of network nodes, called "mining nodes" or miners, will collect all valid transactions from the transaction pool and create the candidate blocks. They also create a Coinbase transaction for themselves to get rewarded and collect transaction fees, in the event they win the mining race and add the block to the chain. All nodes will verify the new block and add it to their own copy of the blockchain. Magically, Bob will be able to see the payment from Alice and 10 BTC in his wallet.
When creating a Bitcoin transaction, the wallet application has to follow the Bitcoin protocol rules and creates the transaction data structure in line with the Bitcoin specification. Invalid transactions will be rejected by the network. For details of the Bitcoin transaction and block structure, please refer to https://en.Bitcoin.it/wiki/.
- Bitcoin block structure:
The following table shows the data structure within a Bitcoin block:
- Block header structure:
The following table shows the data structure for a block header:
In particular, hashPrevBlock references the 256-bit hash value of the previous block, and hashMerkleRoot is the hash Merkle root of all transactions in the block, including coinbase transactions. And the nonce is the magic number that miners need to find so that the SHA-256 hash value of the block header is smaller than or equal to the blockchain-defined specific target.
- Transaction structure in Bitcoin:
The following screenshot shows the general data structure of a Bitcoin transaction:
A transaction can have many inputs and outputs, as specified in the field of list of inputs and list of outputs fields. The input structure is shown as follows:
The following table shows the structure for the output:
Bitcoin protocol rules define a set of validation rules, including syntactic rules and valid values. Bookkeepers, or miner nodes, need to validate transactions according to those rules before the transaction is added to the pool. It also checks the following ():
- Transaction duplication: This is to see whether we have matching transactions in the transaction pool or in a block in the main branch.
- Double spend: This is to check whether the input is used to pay concurrently in any other transactions in the pool or in the main branch.
- Orphan transaction: For each input, this checks whether we can find the reference output transaction in the main branch and the transaction pool.
- Coinbase maturity: This is to make sure coins from the coinbase transaction are mature enough to be spent.
- Overdraft: This checks the inputs and outputs to make sure there is enough to make the payment and be able to make a reasonable transaction fee.
Once a miner completes a new block with the mining, the new block will be broadcasted to the Bitcoin network for verification. Each full node, including mining nodes, will verify the new block and all transactions within the block. The same set of transaction validation rules will be applied. For block verification, all nodes check whether the block has the right cryptographic hash and the nonce makes the hash smaller than the target. The miner will add the block to the longest chain. As we discussed earlier, temporary forking may happen; a Bitcoin block tends to self-heal and only the blocks in the longest chain will stay.
Thanks to Bitcoin, blockchain technology has attracted worldwide attention. Like any new technology, it has its limitations. Notable limitations include the following:
- Scalability and throughput: Scalability is the major concern in the Bitcoin network, or in general, any PoW-based blockchain. By design, every transaction has to be verified by all nodes, and it takes about an average of 10 minutes to create a new block with the block size limited to 1 MB. Block size and frequency limitations further constrain the network's throughput.
- Transaction processing cost: Mining in the Bitcoin network is costly and energy-intensive. The miners who add new blocks to the blockchain are rewarded with Bitcoins. As Bitcoin supply is close to the maximum capacity of 21 M Bitcoins, mining becomes less profitable. Miners will more and more rely on transaction fees to offset the mining cost and make a profit. It will drastically increase the transaction cost in Bitcoin. Please refer to for real-time transaction fees in the Bitcoin network.
- Security and privacy: Bitcoin has the 51% attack issue, at least in theory, if the majority of CPU hashing power is controlled by dishonest miners. It may not be economically feasible to launch such an attack on the main Bitcoin network. But recently, at least five cryptocurrencies with much smaller networks have been hit with an attack like this. By design, all transactions are permanently stored in the Bitcoin network and can be traced to the involving parties. They are made public. This greatly improves transparency, however, unfortunately, also raises privacy concerns.
- Usability: Bitcoin uses a stack-based scripting system for transaction processing. It supports very rudimental operations and lacks the functionalities of modern programming languages. It is Turing-incomplete and inhibits the ability to build more sophisticated real-world business and payment applications.
Altcoins are cryptocurrencies other than Bitcoin. Many altcoins are variations of Bitcoin with changes and improvements created to address some particular limitations we talked in an earlier section. Some, like Ethereum and EOS, are intended as a new blockchain platform for building decentralized applications. According to http://coinmarketcap.com, the following are the top ten altcoins based on the market cap, as of Feb 17, 2019:
Ethereum: It is one of the best known smart contract platforms that enables Decentralized Applications (DApps). It was invented by Vitalik Buterin in 2013. Ether is the native currency in Ethereum platform. It goes with the symbol "ETH". We will dive into Ethereum details in the rest of this book.
XRP: Ripple (XRP) is a digital asset that enables value transfer in the Ripple network. Unlike most of the altcoins, Ripple doesn't have a blockchain. It is a Distributed Ledger Technology (DLT) based on the Ripple Consensus ledger.
Litecoin: This is almost identical to Bitcoin except that the time for adding a new block was reduced from 10 minutes to 2 minutes.
EOS: Similar to Ethereum, EOS is a blockchain platform intended to bring together the best features from different blockchain platforms and make it easy for developing DApps. It introduces some sort of centralization components into the network, as a means of addressing scalability and throughput issues.
Bitcoin Cash: This is a hard fork of the Bitcoin chain that was created because of a group of Bitcoin core developers that wanted to use a different way of addressing the scalability issue.
Vitalik Buterin, the founder of Ethereum, addressed the limitations of Bitcoin quite differently. While working on Bitcoin, he recognized that Bitcoin's stack-based scripting is very limited and lacks the functionality and capability for application development beyond the transfer of cryptocurrency ownership. He saw it as a huge opportunity and began writing his own whitepaper in 2013.
In his famous Ethereum whitepaper (https://github.com/ethereum/wiki/wiki/White-Paper), Vitalik laid out his vision and intent to build a blockchain that includes the following:
- It has a built-in Turing-complete programming language.
- It establishes a smart contract and decentralized application platform allowing anyone to define, create, and trade all types of cryptocurrencies and crypto-assets.
Similar to Bitcoin, Ethereum is built on blockchain technology. It has all of the critical characteristics of a blockchain. It is a shared distributed ledger on top of a decentralized P2P network. It works in a similar way to what we discussed in the Bitcoin and cryptocurrency section. Unlike Bitcoin, which is a decentralized state transition system with limited decentralized computing capability via Bitcoin scripting, Ethereum is a decentralized computing and data platform featuring Turing-complete smart contract functionality.
Ethereum introduced a few new and critical concepts, including a smart contract, an Ethereum Virtual Machine (EVM), and an account. We will cover them in detail in the rest of this book.
A smart contract is programming code that is stored and executed on the blockchain. Ethereum now has a Turing-complete language, Solidity, which enables developers to develop and deploy smart contracts. In addition to moving ether, the cryptocurrency in Ethereum network, between accounts, Ethereum smart contract code can support more modern program language constructs such as loops and perform much more complex computations, including data access, cryptographic algorithms, and function calls. Each such operation has a gas price associated with it. That is how Ethereum calculates the transaction cost of the smart contract and, through a gas limit, protects a smart contract from infinite loops or programming errors.
A smart contract is like a scripted agreement between interacting parties; the code built into the contract is stored on the Ethereum blockchain and cannot be tampered with or removed. This greatly increases the credibility of the legal document.
The EVM is the runtime environment for smart contracts in Ethereum. It is a virtual operating system deployed as an Ethereum client to all network nodes across the Globe. Similar to the JVM in the Java world, the contract code is compiled into the bytecode, which is loaded into EVM as part of contract creation.
There is no account concept in Bitcoin. Instead, Bitcoin uses the concept of UTXO to keep track of money transfers and account balances. Ethereum introduces the concept of the world state and account. The world state comprises a mapping of all accounts and their public addresses. To facilitate both state transactions and decentralized computing, Ethereum introduces two types of account: Externally Owned Accounts (EOAs), controlled by private keys, and contract accounts, controlled by their contract code.
In this chapter, we explained key blockchain components and elements and the different characteristics of blockchain, and we discussed how blockchain works. We reviewed cryptography technologies and how they were leveraged in the blockchain. We illustrated how PoW and PoS consensus mechanisms work. We went over key concepts in Bitcoin, as well as some Bitcoin limitations. We provided a short overview of cryptocurrencies and altcoins. We also briefly introduced Ethereum, as well as the key difference between Bitcoin and Ethereum.
We will delve into the Ethereum architecture and ecosystem in greater depth in the next two chapters. We will show you, step-by-step, how to develop and deploy decentralized applications in Sections 2 and 3 of this book. Stay tuned.