13

TLS Handshake Protocol Revisited

In previous chapters, you learned about the cryptographic primitives and mechanisms required to understand the inner workings of the TLS 1.3 handshake. Now is a good time to look at the TLS handshake from a bird’s-eye view.

In this chapter, we will zoom out of the cryptographic details and revisit how the individual steps combine in the overall scheme of things. More precisely, we will discuss the TLS handshake protocol with the help of state machines for the TLS server and TLS client specified in RFC 8446. Moreover, we will show you how you can use s˙client, a TLS client program from the popular OpenSSL toolkit, to conduct your own experiments with TLS.

Upon completion of the chapter, you will have a comprehensive understanding of how the individual protocol steps fit together, both on Alice’s and on Bob’s side. In terms of skills acquired, you will gain the following:

• A good overview of the entire TLS handshake

• Familiarity...

13.1 TLS client state machine

Appendix A of RFC 8446, the IETF specification of TLS 1.3, summarizes valid states and state transitions for TLS 1.3 server and client. The client state machine is shown in Figure 13.1. Labels in square brackets indicate actions the client performs only under specific circumstances. Label k = x indicates that the key k is set to value x.

Figure 13.1: State machine and state transitions of a TLS 1.3 client

Client Bob starts the TLS handshake by sending the ClientHello message to server Alice. If Bob and Alice have agreed upon a secret key in a previous TLS session, Bob may use this key to encrypt early data.

Bob then transitions into state WAIT˙SH, denoted by W_SH, where he waits for the ServerHello message from Alice. If Bob’s ClientHello contains parameter values that Alice does not support, she replies with a HelloRetryRequest message, thereby making Bob switch back to the initial state and re-send ClientHello with different parameters...

13.2 TLS server state machine

The server state machine is shown in Figure 13.2. Like with the client state machine, labels in square brackets indicate actions that the server performs only under specific circumstances.

Figure 13.2: State machine and state transitions of a TLS 1.3 server

For server Alice, the TLS handshake is triggered by receiving the ClientHello message from client Bob. Upon receiving this message, Alice transitions to the state RECVD˙CH, denoted by R_CH in Figure 13.2.

If Bob’s message contains parameters that Alice does not support, she replies with a HelloRetryRequest and switches into the initial state S.

Otherwise, Alice selects desired parameters from those offered by Bob in his ClientHello and replies with the ServerHello message. In addition, Alice sends EncryptedExtensions, encrypted with the handshake key.

Alice also sends the CertificateRequest message if she wants Bob to authenticate himself using his certificate. Moreover,...

13.3 Finished message

Bob’s Finished is the final message in the TLS handshake protocol. This message authenticates the handshake as well as the secret shared keys that Alice and Bob agreed upon.

Both Alice and Bob (when he receives Alice’s Finished message while in state W_F ) verify the correctness of this message’s contents and immediately terminate the TLS handshake with decrypt˙error if the verification fails.

After Alice and Bob transmitted their Finished messages and successfully verified the received Finished message, they can send and receive application data over the secure channel established using the TLS handshake.

Alternatively, Alice and Bob may transmit data before receiving the peer’s Finished message in the following situations:

• Bob sends 0-RTT data.

• Alice sends application data after sending her first flight, that is, after her ServerHello message. However, since the TLS handshake is incomplete, she has no assurance of Bob’s...

13.4 Early data

Bob has to send an EndOfEarlyData message upon receiving Alice’s Finished message if Alice sent early˙data in her EncryptedExtensions message.

If Alice did not send the early˙data in the EncryptedExtensions, then Bob does not send the EndOfEarlyData message. EndOfEarlyData indicates that all 0-RTT application˙data messages – if any were sent – were successfully transmitted and the following messages are secured using the handshake traffic keys.

13.5 Post-handshake messages

In TLS 1.3, Alice and Bob can send further messages after their main handshake. These post-handshake messages have the handshake content type and are encrypted under the corresponding application traffic key.

13.5.1 The NewSessionTicket message

Any time after receiving Bob’s Finished message, Alice can send a NewSessionTicket message. NewSessionTicket creates a unique link between the value of the ticket and the secret pre-shared key that is derived from resumption˙master˙secret (see Section 12.2, TLS secrets, in Chapter 12, Secrets and Keys in TLS 1.3, for more details).

Bob, in turn, can use this pre-shared key for future handshakes with Alice by including that ticket value in the pre˙shared˙key extension in his ClientHello message.

Alice can also send Bob multiple tickets. As an example, she could send a new ticket following the post-handshake authentication so she can encapsulate the additional client authentication...

13.6 OpenSSL s_client

OpenSSL is an open source project implementing the OpenSSL software, a commercial-grade, feature-rich toolkit for cryptography and TLS [137]. OpenSSL’s technical decision making is governed by the OpenSSL Technical Committee (OTC) and the software is published under an Apache-style license, making it suitable for both non-commercial and commercial purposes.

At the time of this writing, the latest stable OpenSSL version is the 3.1 series, which the OpenSSL project will support until March 14, 2025. In addition, OpenSSL 3.0 series is available as a Long-Term Support (LTS) version, which will be supported until September 7, 2026.

The OpenSSL source code is hosted at https://github.com/openssl/openssl. The software includes the following:

• libssl, the implementation of TLS protocol versions up to TLS 1.3

• libcrypto, a feature-rich cryptography library that is the basis for libssl, but can also be used as a standalone library

• The openssl command-line tool,...

13.7 Summary

In this chapter, we looked at the TLS handshake protocol from a bird’s-eye view, with the help of TLS client and server state machines. The state machines illustrate how the TLS protocol works on a higher level. In addition, we covered the Finished message and several post-handshake messages in more detail.

We also learned how to use OpenSSL, a popular cryptography and TLS toolkit, and, especially, the s˙client OpenSSL tool to experiment with TLS. We discussed how to install and how to use s˙client and presented several experiments that you can reproduce and enhance on your own. These experiments allow you to observe TLS at work and take a close look at cryptographic mechanisms and TLS features used in the handshake protocol.

This chapter concludes the second part of the book. In the next part, we will study the TLS record protocol. The record protocol protects application data using shared secret keys established during the TLS handshake. We will start...