12.2. The ESP Header

The format of the ESP packet is shown in Figure 12.1.

Figure 12.1. The ESP Header and Trailer

[View full size image]

As with AH, the SPI, the destination address, and the IPsec protocol are used to uniquely identify the SA that applies to this packet. Also as with AH, the sequence number is used to provide the antireplay function. When the SA is established, the sequence number is initialized to 0. Before each packet is sent, the sequence number is incremented by 1 and placed in the ESP header. To ensure that no packet will be accepted more than once, the sequence number is not allowed to wrap to 0. Once the sequence number 2³² - 1 is used, a new SA and, except in the case of manual keying, a new authentication key are established.

As we saw in Chapter 3, some encryption algorithms require an initialization vector, especially for block ciphers used in CBC mode. When an explicit IV is required, it is included in the payload data. In principle, we could send the IV in the first datagram and let the receiver cache the most recent encrypted block for use with the next block's CBC operation. In practice, the unreliable delivery of IP datagrams makes this impractical, and IPsec requires that an IV be sent with each IP datagram [Madson and Doraswamy 1998, Pereira and Adams 1998]. This makes sense because it makes the protocol simpler and allows the receiver to decrypt packets even if they arrive out of order or are lost, a common occurrence. The IV, if needed, and the payload data are placed in the IV and payload data field.

As we have seen, block ciphers require that plaintext be padded to a multiple of the block size. Such padding, if needed, is placed immediately after the payload data in the padding field. Even if a stream cipher or NULL encryption is used, we may require padding for alignment or data-hiding purposes. For example, the next header field must be right aligned on a 4-byte boundary, as shown in Figure 12.1, so that the authentication data will start on a 4-byte boundary. It is also possible to add a random number of padding bytes to hide the length of the payload data.

In any event, 0 to 255 bytes of padding are added to the payload data. Unless the encryption algorithm specifies otherwise, the first padding byte must be 0x01, the second 0x02, and so on. RFC 2406 says that the receiver should inspect the padding bytes to verify that they meet the prescribed values. This check serves to verify that the decryption was successful and provides a small amount of protection against cut-and-paste attacks when authentication is not used [Doraswamy and Harkins 1999].

The length of the padding is in the pad length field. It can take on any value between 0 and 255 inclusive. The pad length field is always present, even if there is no padding.

The next header field indicates what type of data is in the IV and payload data field. We shall see how the next header field is used and some of its common values shortly.

The authentication data field contains an integrity check value for the ESP packet. The ICV is calculated over the entire ESP packet except for the authentication data field itself. The ICV must start on a 4-byte boundary and must be a multiple of 32-bit words.

The two most common authentication methods are HMAC-MD5-96 [Madson and Glenn 1998a] and HMAC-SHA1-96 [Madson and Glenn 1998b]. Each method takes the first 96 bits from the HMAC-MD5 or HMAC-SHA-1 algorithm (described in Chapter 3) as the ICV. Although it is counterintuitive, restricting the output of the HMACs to the first 96 bits increases the security of the HMAC because it gives an attacker less information to work with. This is discussed in RFC 2104 ([Krawczyk, Bellare, and Canetti 1997]) and [Bellare, Canetti, and Krawczyk 1996].

For what follows, it is convenient to think of the ESP packet as consisting of four parts:

We will refer to these fields in the rest of this chapter.