10.3. IPsec Modes

As noted earlier, AH and ESP can operate in either transport mode or tunnel mode. In this section, we examine these modes and when we should use one or the other. We'll revisit these modes when we make our detailed examinations of AH (Chapter 11) and ESP (Chapter 12).

At a technical level, the two modes differ in how they encapsulate the data and thus in what data they protect. From an operational point of view, they differ in whether they are protecting communications between two hosts or protecting communications between two networks or between a host and a network.

Transport Mode

Transport mode is meant to be used between two fixed hosts, or to put it another way, when the VPN endpoints are the final destinations of the traffic in the VPN. In particular, transport mode cannot be used to connect two networks or a network and a host. Transport mode is illustrated in Figure 10.3.

The name transport mode comes from the fact that transport-mode VPNs protect the transport-layer data in the datagram.

This isn't quite accurate, because they also protect ICMP and similar protocol data. The point is that they protect data that comes after the IP header, which is most often data from a transport protocol. Looked at another way, a transport-mode VPN protects whatever protocol was originally specified in the protocol field (Figure 2.11) of the IP header.

From a practical standpoint, this means that the AH or ESP header is placed after the IP header in the datagram.

Figure 10.4 demonstrates this encapsulation. In the case of AH, of course, there is no IPsec trailer.

Figure 10.4. Transport-Mode Encapsulation

[View full size image]

The figure shows why transport mode can be used only where the VPN endpoints are the final destinations. On the one hand, the datagram must be delivered to the VPN endpoint so that it can be decrypted and/or authenticated. On the other hand, there is only one IP header, so its destination must be its final destination.

Transport-mode VPNs may seem limited, but notice that they provide end-to-end security. If our threat model doesn't trust other hosts on our or the remote network, a transport-mode VPN meets that constraint. The other type of IPsec VPN, tunnel mode, does not necessarily provide end-to-end security.

Tunnel Mode

The other mode that AH and ESP can operate in is called tunnel mode. It is more flexible than transport mode, but this flexibility comes at the expense of increased bandwidth requirements.

The typical use of tunnel mode is to connect either two networks or a host and a network: for example, a remote office network to a home office network, or a "road warrior" to the home network. Figure 10.5 shows an example of this. Note that access to the networks is through a security gateway.

Figure 10.5. Tunnel-Mode VPNs

[View full size image]

In the figure, the remote and home networks are connected through the tunnel-mode VPN by way of the security gateways, GW₁ and GW₂. These gateways handle the encryption, decryption, antireplay, and authentication functions. These security functions are completely transparent to the hosts on the two networksthey merely send datagrams to their peer on the other network just as they would if the gateways and VPN weren't there.

In the case of the remote host, the gateway function is built into the remote host itself. Everything is exactly like the two-networks situation, except that there is no network behind the remote host.

Just as transport mode is named for the fact that ESP or AH is applied to the transport data, tunnel mode is named for the fact that ESP or AH is applied to a tunnel. We can see this clearly in Figure 10.6, which illustrates tunnel-mode encapsulation. If we look closely at the figure, and compare it to Figure 10.4, we see that we can describe tunnel mode as transport mode applied to an IP-in-IP tunnel (Section 4.2).

Figure 10.6. Tunnel-Mode Encapsulation

[View full size image]

This fact explains both the name tunnel mode and how the security gateways function. The outer IP header's source and destination addresses are those of the security gateways (or the remote host and a security gateway). Thus, the VPN traffic is delivered to one of the endpoints, where the decryption and/or authentication is applied, and the outer IP header, IPsec header, and, if present, the IPsec trailer are removed. This leaves the inner IP datagram, whose IP header has the address of the final destination. The security gateway now forwards the inner IP datagram to its final destination.

The two IP headers are what allow the VPN endpoint to be different from the final destination and, ultimately, what distinguishes transport mode from tunnel mode. Notice that except for the IPsec protocol details, tunnel mode operates exactly like the tunnels that we encountered in Chapter 4.

There is no reason, of course, that we couldn't use tunnel mode instead of transport mode between two fixed hosts. In this case, the source and destination addresses of the inner and outer IP headers would be the same. The disadvantage of this is the extra bandwidth required by the additional IP header. Indeed, we could think of transport mode as an optimization of tunnel mode for the special case of two fixed hosts.

As pointed out in [Ferguson and Schneier 1999], it would be equally easy to compress the data in the inner IP header, allowing us to eliminate transport mode and the extra complexity it entails, at virtually no cost in extra bandwidth. Doing so would eliminate the need for AHbecause the inner IP header is completely protected in tunnel modeand would therefore simplify IPsec considerably.