Understanding Ethernet over MPLS Technology

EoMPLS, as specified in the draft-martini discussed in Chapter 6, allows Layer 2 Ethernet frames to be transported across a Multiprotocol Label Switching (MPLS) core network. For the label switch router (LSR) to switch Layer 2 virtual circuits (VC), it must have IP connectivity to transport any Layer 2 attachment services. Thus, the edge LSRs must have the capability to switch Layer 2 VCs. EoMPLS has several mechanisms in place to support such transport. These mechanisms are further explained in the following sections:

EoMPLS Label Stack

Most common EoMPLSs use a two-level label stack. (The VC label should always have at least one label, and the packet-switched network (PSN) label should have zero or more label stack entries.) This means that the Ethernet packets transported between the egress and ingress LSRs are sent containing two labels: the top, or outer, and the bottom, or inner. The outer label, also known as the tunnel label or Interior Gateway Protocol (IGP) label, sends packets over the MPLS backbone. The egress LSR assigns the inner label, known as the VC label (or pseudowire) label). The VC label identifies the attachment circuit on the egress LSR. The egress LSR binds the Layer 2 egress interface with the configured VC ID and sends the VC label to the ingress provider edge (PE) by using the targeted Layer Distribution Protocol (LDP) session. For more information about a targeted LDP session, review the "Establishing AToM Pseudowires" section of Chapter 6.

The next two sections describe the EoMPLS packet format and maximum transmission unit (MTU) size requirements.

Packet Format

Figure 7-1 demonstrates the EoMPLS encapsulation format, but it also applies to other forms of transport over MPLS.

The bottom VC label and the top tunnel label comprise the two levels of the label stack. The tunnel label switches packets from the ingress PE to the egress PE. The ingress LSR sets the VC label's Time to Live (TTL) field to a value of 2 (in this case), and it sets the TTL of the tunnel label to 255. To indicate that the VC label is at the bottom of the stack, the ingress PE marks the VC label's end-of-stack bit with the value of 1.

MTU Size Requirements

In the most common scenario, the two level stack entries of the EoMPLS label stack append 8 bytes to a Layer 2 frame (4 bytes each). In addition, an optional 4-byte control word is always preferred in Cisco routers. Figure 7-2 shows the overhead introduced by the addition of the tunnel and the VC labels plus the optional control word on top of the original Ethernet frame and the L2 header.

Assuming that Layer 2 in the packet-switched network (PSN) is Ethernet in Figure 7-2, the PSN Layer 2 header contains the following:

The MPLS label is set to 0x8847, which indicates that the frame carries an MPLS unicast packet. If the PSN uses a different Layer 2 technology, the upper layer identification specifies an MPLS packet, such as Cisco High-Level Data Link Control (HDLC) Type or PPP Data Link Layer protocol.

Note

The preceding fields describe the Layer 2 header in an Ethernet PSN. Do not confuse them with the Ethernet header fields in the transported Layer 2 frame from the customer edge (CE) device.

In addition, Figure 7-2 shows a label stack of 2 label stack entries (LSE), each containing the following:

Finally, Figure 7-2 shows an optional 4-byte control word and the original Ethernet frame. The original Ethernet frame's header from the CE device that is transported in EoMPLS is at least 14 bytes and contains the following:

In the case of 802.1q Ethernet VLAN transport, the Ethernet overhead is 18 bytes, with the addition of the 4-byte VLAN Tag header, also referred to as the 802.1q header. An Ethertype with a value of 0x8100 indicates that there is a VLAN Tag header between the Ethernet and upper-layer headers. The 802.1q header is as follows:

Table 7-1. Summary of Layer 2 and MPLS Components
Field	Bits	How	Comments
Destination MAC address	First 6 bytes.	FIB^[1] lookup, included in the rewrite string.	From ARP^[2].
Source MAC address	Next 6 bytes.	MAC address of the router or switch.	Field is always the same.
Ethertype	Next 2 bytes.	0x8847.	MPLS Unicast indicated by 0x8847.
Tunnel labelMPLS label	20 bits (bits 019 after MAC header).	Derived from PSN label for the remote PE's FEC^[3].	Information can be obtained through FIB lookup.
Tunnel labelEXP^[4] bits	3 bits. Bits 2022 of tunnel label.	Experimental bits in the tunnel PSN.	Intermediate LSRs can modify field when switching the packet through the PSN.
Tunnel labelS bit	1 bit. Bit 23 of tunnel label.	0.	The S bit in the Tunnel Label field is always set to 0 for the tunnel label to indicate that another LSE^[5] follows.
Tunnel labelTTL field	8 bits. Bits 2431 of tunnel label.	Initially set to 255 or can be derived from the transported IP header.	This field is decremented in intermediate LSRs by means of TTL processing.
VC labelMPLS label	20 bits. Bits 019 after tunnel label.	Derived from the incoming packet's VLAN tag and the ingress port (that is, incoming attachment circuit).	VC label is associated with the egress attachment circuit and advertised through LDP. You obtain this information from the FIB lookup.
VC labelEXP bits	3 bits. Bits 20-22 of VC label.	Same as the tunnel EXP bits.	You can configure this field.
VC labelS bit	1 bit. Bit 23 of VC label.	1.	Because the VC label is the last label in the MPLS label stack, the S bit is always set to 1.
VC labelTTL field	8 bits. Bits 2431 of VC label.	2.	This field is always set to 2.

You can see a sample EoMPLS packet showing the fields from Example 7-1. The EoMPLS packet over Ethernet PSN that contains a two-label stack and control word transports an 802.1q VLAN frame that contains an Internet Control Message Protocol (ICMP) echo request (ping) sent from a CE device. The different layers of protocols are highlighted.

Example 7-1. EoMPLS Frame Sample (Ethernet Frame)

The overhead incurred when Ethernet frames were transported over MPLS and rules and restrictions were imposed on the MPLS network.

For instance, the MTU configuration of the MPLS network should accommodate the largest expected frame size in the label-switched paths (LSPs) plus the header (Ethernet frame header, in this case), control word, and 8 additional label stack bytes. This includes configuration of the CE and PE links.

Figure 7-3 illustrates a sample network over which you can see the MTU calculation for VLAN-tunneled modes. (Both modes are discussed in the "Supported VC Types" section of this chapter.) To verify these calculations, you perform pings with different packet sizes from the CE R200 to the CE R204.

Using the network depicted in Figure 7-3, you can see in Example 7-2 the different results when sending pings with the don't fragment (DF) bit set from R200 to R204, with sizes of 1470 and 1471 bytes, respectively.

Example 7-2. Probing for MTU Limits in EoMPLS

This formula uses the following values for VLAN transport and a two-label stack and provides the core MTU needed to transport 1500-byte packets from the CE:

You input the value of 1470 bytes in the preceding formula as the edge MTU because that is the largest unfragmented packet that was successfully transported. The result of the formula is a core MTU that is greater than or equal to 1500 bytes, which is the actual MTU that is configured in the core.

On the other hand, if you want to transport 1500-byte packets from the CE device, you can substitute that value for the Edge MTU in the general formula to calculate the corresponding Core MTU needed:

In this case, you need to configure the MTU links to allow for 1530-byte packets.

Table 7-2 outlines the MTU calculation to show that the overhead is 30 bytes. That is why only packets that are up to 1470 bytes with DF bit set are successfully transported in Example 7-2.

Table 7-2. Calculating MTU Requirements for Ethernet VLAN Transport
Layer	Description	Core Overhead
Transported	Ethernet VLAN	18 bytes
AToM	Control word	4 bytes
MPLS	MPLS stack entries * MPLS header size	2 headers * 4 bytes/header = 8 bytes
Total		30 bytes

Keeping in mind that the transport overhead for VLAN-tunneled is 18 bytes, the transport overhead for port-tunneled is 14 bytes, and that MPLS traffic engineering (TE) fast reroute (FRR) uses an additional label stack entry, you can see the MTU calculations for various cases in Table 7-3. (All values are in bytes.) Note that the sizes in square brackets indicate the values when the optional control word is not used.

Table 7-3. Calculating MTU Requirements for EoMPLS Cases
Field	Edge	Transport	AToM (Control Word)	MPLS Stack	MPLS Header	MPLS	Total
EoMPLS Port Mode	1500	14	4 [0]	2 LSEs	4 bytes/LSE	8	1526 [1522]
EoMPLS VLAN Mode	1500	18	4 [0]	2 LSEs	4 bytes/LSE	8	1530 [1526]
EoMPLS Port + TE FRR	1500	14	4 [0]	3 LSEs	4 bytes/LSE	8	1530 [1526]
EoMPLS VLAN + TE FRR	1500	18	4 [0]	3 LSEs	4 bytes/LSE	12	1534 [1530]

Under certain circumstances, the egress label edge router (LER) might receive a packet that exceeds the MTU of the egress interface. For instance, a PE might receive a jumbo frame that exceeds the default 1500 MTU in the core. The core must be able to transport larger frame bytes.

If the MTU is not increased between the PE and P routers and the encapsulated packet on the ingress PE exceeds the LSP MTU, the packet is dropped on the PE. Likewise, if a packet arrives at the egress LSR that exceeds the MTU of the egress Layer 2 interface (VLAN), it is dropped. In addition, the MPLS backbone does not support fragmentation of the Layer 2 packets. Therefore, the MTU of all intermediate circuits must be able to handle the biggest Layer 2 packet that is transported.

The MTU configuration of the ingress and egress PEs needs to match. Otherwise, a VC negotiation occurs, but it is rejected and the circuit fails to come up. Resolving this issue requires configuring the same MTU parameters on each side. To avoid packet drops in the core, the mpls mtu number on interfaces must be increased on P and PE routers to meet requirements.

Supported VC Types

As mentioned in the previous section, EoMPLS can operate in two modes: port-tunneling mode and VLAN-tunneling mode. Port-tunneling is also referred to as port-to-port transport. VLAN-tunneled interfaces are VC type 4, or, 0x0004, and port-tunneled interfaces are VC type 5, or 0x0005 (as specified in the draft-martini). Cisco devices support both VC types.

Note

The newest Cisco implementations provide for auto-sensing of the VC type.

In VLAN-tunneling mode, the ingress information for the VLAN is contained within the dot1Q header of the packet. (Refer to Chapter 4, "LAN Protocols," for more information on dot1Q.) By looking at the VLAN ID in the dot1Q header, the network processor (NP) can determine the next step in processing, described in the "Label Imposition" section of this chapter.

In port-tunneling mode, the packet does not have ingress port information. For inclusion of ingress information, the port-tunneled interface is put into the QinQ mode. A hidden VLAN is then created and added onto the packet. A hidden VLAN is a VLAN that is numbered outside the allowed range for VLAN IDs. This is how the NP learns the ingress information. The hidden VLAN concept applies to a switch platform (that is, 6500 and 7600 platforms). In contrast, VLAN-tunneled mode does not require a hidden VLAN. The NP can discern the ingress information from the packet's dot1Q header.

A similar concept applies to routers (VLAN stacking method). It involves the use of subinterfaces.

Another difference between the port-tunneling mode and VLAN-tunneling mode is in the handling of VLAN IDs. In port-tunneling mode, the VLAN ID is transparently passed from the ingress PE to the egress PE over MPLS in a single VLAN. In VLAN-tunneling mode, however, the VLAN ID at each end of the EoMPLS tunnel can be different. To overcome this, the egress side of the tunnel that is mapped to a VLAN rewrites the VLAN ID in outgoing dot1Q packets to the ID of the local VLAN.

Label Imposition

Earlier in this chapter, you learned about the format of the EoMPLS label stack. Adding these labels onto a packet is referred to as label imposition. You perform label imposition on the label imposition router or an ingress PE. Routers receive a Layer 2 packet and encapsulates it for the MPLS backbone.

Depending on whether the port-tunneling or VLAN-tunneling mode is used, the interfaces that receive the Layer 2 packets can be either Ethernet port interfaces or VLAN interfaces (or subinterfaces). To impose labels on packets, the ingress PE router maintains a table, which associates an EoMPLS tunnel with an interface/FEC. This table keeps the information needed for sending the packet, such as the outgoing interface and encapsulation.

A two-level label stack, described earlier in the "EoMPLS Label Stack" section of this chapter, is learned via the LDP for each VC. An NP also provides a MAC rewrite that contains the outgoing Layer 2 encapsulation in addition to the label stack that changes the encapsulation of the packet to comply with the outgoing interface.
The label stack and the output MAC encapsulation are prepended to the packet, and the packet is forwarded to the egress interface for transmission.
The outgoing VC label's TTL field is set to 2.

Label Disposition

Label disposition refers to label removal from a packet. It is performed at the egress PE, or the label disposition router. This router receives a packet, strips the bottom (VC) label (and the top [PSN] label if it exists, in cases such as explicit-null), and sends the remaining Layer 2 frame out of the egress attachment circuit interface.

After label imposition, the packets travel across the MPLS core network via standard label switching and arrive at the egress PE. At that point, they might already have their tunnel label removed and be left with only the VC label. This might have occurred because the next to last (penultimate) router "popped" the tunnel label prior to transmitting the packet to the egress PE. This process is commonly referred to as Penultimate Hop Popping (PHP). If hop popping is supported at the penultimate router, the egress PE requests it. It advertises an implicit null label to its directly connected neighbor via BGP, which causes the neighbor to pop the tunnel label when switching the packets to the egress PE.

When the egress PE receives the packet with the VC label, it needs to select an appropriate form of disposition. For this, the egress PE checks the label forwarding information base (LFIB). The LFIB contains information about the binding between the outgoing interface and a given VC ID, which was initially inserted into the LFIB with the VC label. The LFIB lookup informs the PE that EoMPLS disposition will be performed and finds the corresponding egress interface for the VC. The VC label is then popped, the VLAN ID is rewritten (if needed), and the frame is transmitted to the proper outgoing interface.

Note

The ingress and egress PEs are the only two routers in the MPLS backbone with knowledge of the Layer 2 transport VCs. No other intermediate hops have table entries for the Layer 2 transport VCs. Therefore, only PEs require EoMPLS functionality.

Figure 7-4 illustrates the process of imposition and disposition, where traffic flow is bound first from Site 1 to Site 2 and then in the opposite direction.

The traffic flow over an EoMPLS VC, between the imposition and disposition PEs, follows the same path across the MPLS backbone, unless routing changes within the network of the provider cause the change.

Note

For an LSP to be present from PE to PE, routes from a PE that its neighbors discover cannot be summarized. They must have a mask of /32.