Common Troubleshooting TechniquesThis section introduces you to the most common troubleshooting techniques for PEs in EoMPLS. You first learn the commands and outputs for the Cisco router PEs and then learn to troubleshoot Cisco 7600 series switches. Troubleshooting EoMPLS on RoutersThe first common step in troubleshooting problems is attempting to discover failure by verifying the status of a VC by issuing the show mpls l2transport vc command. Three conditions must be met so that the VC is UP:
If the status field is marked DOWN (that is, the VC is not ready to carry traffic between the two VC endpoints), as shown in the output of Example 7-34, execute the show mpls l2transport vc detail command seen in Example 7-35 for more in-depth information. Example 7-34. show mpls l2transport vc CommandNewYork#show mpls l2transport vc Local intf Local circuit Dest address VC ID Status ------------- ----------------------- --------------- ---------- ---------- Et0/0 Ethernet 192.168.1.102 100 DOWN Example 7-35. show mpls l2transport vc detail CommandNewYork#show mpls l2transport vc detail Local interface: Et0/0 up, line protocol up, Ethernet up Destination address: 192.168.1.102, VC ID: 100, VC status: down Preferred path: not configured Default path: active Tunnel label: 16, next hop point2point Output interface: Se5/0, imposed label stack {16 16} Create time: 00:18:10, last status change time: 00:03:51 Signaling protocol: LDP, peer 192.168.1.102:0 up MPLS VC labels: local 16, remote 16 Group ID: local 0, remote 0 MTU: local 1500, remote unknown Remote interface description: Sequencing: receive disabled, send disabled VC statistics: packet totals: receive 0, send 0 byte totals: receive 0, send 0 packet drops: receive 0, send 78 Table 7-4 describes some of the significant fields of the show mpls l2transport vc detail command output.
In the show mpls l2transport vc detail command output, pay attention to the remote unknown next to the local MTU. One of the possible causes is a remote interface down or an MTU mismatch. Verify to make sure that MTU on each side is the same. If an EoMPLS tunnel is still down after this and you cannot pass traffic, perform another check by issuing the show mpls forwarding-table command, as demonstrated in Example 7-36. Example 7-36. show mpls forwarding-table CommandNewYork#show mpls forwarding-table Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface 17 Untagged 10.1.1.0/24 0 Se5/0 point2point 18 Untagged 192.168.1.101/32 0 Se5/0 point2point 19 Untagged 192.168.1.102/32 0 Se5/0 point2point 20 Untagged l2ckt(100) 4592 Et0/0.100 point2point The Untagged result in the Outgoing tag or VC field indicates that an MPLS label might not be exchanged between the PE and P (Denver) routers. A couple possible causes exist. Either the mpls ip has not been enabled per interface, or CEF is disabled (or not enabled) on the P or PE router. To verify, issue the show mpls ldp discovery command, as in Example 7-37. Example 7-37. show mpls ldp discovery CommandNewYork#show mpls ldp discovery Local LDP Identifier: 192.168.1.103:0 Discovery Sources: Targeted Hellos: 192.168.1.103 -> 192.168.1.102 (ldp): active/passive, xmit/recv LDP Id: 192.168.1.102:0 The output shows whether you have a direct LDP session open between directly connected MPLS-enabled interfaces. By observing this behavior, you can conclude that MPLS indeed was not enabled per interface facing the core. To solve this problem, enable MPLS on an interface or check whether CEF is enabled. Two common issues result when the circuit does not come up:
Examples 7-38 and 7-39 display the output of the show mpls l2transport vc vciddetail command with the two conditions, respectively. Example 7-38. Remote Port Down or Not ConfiguredNewYork#show mpls l2transport vc 10 detail Local interface: FastEternet0/0.10 up, line protocol up, Eth VLAN 10 up Destination address: 192.168.1.102, VC ID: 10, VC status: down Tunnel label: not ready Output interface: unknown, imposed label stack {} Create time: 22:31:53, last status change time: 04:02:56 Signaling protocol: LDP, peer 192.168.1.102:0 up MPLS VC labels: local 19, remote unassigned Group ID: local 0, remote unknown MTU: local 1500, remote unknown Remote interface description: Sequencing: receive disabled, send disabled VC statistics: packet totals: receive 1650, send 1743 byte totals: receive 552557, send 550044 packet drops: receive 0, send 7 Example 7-39. MTU MismatchNewYork#show mpls l2transport vc 10 detail Local interface: FastEternet0/0.10 up, line protocol up, Eth VLAN 10 up Destination address: 192.168.1.102, VC ID: 10, VC status: down Tunnel label: not ready Output interface: unknown, imposed label stack {} Create time: 22:36:10, last status change time: 00:00:20 Signaling protocol: LDP, peer 192.168.1.102:0 up MPLS VC labels: local 19, remote 21 Group ID: local 0, remote 0 MTU: local 1500, remote 1000 Remote interface description: *** To SanFran *** Sequencing: receive disabled, send disabled VC statistics: packet totals: receive 1880, send 1901 byte totals: receive 168476, send 155436 packet drops: receive 0, send 13 The highlighted portions of Examples 7-38 and 7-39 call attention to the faulty conditions. Compare the output to output of the same command in an operational environment from Example 7-40. Example 7-40. Working ExampleNewYork#show mpls l2transport vc 10 detail Local interface: FastEternet0/0.10 up, line protocol up, Eth VLAN 10 up Destination address: 192.168.1.102, VC ID: 10, VC status: up Preferred path: not configured Default path: active Tunnel label: 17, next hop 10.1.1.202 Output interface: Et1/0, imposed label stack {17 21} Create time: 23:06:37, last status change time: 00:30:47 Signaling protocol: LDP, peer 192.168.1.102:0 up MPLS VC labels: local 19, remote 21 Group ID: local 0, remote 0 MTU: local 1500, remote 1500 Remote interface description: *** To SanFran *** Sequencing: receive disabled, send disabled VC statistics: packet totals: receive 1683, send 1777 byte totals: receive 565455, send 563328 packet drops: receive 0, send 7 Example 7-41 presents the verification and configuration sequence of enabling MPLS on the Serial 5/0 interface. Example 7-41. Configuring MPLS on an InterfaceNewYork#show mpls interfaces Interface IP Tunnel Operational Se5/0 No Yes No configure terminal interface serial 5/0 mpls ip NewYork#show mpls interfaces Interface IP Tunnel Operational Se5/0 Yes Yes Yes Next, check whether MPLS is directly receiving labels and exchanging LDP between the PE and the P by reissuing the show mpls forwarding-table command. The output is provided in Example 7-42. Example 7-42. show mpls forwarding-table CommandNewYork#show mpls forwarding-table Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface 17 Pop tag 10.1.1.0/24 0 Se5/0 point2point 18 Pop tag 192.168.1.101/32 0 Se5/0 point2point 19 16 192.168.1.102/32 0 Se5/0 point2point 20 Untagged l2ckt(100) 5310 Et0/0.100 point2point To ensure that LDP xmit/recv (transmit/receive) is occurring in both directions, use the show mpls ldp discovery command again, as in Example 7-43. Example 7-43. show mpls ldp discovery CommandNewYork#show mpls ldp discovery Local LDP Identifier: 192.168.1.103:0 Discovery Sources: Interfaces: Serial5/0 (ldp): xmit/recv LDP Id: 192.168.1.101:0 Targeted Hellos: 192.168.1.103 -> 192.168.1.102 (ldp): active/passive, xmit/recv LDP Id: 192.168.1.102:0 Example 7-44 shows that the VC is now ready and operational and should be able to send traffic from CE to CE. Example 7-44. show mpls l2transport vc Command
NewYork#show mpls l2transport vc
Local intf Local circuit Dest address VC ID Status
------------- ----------------------- --------------- ---------- ----------
Et0/0 Ethernet 192.168.1.102 100 UP
Verify with the show mpls l2transport vc detail command output (shown in Example 7-45) that the packets are being sent and received. Example 7-45. show mpls l2transport vc detail CommandNewYork#show mpls l2transport vc detail Local interface: Et0/0 up, line protocol up, Ethernet up Destination address: 192.168.1.102, VC ID: 100, VC status: up Preferred path: not configured Default path: active Tunnel label: 16, next hop point2point Output interface: Se5/0, imposed label stack {16 16} Create time: 00:18:10, last status change time: 00:03:51 Signaling protocol: LDP, peer 192.168.1.102:0 up MPLS VC labels: local 16, remote 16 Group ID: local 0, remote 0 MTU: local 1500, remote 1500 Remote interface description: Sequencing: receive disabled, send disabled VC statistics: packet totals: receive 56, send 56 byte totals: receive 6019, send 6014 packet drops: receive 0, send 78 Debugging EoMPLS Operation on PE RoutersYou can use several useful debugging commands to verify and troubleshoot EoMPLS operation on PE routers. For dot1Q operation, look for VC type 4 in output generated by the debug mpls l2transport signaling message command, as shown in Example 7-46. Example 7-46. Debugging dot1QNewYork#debug mpls l2transport signaling message NewYork(config)#interface ethernet 0/1.100 NewYork(config-subif)#shutdown 00:19:51: AToM LDP [192.168.1.102]: Sending label withdraw msg vc type 4, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 1500 00:19:51: AToM LDP [192.168.1.102]: Received label release msg, id 78 vc type 4, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 0 NewYork(config-subif)#no shutdown 00:21:56: AToM LDP [192.168.1.102]: Sending label mapping msg vc type 4, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 1500 In troubleshooting the port-based EoMPLS operation, look for VC type 5 in the debug mpls l2transport signaling message command output, as shown in Example 7-47. Example 7-47. Debugging Port to PortNewYork#debug mpls l2transport signaling message AToM LDP message debugging is on ! NewYork(config)#interface ethernet 0/0 NewYork(config-if)#shutdown 00:08:39: AToM LDP [192.168.1.102]: Sending label withdraw msg vc type 5, cbit 1, vc id 100, group id 0, vc label 16, status 0, mtu 1500 00:08:39: AToM LDP [192.168.1.102]: Received label release msg, id 34 vc type 5, cbit 1, vc id 100, group id 0, vc label 16, status 0, mtu 0 00:08:41: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down NewYork(config-if)#no shutdown 00:08:42: AToM LDP [192.168.1.102]: Sending label mapping msg vc type 5, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 1500 00:08:44: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up 00:08:45: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up Another helpful command is debug acircuit event for information on all attachment circuits, as illustrated in Example 7-48. Example 7-48. debug acircuit event CommandNewYork#debug acircuit event NewYork(config)#int e 0/0 NewYork(config-if)#no shutdown 00:12:59: ACLIB [192.168.1.102, 100]: SW AC interface UP for Ethernet interface Et0/0 00:12:59: ACLIB [192.168.1.102, 100]: pthru_intf_handle_circuit_up() calling acmgr_circuit_up 00:12:59: ACLIB [192.168.1.102, 100]: Setting new AC state to Ac-Connecting 00:12:59: ACLIB: Update switching plane with circuit UP status 00:12:59: ACLIB [192.168.1.102, 100]: SW AC interface UP for Ethernet interface Et0/0 00:12:59: ACLIB [192.168.1.102, 100]: pthru_intf_handle_circuit_up() ignoring up event. Already connected or connecting. 00:12:59: Et0/0 ACMGR: Receive <Circuit Up> msg 00:12:59: Et0/0 ACMGR: circuit up event, SIP state chg fsp up to connected, action is p2p up forwarded 00:12:59: ACLIB: pthru_intf_response hdl is 8C000002, response is 2 00:12:59: ACLIB [192.168.1.102, 100]: Setting new AC state to Ac-Connected 00:12:59: AToM LDP [192.168.1.102]: Sending label mapping msg vc type 5, cbit 1, vc id 100, group id 0, vc label 16, status 0, mtu 1500 00:12:59: Et0/0 ACMGR: Rcv SIP msg: resp peer-to-peer msg, hdl 8C000002, sss_hdlB4000003 00:12:59: Et0/0 ACMGR: remote up event, SIP connected state no chg, action is ignore 00:13:01: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up 00:13:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up Use the debug mpls l2transport vc event command to see the AToM event messages about the VCs, as shown in Example 7-49. Watch how the messages reflect the shutdown of the interface and the healthy recovery. Example 7-49. debug mpls l2transport vc event CommandNewYork#debug mpls l2transport vc event NewYork(config)#interface ethernet 0/0 NewYork(config-if)#shutdown 00:14:37: AToM MGR [192.168.1.102, 100]: Local end down, vc is down 00:14:37: AToM MGR [192.168.1.102, 100]: Unprovision SSM segment 00:14:37: AToM SMGR: Submit Imposition Update 00:14:37: AToM SMGR: Submit Disposition Update 00:14:37: AToM SMGR [192.168.1.102, 100]: Event Imposition Disable 00:14:37: AToM SMGR [192.168.1.102, 100]: State [Imposition/Disposition Rdy-> Disposition Rdy] 00:14:37: AToM SMGR [192.168.1.102, 100]: Event Disposition Disable 00:14:37: AToM SMGR [192.168.1.102, 100]: State [Disposition Rdy->Provisioned] 00:14:37: AToM SMGR: Submit SSM event 00:14:37: AToM SMGR: Event SSM event 00:14:37: AToM SMGR [192.168.1.102, 100]: sucessfully teardown sss switch for pwid 5A000000 00:14:37: AToM SMGR [192.168.1.102, 100]: sucessfully processed ssm unprovisioning for pwid 5A000000 00:14:39: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down 00:14:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to down NewYork(config-if)#no shutdown 00:15:16: AToM MGR [192.168.1.102, 100]: Local end up 00:15:16: AToM MGR [192.168.1.102, 100]: Validate vc, activating data plane 00:15:16: AToM SMGR: Submit Imposition Update 00:15:16: AToM SMGR: Submit Disposition Update 00:15:16: AToM SMGR [192.168.1.102, 100]: Event Imposition Enable, imp-ctrlflag 83, remote vc label 16 00:15:16: AToM SMGR [192.168.1.102, 100]: Imposition Programmed, Output Interface: Se5/0 00:15:16: AToM SMGR [192.168.1.102, 100]: State [Provisioned->Imposition Rdy] 00:15:16: AToM SMGR [192.168.1.102, 100]: Event Disposition Enable, disp-ctrlflag 3, local vc label 16 00:15:16: AToM SMGR [192.168.1.102, 100]: State [Imposition Rdy->Imposition/ Disposition Rdy] 00:15:16: AToM SMGR: Submit SSM event 00:15:16: AToM SMGR: Event SSM event 00:15:16: AToM SMGR [192.168.1.102, 100]: sucessfully processed ssm provision request pwid 5A000000 00:15:16: AToM SMGR [192.168.1.102, 100]: Send COMPLETE signal to SSM 00:15:16: AToM SMGR [192.168.1.102, 100]: sucessfully setup sss switch for pwid 5A000000 00:15:16: AToM SMGR: Submit SSM event 00:15:16: AToM SMGR: Event SSM event 00:15:16: AToM SMGR [192.168.1.102, 100]: sucessfully processed ssm bind for pw id 5A000000 00:15:16: AToM MGR [192.168.1.102, 100]: Receive SSM dataplane up notification 00:15:16: AToM MGR [192.168.1.102, 100]: Dataplane activated 00:15:18: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up 00:15:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up To verify the flow of packets across the Layer 2 tunnel, use the debug mpls l2transport packet data command, as shown in Example 7-50. Caution Make sure to use this testing technique in a lab environment only. Example 7-50. debug mpls l2transport packet data CommandNewYork#debug mpls l2transport packet data 00:17:44: ATOM disposition: in Se5/0, size 60, seq 0, control word 0x0 00:17:44: 00 00 0C 00 6C 00 00 00 0C 00 6C 00 90 00 00 00 ^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^ ^^^^^ Dest. Address Source Address Etype = 0x9000 = LOOP 00:17:44: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:17:44: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:17:44: 00 00 00 00 00 00 00 00 00 00 00 00 00:17:44: ATOM disposition: in Se5/0, size 114, seq 0, control word 0x0 00:17:44: 00 00 0C 00 6F 00 00 00 0C 00 6C 00 08 00 45 00 ^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^ ^^^^^ ^^^^^... Dest. Address Source Address | Begins IP packet Note Note that in Example 7-50, the offline hand decoding of the packets is shown in bold. You can see in Example 7-50 that the disposition packets show only the EoMPLS Payload, whereas the imposition packets also include the EoMPLS header. In the first packet, the Ethertype of 0x9000 indicates a loopback packet. For this reason, the source and destination MAC addresses are the same. The second packet shows an IP packet transported in EoMPLS. Finally, in the third packet with the imposition operation, you can also see the Tunnel, MPLS, and AToM headers. Specifically, the Layer 2 is Cisco HDLC with an HDLC type of 0x8847, indicating that MPLS follows. A two-level MPLS stack includes the Tunnel label of 16 and a VC label of 16. Note that these two values do not need to be the same. The label stack is followed by a 4-byte control word and finally an Ethernet frame with an Ethertype of 0x0800 transporting an IP datagram with an ICMP packet. Troubleshooting EoMPLS on SwitchesTroubleshooting commands on switches are, for the most part, the same as those on routers. However, the output might be different, as you learn in this section. For instance, use the output of the command show mpls l2transport vc from Example 7-51 to get information about the VCs just as you would use this command on routers. Example 7-51. show mpls l2transport vc Command on a Switch
Metro-switch#show mpls l2transport vc
Transport Client VC Trans Local Remote Tunnel
VC ID Intf State Type VC Label VC Label Label
200 Vl200 UP vlan 215 215 implc-null
Table 7-5 explains the most pertinent fields of the show mpls l2transport vc command.
For more in-depth information about the VCs, use the show mpls l2transport vc detail command, as shown in Example 7-52. Example 7-52. show mpls l2transport vc detail Command
Metro-switch# show mpls l2transport vc detail
vcid: 200, local groupid: 24, remote groupid: 102 (vc is up)
client: Vl200 is up, destination: 1.2.2.1, Peer LDP Ident: 1.2.1.1:0
local label: 215, remote label: 215, tunnel label: implc-null
outgoing interface: s0/0, next hop: point2point
Local MTU: 1500, Remote MTU: 1500
Remote interface description: Vlan 200
imposition: LC Programmed
current imposition/last disposition slot: 2/32
Packet totals(in/out): 6246/6159
byte totals(in/out): 536999/444722
Table 7-6 describes the various output fields.
|