Troubleshooting Guide
Page 1
Cisco Nexus 5000 Series Troubleshooting Guide December 15, 2011 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 Text Part Number: OL-25300-01 Send document comments to nexus5k-docfeedback@cisco.com.
Cisco Nexus 5000 Series Troubleshooting Guide December 15, 2011 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 Text Part Number: OL-25300-01 Send document comments to nexus5k-docfeedback@cisco.com.
Troubleshooting Guide
Page 2
...examples, command display output, network topology diagrams, and other figures included in the U.S. Cisco Nexus 5000 Series Troubleshooting Guide © 2011 Cisco Systems, Inc. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH..., Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Lumin, Cisco Nexus, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, Continuum, EtherFast, EtherSwitch, Event Center, Explorer, Follow...
...examples, command display output, network topology diagrams, and other figures included in the U.S. Cisco Nexus 5000 Series Troubleshooting Guide © 2011 Cisco Systems, Inc. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH..., Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Lumin, Cisco Nexus, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, Continuum, EtherFast, EtherSwitch, Event Center, Explorer, Follow...
Troubleshooting Guide
Page 3
... Layer 2 Switching Issues 3-1 MAC Address Table 3-1 Spanning Tree Protocol 3-3 Multicast 3-5 VLANs 3-6 Registers and Counters 3-10 Troubleshooting QoS Issues 4-1 Policy Maps 4-1 Improper Configurations 4-2 PFC 4-8 Registers and Counters 4-10 Troubleshooting SAN Switching Issues 5-1 Overview 5-1 NPV 5-2 Zoning 5-7 SAN Port Channels 5-13 FC Services 5-16 Cisco Fabric Services 5-31 VSANs 5-40 Registers and Counters 5-48 Cisco Nexus 5000 Series Troubleshooting Guide...
... Layer 2 Switching Issues 3-1 MAC Address Table 3-1 Spanning Tree Protocol 3-3 Multicast 3-5 VLANs 3-6 Registers and Counters 3-10 Troubleshooting QoS Issues 4-1 Policy Maps 4-1 Improper Configurations 4-2 PFC 4-8 Registers and Counters 4-10 Troubleshooting SAN Switching Issues 5-1 Overview 5-1 NPV 5-2 Zoning 5-7 SAN Port Channels 5-13 FC Services 5-16 Cisco Fabric Services 5-31 VSANs 5-40 Registers and Counters 5-48 Cisco Nexus 5000 Series Troubleshooting Guide...
Troubleshooting Guide
Page 4
Contents Send document comments to nexus5k-docfeedback@cisco.com. Troubleshooting Security Issues 6-1 Roles 6-1 AAA 6-4 Troubleshooting System Management Issues 7-1 SNMP 7-1 Logging 7-3 Traps 7-4 DNS 7-4 Troubleshooting Virtual Port Channel Issues 8-1 Improper Configurations 8-1 Troubleshooting Config-Sync Issues 9-1 Commit Failure 9-1 Import Failure 9-3 Merge Failure 9-5 Switch-profile Deletion Failure 9-6 Verify Failure 9-8 Cisco Nexus 5000 Series Troubleshooting Guide iv OL-25300-01
Contents Send document comments to nexus5k-docfeedback@cisco.com. Troubleshooting Security Issues 6-1 Roles 6-1 AAA 6-4 Troubleshooting System Management Issues 7-1 SNMP 7-1 Logging 7-3 Traps 7-4 DNS 7-4 Troubleshooting Virtual Port Channel Issues 8-1 Improper Configurations 8-1 Troubleshooting Config-Sync Issues 9-1 Commit Failure 9-1 Import Failure 9-3 Merge Failure 9-5 Switch-profile Deletion Failure 9-6 Verify Failure 9-8 Cisco Nexus 5000 Series Troubleshooting Guide iv OL-25300-01
Troubleshooting Guide
Page 5
...-01 Cisco Nexus 5000 Series Troubleshooting Guide v Organization This reference is for problems that can occur with Layer 2 switching in the Cisco Nexus 5000 Series switch. Troubleshooting Layer 2 Switching Issues Describes how to identify and resolve problems that can occur with FCoE in the Cisco Nexus 5000 Series switch. Troubleshooting FCoE Issues Describes how to identify and resolve problems that can occur with QoS in the Cisco Nexus 5000 Series switch...
...-01 Cisco Nexus 5000 Series Troubleshooting Guide v Organization This reference is for problems that can occur with Layer 2 switching in the Cisco Nexus 5000 Series switch. Troubleshooting Layer 2 Switching Issues Describes how to identify and resolve problems that can occur with FCoE in the Cisco Nexus 5000 Series switch. Troubleshooting FCoE Issues Describes how to identify and resolve problems that can occur with QoS in the Cisco Nexus 5000 Series switch...
Troubleshooting Guide
Page 6
...Document Conventions Command descriptions use these conventions: screen font Terminal sessions and information that the switch displays are in the Cisco Nexus 5000 Series switch. Elements in boldface. Caution Means reader be careful. In this situation, you might do... are grouped in equipment damage or loss of characters. Describes how to identify and resolve problems that can occur with system management and the Cisco Nexus 5000 Series switch. Arguments for which you must enter is in the manual. Do not use these conventions: Convention boldface font italic font [ ] [...
...Document Conventions Command descriptions use these conventions: screen font Terminal sessions and information that the switch displays are in the Cisco Nexus 5000 Series switch. Elements in boldface. Caution Means reader be careful. In this situation, you might do... are grouped in equipment damage or loss of characters. Describes how to identify and resolve problems that can occur with system management and the Cisco Nexus 5000 Series switch. Arguments for which you must enter is in the manual. Do not use these conventions: Convention boldface font italic font [ ] [...
Troubleshooting Guide
Page 7
... are related Cisco Nexus 5000 Series and Cisco Nexus 2000 Series Fabric Extender documents: Release Notes Cisco Nexus 5000 Series and Cisco Nexus 2000 Series Release Notes Cisco Nexus 5000 Series Switch Release Notes Configuration Guides Cisco Nexus 5000 Series Configuration Limits for Cisco NX-OS Release 5.0(2)N1(1) Cisco Nexus 5000 Series Configuration Limits for Cisco NX-OS Release 4.2(1)N1(1) and Release 4.2(1)N2(1) Cisco Nexus 5000 Series NX-OS Fibre Channel over Ethernet Configuration Guide Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide Cisco Nexus 5000 Series NX-OS...
... are related Cisco Nexus 5000 Series and Cisco Nexus 2000 Series Fabric Extender documents: Release Notes Cisco Nexus 5000 Series and Cisco Nexus 2000 Series Release Notes Cisco Nexus 5000 Series Switch Release Notes Configuration Guides Cisco Nexus 5000 Series Configuration Limits for Cisco NX-OS Release 5.0(2)N1(1) Cisco Nexus 5000 Series Configuration Limits for Cisco NX-OS Release 4.2(1)N1(1) and Release 4.2(1)N2(1) Cisco Nexus 5000 Series NX-OS Fibre Channel over Ethernet Configuration Guide Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide Cisco Nexus 5000 Series NX-OS...
Troubleshooting Guide
Page 8
Preface Regulatory Compliance and Safety Information for the Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders Licensing Guide Cisco NX-OS Licensing Guide Command References Cisco Nexus 5000 Series Command Reference Technical References Cisco Nexus 5000 Series and Cisco Nexus 2000 Series Fabric Extender MIBs Reference Error and System Messages Cisco NX-OS System Messages Reference Troubleshooting Guide Cisco Nexus 5000 Troubleshooting Guide Cisco Nexus 5000 Series Troubleshooting Guide viii OL-25300-01
Preface Regulatory Compliance and Safety Information for the Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders Licensing Guide Cisco NX-OS Licensing Guide Command References Cisco Nexus 5000 Series Command Reference Technical References Cisco Nexus 5000 Series and Cisco Nexus 2000 Series Fabric Extender MIBs Reference Error and System Messages Cisco NX-OS System Messages Reference Troubleshooting Guide Cisco Nexus 5000 Troubleshooting Guide Cisco Nexus 5000 Series Troubleshooting Guide viii OL-25300-01
Troubleshooting Guide
Page 9
...Gather information that defines the specific symptoms. Identify all potential problems that may occur when configuring and using the Cisco Nexus 5000 Series switch. The following chapters in a reload of the switch. To identify the possible problems, you need to use a variety of tools and understand the overall configuration.The...Nov 6 15:15:01 2010 Reason: Reset triggered due to HA policy of Reset OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-1 switch# show system reset-reason commands to display the reason for the crash. Send document comments to nexus5k-docfeedback...
...Gather information that defines the specific symptoms. Identify all potential problems that may occur when configuring and using the Cisco Nexus 5000 Series switch. The following chapters in a reload of the switch. To identify the possible problems, you need to use a variety of tools and understand the overall configuration.The...Nov 6 15:15:01 2010 Reason: Reset triggered due to HA policy of Reset OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-1 switch# show system reset-reason commands to display the reason for the crash. Send document comments to nexus5k-docfeedback...
Troubleshooting Guide
Page 10
... port. 2010 Nov 6 08:00:51 TTPSW-5020SF1 %$ VDC-1 %$ %ETHPORT-2-IF_DOWN_ERROR_DISABLED: Interface Ethernet103/1/1 is 1 for a Nexus 5000 switch.) switch#show cores commands. Troubleshooting Basics Chapter 1 Troubleshooting Overview Send document comments to HA policy of the crash with BPDU Guard enabled....: Service "clis" (PID 4155) hasn't caught signal 11 (core will be saved). switch#show cores command to display the processes and if a core was recorded. Cisco Nexus 5000 Series Troubleshooting Guide 1-2 OL-25300-01 Service: clis hap reset Version: 4.2(1)N2(1) 2) At ...
... port. 2010 Nov 6 08:00:51 TTPSW-5020SF1 %$ VDC-1 %$ %ETHPORT-2-IF_DOWN_ERROR_DISABLED: Interface Ethernet103/1/1 is 1 for a Nexus 5000 switch.) switch#show cores commands. Troubleshooting Basics Chapter 1 Troubleshooting Overview Send document comments to HA policy of the crash with BPDU Guard enabled....: Service "clis" (PID 4155) hasn't caught signal 11 (core will be saved). switch#show cores command to display the processes and if a core was recorded. Cisco Nexus 5000 Series Troubleshooting Guide 1-2 OL-25300-01 Service: clis hap reset Version: 4.2(1)N2(1) 2) At ...
Troubleshooting Guide
Page 11
... Fabric Shortest Path First OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-3 Best Practices Best practices are the recommended steps you should take to ensure the proper operation of your switch. • Maintain a consistent Cisco NX-OS release across all your Cisco Nexus 5000 switches. • Refer to the release notes for your Cisco SAN-OS release for the latest...
... Fabric Shortest Path First OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-3 Best Practices Best practices are the recommended steps you should take to ensure the proper operation of your switch. • Maintain a consistent Cisco NX-OS release across all your Cisco Nexus 5000 switches. • Refer to the release notes for your Cisco SAN-OS release for the latest...
Troubleshooting Guide
Page 12
...your specific problem. Fabric Manager Tools and CLI Commands Chapter 1 Troubleshooting Overview Send document comments to the switch prompt. NX-OS Tips Displaying what you can display required data from within the configuration mode, so there is ...a portion of lines exclude Exclude lines that match include Include lines that match Cisco Nexus 5000 Series Troubleshooting Guide 1-4 OL-25300-01 switch(config)# show run switch(config)# show interface brief Pipe command switch# show running-config interface version 4.0(1a)N2(1) interface vfc29 no shutdown bind interface...
...your specific problem. Fabric Manager Tools and CLI Commands Chapter 1 Troubleshooting Overview Send document comments to the switch prompt. NX-OS Tips Displaying what you can display required data from within the configuration mode, so there is ...a portion of lines exclude Exclude lines that match include Include lines that match Cisco Nexus 5000 Series Troubleshooting Guide 1-4 OL-25300-01 switch(config)# show run switch(config)# show interface brief Pipe command switch# show running-config interface version 4.0(1a)N2(1) interface vfc29 no shutdown bind interface...
Troubleshooting Guide
Page 13
...Send document comments to files and flash areas in the switch. Using the pipe command to only display required keyword switch# show tech-support details command to a file and then gzip the file. OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-5 bootflash: core: debug: ftp: licenses...switchport mode fex-fabric switchport speed 1000 switchport mode SD no system default switchport shutdown Copy command switch# copy ? switch# show tech-support aaa > bootflash:ciscolive09 switch# dir 103557265 12451 49152 20058112 20193280 76930262 103557265 4096 Apr 01 17:39:22 2009 Apr...
...Send document comments to files and flash areas in the switch. Using the pipe command to only display required keyword switch# show tech-support details command to a file and then gzip the file. OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-5 bootflash: core: debug: ftp: licenses...switchport mode fex-fabric switchport speed 1000 switchport mode SD no system default switchport shutdown Copy command switch# copy ? switch# show tech-support aaa > bootflash:ciscolive09 switch# dir 103557265 12451 49152 20058112 20193280 76930262 103557265 4096 Apr 01 17:39:22 2009 Apr...
Troubleshooting Guide
Page 14
... procedure in the copy command section. If you do not specify a filename, NX-OS creates the file as volatile:show_tech_out.gz. switch# copy volatile:show_tech_out.gz ? switch(config-if)# show logging Cisco Nexus 5000 Series Troubleshooting Guide 1-6 OL-25300-01 Fabric Manager Tools and CLI Commands Chapter 1 Troubleshooting Overview Send document comments to narrow the scope...
... procedure in the copy command section. If you do not specify a filename, NX-OS creates the file as volatile:show_tech_out.gz. switch# copy volatile:show_tech_out.gz ? switch(config-if)# show logging Cisco Nexus 5000 Series Troubleshooting Guide 1-6 OL-25300-01 Fabric Manager Tools and CLI Commands Chapter 1 Troubleshooting Overview Send document comments to narrow the scope...
Troubleshooting Guide
Page 15
... Name - Chapter 1 Troubleshooting Overview Fabric Manager Tools and CLI Commands Send document comments to , or originate from, the Nexus 5000 control plane. Node to switch or node to a second port for analysis. OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-7 debugging Size - 4194304 Viewing Severity Levels in the Device Manager Ethanalyzer and SPAN Ethanalyzer is a feature...
... Name - Chapter 1 Troubleshooting Overview Fabric Manager Tools and CLI Commands Send document comments to , or originate from, the Nexus 5000 control plane. Node to switch or node to a second port for analysis. OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-7 debugging Size - 4194304 Viewing Severity Levels in the Device Manager Ethanalyzer and SPAN Ethanalyzer is a feature...
Troubleshooting Guide
Page 16
... the number of the frame to capture. ethanalyzer local sniff-interface write Saves the captured data to nexus5k-docfeedback@cisco.com. switch# ethanalyzer local sniff-interface inbound-hi brief limit-captured-frames 0 Capturing on eth0 2008-08-13 01:34:... 0x8093 [snip] Cisco Nexus 5000 Series Troubleshooting Guide 1-8 OL-25300-01 Command Description ethanalyzer local sniff-interface Captures packets sent or received by the supervisor and provides a summary of packets to display. Note Do not use this option if you press Ctrl-C. Examples switch# ethanalyzer local sniff...
... the number of the frame to capture. ethanalyzer local sniff-interface write Saves the captured data to nexus5k-docfeedback@cisco.com. switch# ethanalyzer local sniff-interface inbound-hi brief limit-captured-frames 0 Capturing on eth0 2008-08-13 01:34:... 0x8093 [snip] Cisco Nexus 5000 Series Troubleshooting Guide 1-8 OL-25300-01 Command Description ethanalyzer local sniff-interface Captures packets sent or received by the supervisor and provides a summary of packets to display. Note Do not use this option if you press Ctrl-C. Examples switch# ethanalyzer local sniff...
Troubleshooting Guide
Page 17
... Manager Tools and CLI Commands Send document comments to the SPAN destination port. The Cisco Nexus 5000 Series switch supports Ethernet, virtual Ethernet, Fibre Channel, virtual Fibre Channel, port channels, SAN port channels, VLANs, and VSANs as SPAN sources. OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-9 The network analyzer can choose the SPAN traffic in a SPAN session...
... Manager Tools and CLI Commands Send document comments to the SPAN destination port. The Cisco Nexus 5000 Series switch supports Ethernet, virtual Ethernet, Fibre Channel, virtual Fibre Channel, port channels, SAN port channels, VLANs, and VSANs as SPAN sources. OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-9 The network analyzer can choose the SPAN traffic in a SPAN session...
Troubleshooting Guide
Page 18
.... SPAN Destinations SPAN destinations refer to the maximum number of traffic from the source ports, VLANs, or VSANs. The Cisco Nexus 5000 Series switch supports Ethernet and Fibre Channel interfaces as source ports. • The Cisco Nexus 5010 switch supports a maximum of source VLANs or VSANs. Port channel and SAN port channel interfaces can be configured as ingress or...
.... SPAN Destinations SPAN destinations refer to the maximum number of traffic from the source ports, VLANs, or VSANs. The Cisco Nexus 5000 Series switch supports Ethernet and Fibre Channel interfaces as source ports. • The Cisco Nexus 5010 switch supports a maximum of source VLANs or VSANs. Port channel and SAN port channel interfaces can be configured as ingress or...
Troubleshooting Guide
Page 19
... destination, the monitor session is up only if the destination port is up DESCRIPTION OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-11 If a destination port is not monitored if it can be a monitor destination....switch(config)# interface fc3/2 switch(config-if)# switchport mode sd switch(config-if)# switchport speed 1000 switch(config-if)# no shut switch(config-if)# exit switch(config)# monitor session 1 switch(config-monitor)# source interface fc3/1 tx switch(config-monitor)# source interface fc3/1 rx switch(config-monitor)# destination interface fc3/2 DS-PAA Nexus...
... destination, the monitor session is up only if the destination port is up DESCRIPTION OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 1-11 If a destination port is not monitored if it can be a monitor destination....switch(config)# interface fc3/2 switch(config-if)# switchport mode sd switch(config-if)# switchport speed 1000 switch(config-if)# no shut switch(config-if)# exit switch(config)# monitor session 1 switch(config-monitor)# source interface fc3/1 tx switch(config-monitor)# source interface fc3/1 rx switch(config-monitor)# destination interface fc3/2 DS-PAA Nexus...
Troubleshooting Guide
Page 20
... select when turning on features enabled in switch memory. • Capture to direct to see name of the output: • Logfile-Data file in NX-OS. Determine the destination of the debug file. 1-12 Cisco Nexus 5000 Series Troubleshooting Guide OL-25300-01 Debug Logging .../2 Debugging Command-Line Debugging Available debugs depend on debugs. You must have administrator privileges to run from the CLI. Debugs can only be run debugs. switch# show monitor session 1 session 1 type : local state : up source intf : rx : fc3/1 tx : fc3/1 both : fc3/1 source VLANs : rx ...
... select when turning on features enabled in switch memory. • Capture to direct to see name of the output: • Logfile-Data file in NX-OS. Determine the destination of the debug file. 1-12 Cisco Nexus 5000 Series Troubleshooting Guide OL-25300-01 Debug Logging .../2 Debugging Command-Line Debugging Available debugs depend on debugs. You must have administrator privileges to run from the CLI. Debugs can only be run debugs. switch# show monitor session 1 session 1 type : local state : up source intf : rx : fc3/1 tx : fc3/1 both : fc3/1 source VLANs : rx ...