Troubleshooting Guide
Page 4
... 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager Installations 2-5 Verifying Cisco SAN-OS Software Installations 2-6 Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-7 Software Installation Reports an Incompatibility 2-7 Diagnosing Compatibility Issues 2-7 Software Installation Ends... Bootflash 2-24 Recognizing Error States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-05
... 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager Installations 2-5 Verifying Cisco SAN-OS Software Installations 2-6 Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-7 Software Installation Reports an Incompatibility 2-7 Diagnosing Compatibility Issues 2-7 Software Installation Ends... Bootflash 2-24 Recognizing Error States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-05
Troubleshooting Guide
Page 5
...cisco.com Recovering the Administrator Password 2-32 Miscellaneous Software Image Issues 2-32 All Ports Down Because of System Health Failure 2-33 Switch Reboots after FCIP Reload 2-33 FCIP Link Fails to Come Up 2-33 Cannot Create, Modify, or Delete Admin Role 2-34 FC IDs Change after Link Reset... Ok LED is Red 4-8 Troubleshooting a Fan Failure Using Device Manager 4-9 Troubleshooting a Fan Failure Using the CLI 4-10 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x v Fan LED is Red 4-6 Troubleshooting the Power Supplies 4-7 Troubleshooting Fan Issues 4-8 Fan Is Not...
...cisco.com Recovering the Administrator Password 2-32 Miscellaneous Software Image Issues 2-32 All Ports Down Because of System Health Failure 2-33 Switch Reboots after FCIP Reload 2-33 FCIP Link Fails to Come Up 2-33 Cannot Create, Modify, or Delete Admin Role 2-34 FC IDs Change after Link Reset... Ok LED is Red 4-8 Troubleshooting a Fan Failure Using Device Manager 4-9 Troubleshooting a Fan Failure Using the CLI 4-10 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x v Fan LED is Red 4-6 Troubleshooting the Power Supplies 4-7 Troubleshooting Fan Issues 4-8 Fan Is Not...
Troubleshooting Guide
Page 6
Contents Send documentation comments to mdsfeedback-doc@cisco.com Temperature Threshold Violations 4-11 Troubleshooting Clock Module Issues 4-12 Troubleshooting Other Hardware Issues 4-13 Troubleshooting Supervisor Issues 4-14 Active Supervisor Reboots ... a Module Not Detected by the Supervisor 4-35 Reinitializing a Failed Module Using Fabric Manager 4-36 Reinitializing a Failed Module Using the CLI 4-37 Module Resets 4-38 5 C H A P T E R Troubleshooting Mixed Generation Hardware 5-1 Overview 5-1 Port Groups 5-2 Port Speed Mode 5-3 Dynamic Bandwidth Management 5-3 Out-of-Service Interfaces...
Contents Send documentation comments to mdsfeedback-doc@cisco.com Temperature Threshold Violations 4-11 Troubleshooting Clock Module Issues 4-12 Troubleshooting Other Hardware Issues 4-13 Troubleshooting Supervisor Issues 4-14 Active Supervisor Reboots ... a Module Not Detected by the Supervisor 4-35 Reinitializing a Failed Module Using Fabric Manager 4-36 Reinitializing a Failed Module Using the CLI 4-37 Module Resets 4-38 5 C H A P T E R Troubleshooting Mixed Generation Hardware 5-1 Overview 5-1 Port Groups 5-2 Port Speed Mode 5-3 Dynamic Bandwidth Management 5-3 Out-of-Service Interfaces...
Troubleshooting Guide
Page 40
... errors. Additional tabs include the following: • Protocol-View protocol-related traffic and error statistics, including link reset counts, offline and non-operational sequence errors, reset protocol errors, and statistics related to buffer-to select from the following : • Rx BB Credit-Configure ...and view buffer-to mdsfeedback-doc@cisco.com Device Manager: Port Selection To drill down timers, BB credits, maximum ...
... errors. Additional tabs include the following: • Protocol-View protocol-related traffic and error statistics, including link reset counts, offline and non-operational sequence errors, reset protocol errors, and statistics related to buffer-to select from the following : • Rx BB Credit-Configure ...and view buffer-to mdsfeedback-doc@cisco.com Device Manager: Port Selection To drill down timers, BB credits, maximum ...
Troubleshooting Guide
Page 41
... in Figure 1-3 shows the overall troubleshooting process. Chapter 1 Troubleshooting Overview Primary Troubleshooting Flowchart Send documentation comments to mdsfeedback-doc@cisco.com • Configure (enable or disable restrictions on oversubscription ratios and bandwidth fairness) • Check port group oversubscription (...ports, port speed, and send and receive speed and includes possible oversubscription indicators) • Show port resources • Reset the module Note Port group oversubscription is supported on 24-port and 48-port 4-Gbps Fibre Channel switching modules, the...
... in Figure 1-3 shows the overall troubleshooting process. Chapter 1 Troubleshooting Overview Primary Troubleshooting Flowchart Send documentation comments to mdsfeedback-doc@cisco.com • Configure (enable or disable restrictions on oversubscription ratios and bandwidth fairness) • Check port group oversubscription (...ports, port speed, and send and receive speed and includes possible oversubscription indicators) • Show port resources • Reset the module Note Port group oversubscription is supported on 24-port and 48-port 4-Gbps Fibre Channel switching modules, the...
Troubleshooting Guide
Page 60
...non-disruptive 3 yes disruptive 4 yes disruptive 5 yes non-disruptive 6 yes non-disruptive Install-type rolling rolling rolling rolling reset reset Reason ------ Syncing image bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin to standby 100% -- SUCCESS Syncing image bootflash:///m9500-sf1ek9-mz...mz.2. 1.1a.bin 100% -- SUCCESS Performing configuration copy 100% -- SUCCESS Module 5: Waiting for module online. 2-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 SUCCESS Compatibility check is not supported Images will be upgraded according ...
...non-disruptive 3 yes disruptive 4 yes disruptive 5 yes non-disruptive 6 yes non-disruptive Install-type rolling rolling rolling rolling reset reset Reason ------ Syncing image bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin to standby 100% -- SUCCESS Syncing image bootflash:///m9500-sf1ek9-mz...mz.2. 1.1a.bin 100% -- SUCCESS Performing configuration copy 100% -- SUCCESS Module 5: Waiting for module online. 2-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 SUCCESS Compatibility check is not supported Images will be upgraded according ...
Troubleshooting Guide
Page 74
... Restarts" section on page 2-27 and the "Switch or Process Resets" section on page 2-31 to mdsfeedback-doc@cisco.com Figure 2-8 Error State if Powered On and Esc Is Pressed Switch or Process Resets When a recoverable or nonrecoverable error occurs, the switch or a ... the system. Possible Cause Solution A recoverable error occurred on the system or on the switch reset. See the "Troubleshooting Clock Module Issues" section on the switch may reset. Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to...
... Restarts" section on page 2-27 and the "Switch or Process Resets" section on page 2-31 to mdsfeedback-doc@cisco.com Figure 2-8 Error State if Powered On and Esc Is Pressed Switch or Process Resets When a recoverable or nonrecoverable error occurs, the switch or a ... the system. Possible Cause Solution A recoverable error occurred on the system or on the switch reset. See the "Troubleshooting Clock Module Issues" section on the switch may reset. Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to...
Troubleshooting Guide
Page 79
... supervisor module is allowed by the policy configured for each process. The show system reset-reason module 5 ----- Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to an unrecoverable reset, see the "Troubleshooting Cisco SAN-OS Software System Reboots" section on page 2-13. To respond to mdsfeedback...
... supervisor module is allowed by the policy configured for each process. The show system reset-reason module 5 ----- Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to an unrecoverable reset, see the "Troubleshooting Cisco SAN-OS Software System Reboots" section on page 2-13. To respond to mdsfeedback...
Troubleshooting Guide
Page 80
...8226; All Ports Down Because of System Health Failure, page 2-33 • Switch Reboots after Link Reset, page 2-34 • Switch Displays Wrong User, page 2-34 2-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Note The clear text password "admin123" ...for accessing a switch. Symptom You forgot the administrator password for accessing a Cisco MDS 9000 Family switch. Step 4 Click Admin > Save Configuration to save the running configuration to mdsfeedback-doc@cisco.com Recovering the Administrator Password You can recover the password using a local console...
...8226; All Ports Down Because of System Health Failure, page 2-33 • Switch Reboots after Link Reset, page 2-34 • Switch Displays Wrong User, page 2-34 2-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Note The clear text password "admin123" ...for accessing a switch. Symptom You forgot the administrator password for accessing a Cisco MDS 9000 Family switch. Step 4 Click Admin > Save Configuration to save the running configuration to mdsfeedback-doc@cisco.com Recovering the Administrator Password You can recover the password using a local console...
Troubleshooting Guide
Page 81
... an error recovery mechanism, leaving the module in an unusable state. Possible Cause If an IPS module with the bug fix. Resetting the module will clear the problem, but the problem could reoccur unless you are down because of a system health failure. Switch... of a System Health Failure. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Miscellaneous Software Image Issues Send documentation comments to mdsfeedback-doc@cisco.com All Ports Down Because of System Health Failure Symptom Console reports all FCIP PortChannels on the module. Possible Cause Solution An incorrect ...
... an error recovery mechanism, leaving the module in an unusable state. Possible Cause If an IPS module with the bug fix. Resetting the module will clear the problem, but the problem could reoccur unless you are down because of a system health failure. Switch... of a System Health Failure. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Miscellaneous Software Image Issues Send documentation comments to mdsfeedback-doc@cisco.com All Ports Down Because of System Health Failure Symptom Console reports all FCIP PortChannels on the module. Possible Cause Solution An incorrect ...
Troubleshooting Guide
Page 82
... change after a link flap. Table 2-13 FC IDs Change After a Link Reset Symptom FC IDs change after a link resets. Possible Cause Solution Following an upgrade from Cisco SAN-OS Release 1.1 to mdsfeedback-doc@cisco.com Cannot Create, Modify, or Delete Admin Role Symptom Cannot create, modify, ..., the switch displays the wrong user. The user shown after a link resets. FC IDs Change after Link Reset Symptom FC IDs change after the nondisruptive upgrade is different from Cisco SAN-OS Release 1.3(x) to Cisco SAN-OS Release 2.0, you issue the show running -config CLI command. ...
... change after a link flap. Table 2-13 FC IDs Change After a Link Reset Symptom FC IDs change after a link resets. Possible Cause Solution Following an upgrade from Cisco SAN-OS Release 1.1 to mdsfeedback-doc@cisco.com Cannot Create, Modify, or Delete Admin Role Symptom Cannot create, modify, ..., the switch displays the wrong user. The user shown after a link resets. FC IDs Change after Link Reset Symptom FC IDs change after the nondisruptive upgrade is different from Cisco SAN-OS Release 1.3(x) to Cisco SAN-OS Release 2.0, you issue the show running -config CLI command. ...
Troubleshooting Guide
Page 108
... to clock switch. Troubleshooting Clock Module Issues A Cisco MDS 9500 Series director has two clock modules: A and B. Explanation System shutdown in the number of seconds shown in a hardware reset of show environment clock Command switch# show environment clock Clock Model Hw Status A DS-C9500-CL 0.0 ok/active B DS-C9500-CL 0.0 ok/standby On a clock...
... to clock switch. Troubleshooting Clock Module Issues A Cisco MDS 9500 Series director has two clock modules: A and B. Explanation System shutdown in the number of seconds shown in a hardware reset of show environment clock Command switch# show environment clock Clock Model Hw Status A DS-C9500-CL 0.0 ok/active B DS-C9500-CL 0.0 ok/standby On a clock...
Troubleshooting Guide
Page 110
...After all critical information is pulled out or restarted. Troubleshooting Supervisor Issues Chapter 4 Troubleshooting Hardware Send documentation comments to mdsfeedback-doc@cisco.com Step 2 Step 3 Number Ports went bad: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 View the error statistics from ... fails to initialize properly. The standby supervisor needs to view the persistent log: • show logging nvram • show system reset-reason • show hardware internal errors command output. When a supervisor reboots, much of the active supervisor. Once the supervisor reboots...
...After all critical information is pulled out or restarted. Troubleshooting Supervisor Issues Chapter 4 Troubleshooting Hardware Send documentation comments to mdsfeedback-doc@cisco.com Step 2 Step 3 Number Ports went bad: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 View the error statistics from ... fails to initialize properly. The standby supervisor needs to view the persistent log: • show logging nvram • show system reset-reason • show hardware internal errors command output. When a supervisor reboots, much of the active supervisor. Once the supervisor reboots...
Troubleshooting Guide
Page 111
... ----- Example 4-7 System Messages for Supervisor Reboot Caused by Failed Process Switch# show system reset-reason CLI command to mdsfeedback-doc@cisco.com Active Supervisor Reboots Symptom Active supervisor reboots. See (Example 4-9.) Optionally, when the ... 4-7 Active Supervisor Reboots Symptom Active supervisor reboots. reset reason for the recent when a supervisor module reboots after a process crash. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-15 See also the "Troubleshooting Cisco SAN-OS Software System Reboots" section on the...
... ----- Example 4-7 System Messages for Supervisor Reboot Caused by Failed Process Switch# show system reset-reason CLI command to mdsfeedback-doc@cisco.com Active Supervisor Reboots Symptom Active supervisor reboots. See (Example 4-9.) Optionally, when the ... 4-7 Active Supervisor Reboots Symptom Active supervisor reboots. reset reason for the recent when a supervisor module reboots after a process crash. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-15 See also the "Troubleshooting Cisco SAN-OS Software System Reboots" section on the...
Troubleshooting Guide
Page 116
Right-click the standby supervisor and select Reset from attempting to fail over to ...Use the show module command on the active supervisor to describe the problem and the workaround. 4-20 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Troubleshooting Supervisor Issues Chapter 4 Troubleshooting Hardware ...-20 JAB070307XG * this terminal session Use the show module Mod Ports Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 powered-up state using Device Manager, follow these steps: Step 1 Step 2 Choose...
Right-click the standby supervisor and select Reset from attempting to fail over to ...Use the show module command on the active supervisor to describe the problem and the workaround. 4-20 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Troubleshooting Supervisor Issues Chapter 4 Troubleshooting Hardware ...-20 JAB070307XG * this terminal session Use the show module Mod Ports Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 powered-up state using Device Manager, follow these steps: Step 1 Step 2 Choose...
Troubleshooting Guide
Page 117
... Output switch# show module 8 Mod Ports Module-Type Model Status 8 8 IP Storage Services Module DS-X9308-SMIP ok Mod Sw Hw World-Wide-Name(s) (WWN) 8 2.1(2) 0.206 21:c1:00..., page 4-36 • Reinitializing a Failed Module Using the CLI, page 4-37 • Module Resets, page 4-38 Overview of missing boot statements. Entering a copy slot0: bootflash: CLI command copied the...4 Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to mdsfeedback-doc@cisco.com In this error, the current Flash images were unable to be copied from...
... Output switch# show module 8 Mod Ports Module-Type Model Status 8 8 IP Storage Services Module DS-X9308-SMIP ok Mod Sw Hw World-Wide-Name(s) (WWN) 8 2.1(2) 0.206 21:c1:00..., page 4-36 • Reinitializing a Failed Module Using the CLI, page 4-37 • Module Resets, page 4-38 Overview of missing boot statements. Entering a copy slot0: bootflash: CLI command copied the...4 Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to mdsfeedback-doc@cisco.com In this error, the current Flash images were unable to be copied from...
Troubleshooting Guide
Page 118
... by your customer support representative. Transient The module reloaded. Use the show environment power CLI command to show system reset-reason module 4-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Good The module has been powered down . Transient The... a module is declared online. Troubleshooting Switching and Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to mdsfeedback-doc@cisco.com The module status indicates the state of an error. The chassis does not have enough remaining power to power up...
... by your customer support representative. Transient The module reloaded. Use the show environment power CLI command to show system reset-reason module 4-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Good The module has been powered down . Transient The... a module is declared online. Troubleshooting Switching and Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to mdsfeedback-doc@cisco.com The module status indicates the state of an error. The chassis does not have enough remaining power to power up...
Troubleshooting Guide
Page 124
... Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 8 8 IP Storage Services Module powered-dn Mod Sw Hw World-Wide-Name(s) (WWN) 5 2.1(2) 1.1 -- 6 2.1(2) 0.602 -- 4-28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-..."Reinitializing a Failed Module Using the CLI" section on page 4-37. Right-click the module in Device Manager and select Reset or use the reload module CLI command to register with the supervisor. See the "Reinitializing a Failed Module Using Fabric Manager...
... Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 8 8 IP Storage Services Module powered-dn Mod Sw Hw World-Wide-Name(s) (WWN) 5 2.1(2) 1.1 -- 6 2.1(2) 0.602 -- 4-28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-..."Reinitializing a Failed Module Using the CLI" section on page 4-37. Right-click the module in Device Manager and select Reset or use the reload module CLI command to register with the supervisor. See the "Reinitializing a Failed Module Using Fabric Manager...
Troubleshooting Guide
Page 128
... module. Recommended Action Collect information about the module by entering the show module internal all module CLI command. resetting. The module manager will reset the module. Recommended Action Collect module information by entering the show module internal all module CLI command. Explanation...]-[dec]/[dec] ([chars]) due to [chars] in device [dec] (device error [hex]). Module manager is going to mdsfeedback-doc@cisco.com Error Message MODULE-2-MOD_NOT_ALIVE: Module [dec] not responding... Explanation Port loop-back test failure. Explanation The module is required. Error ...
... module. Recommended Action Collect information about the module by entering the show module internal all module CLI command. resetting. The module manager will reset the module. Recommended Action Collect module information by entering the show module internal all module CLI command. Explanation...]-[dec]/[dec] ([chars]) due to [chars] in device [dec] (device error [hex]). Module manager is going to mdsfeedback-doc@cisco.com Error Message MODULE-2-MOD_NOT_ALIVE: Module [dec] not responding... Explanation Port loop-back test failure. Explanation The module is required. Error ...
Troubleshooting Guide
Page 129
...bad: 1,2,3,4,5,6,7,8 exception information --- device id: 5 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-33 Right-click the module in Device Manager and select Reset or use the reload module CLI command to restart the module. ... exceptionlog module 8 ********* Exception info for something similar to: Rx MTS_OPC_SSA_LOST_SYNC_SERIAL slot 8 fabric 0 link 0 to mdsfeedback-doc@cisco.com Table 4-12 Module is Automatically Reloaded Symptom Module is automatically reloaded. The module experienced runtime diagnostic failures. Chapter 4 ...
...bad: 1,2,3,4,5,6,7,8 exception information --- device id: 5 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-33 Right-click the module in Device Manager and select Reset or use the reload module CLI command to restart the module. ... exceptionlog module 8 ********* Exception info for something similar to: Rx MTS_OPC_SSA_LOST_SYNC_SERIAL slot 8 fabric 0 link 0 to mdsfeedback-doc@cisco.com Table 4-12 Module is Automatically Reloaded Symptom Module is automatically reloaded. The module experienced runtime diagnostic failures. Chapter 4 ...