Troubleshooting Guide
Page 4
... 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager Installations 2-5 Verifying Cisco SAN-OS Software Installations 2-6 Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-7 Software Installation Reports an Incompatibility 2-7 Diagnosing Compatibility Issues 2-7 Software Installation Ends... Bootflash 2-24 Recognizing Error States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-05
... 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager Installations 2-5 Verifying Cisco SAN-OS Software Installations 2-6 Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-7 Software Installation Reports an Incompatibility 2-7 Diagnosing Compatibility Issues 2-7 Software Installation Ends... Bootflash 2-24 Recognizing Error States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-05
Troubleshooting Guide
Page 5
...cisco.com Recovering the Administrator Password 2-32 Miscellaneous Software Image Issues 2-32 All Ports Down Because of System Health Failure 2-33 Switch Reboots after FCIP Reload 2-33 FCIP Link Fails to Come Up 2-33 Cannot Create, Modify, or Delete Admin Role 2-34 FC IDs Change after Link Reset... Ok LED is Red 4-8 Troubleshooting a Fan Failure Using Device Manager 4-9 Troubleshooting a Fan Failure Using the CLI 4-10 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x v Fan LED is Red 4-6 Troubleshooting the Power Supplies 4-7 Troubleshooting Fan Issues 4-8 Fan Is Not ...
...cisco.com Recovering the Administrator Password 2-32 Miscellaneous Software Image Issues 2-32 All Ports Down Because of System Health Failure 2-33 Switch Reboots after FCIP Reload 2-33 FCIP Link Fails to Come Up 2-33 Cannot Create, Modify, or Delete Admin Role 2-34 FC IDs Change after Link Reset... Ok LED is Red 4-8 Troubleshooting a Fan Failure Using Device Manager 4-9 Troubleshooting a Fan Failure Using the CLI 4-10 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x v Fan LED is Red 4-6 Troubleshooting the Power Supplies 4-7 Troubleshooting Fan Issues 4-8 Fan Is Not ...
Troubleshooting Guide
Page 6
Contents Send documentation comments to mdsfeedback-doc@cisco.com Temperature Threshold Violations 4-11 Troubleshooting Clock Module Issues 4-12 Troubleshooting Other Hardware Issues 4-13 Troubleshooting Supervisor Issues 4-14 Active Supervisor Reboots ... a Module Not Detected by the Supervisor 4-35 Reinitializing a Failed Module Using Fabric Manager 4-36 Reinitializing a Failed Module Using the CLI 4-37 Module Resets 4-38 5 C H A P T E R Troubleshooting Mixed Generation Hardware 5-1 Overview 5-1 Port Groups 5-2 Port Speed Mode 5-3 Dynamic Bandwidth Management 5-3 Out-of-Service Interfaces...
Contents Send documentation comments to mdsfeedback-doc@cisco.com Temperature Threshold Violations 4-11 Troubleshooting Clock Module Issues 4-12 Troubleshooting Other Hardware Issues 4-13 Troubleshooting Supervisor Issues 4-14 Active Supervisor Reboots ... a Module Not Detected by the Supervisor 4-35 Reinitializing a Failed Module Using Fabric Manager 4-36 Reinitializing a Failed Module Using the CLI 4-37 Module Resets 4-38 5 C H A P T E R Troubleshooting Mixed Generation Hardware 5-1 Overview 5-1 Port Groups 5-2 Port Speed Mode 5-3 Dynamic Bandwidth Management 5-3 Out-of-Service Interfaces...
Troubleshooting Guide
Page 40
...-to select from the following : • Protocol-View protocol-related traffic and error statistics, including link reset counts, offline and non-operational sequence errors, reset protocol errors, and statistics related to buffer-to- In Summary View, choose one or more interfaces, and... including pacing, disparity, EOF, OOF, and order sets errors. Troubleshooting Basics Chapter 1 Troubleshooting Overview Send documentation comments to mdsfeedback-doc@cisco.com Device Manager: Port Selection To drill down timers, BB credits, maximum receive buffer size. Select and double-click any port....
...-to select from the following : • Protocol-View protocol-related traffic and error statistics, including link reset counts, offline and non-operational sequence errors, reset protocol errors, and statistics related to buffer-to- In Summary View, choose one or more interfaces, and... including pacing, disparity, EOF, OOF, and order sets errors. Troubleshooting Basics Chapter 1 Troubleshooting Overview Send documentation comments to mdsfeedback-doc@cisco.com Device Manager: Port Selection To drill down timers, BB credits, maximum receive buffer size. Select and double-click any port....
Troubleshooting Guide
Page 41
... port Fabric services issues Port state Modules up? Chapter 1 Troubleshooting Overview Primary Troubleshooting Flowchart Send documentation comments to mdsfeedback-doc@cisco.com • Configure (enable or disable restrictions on oversubscription ratios and bandwidth fairness) • Check port group oversubscription (... ports, port speed, and send and receive speed and includes possible oversubscription indicators) • Show port resources • Reset the module Note Port group oversubscription is supported on 24-port and 48-port 4-Gbps Fibre Channel switching modules, the 32...
... port Fabric services issues Port state Modules up? Chapter 1 Troubleshooting Overview Primary Troubleshooting Flowchart Send documentation comments to mdsfeedback-doc@cisco.com • Configure (enable or disable restrictions on oversubscription ratios and bandwidth fairness) • Check port group oversubscription (... ports, port speed, and send and receive speed and includes possible oversubscription indicators) • Show port resources • Reset the module Note Port group oversubscription is supported on 24-port and 48-port 4-Gbps Fibre Channel switching modules, the 32...
Troubleshooting Guide
Page 60
...the installation (y/n)? [n] y Install is in progress, please wait. SUCCESS Module 5: Waiting for module online. 2-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 SUCCESS Extracting "loader" version from image bootflash:///m9500-sf1ek9-kickstart...disruptive 6 yes non-disruptive Install-type rolling rolling rolling rolling reset reset Reason ------ SUCCESS Syncing image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin to mdsfeedback-doc@cisco.com 100% -- Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades Chapter 2 Troubleshooting Installs, Upgrades...
...the installation (y/n)? [n] y Install is in progress, please wait. SUCCESS Module 5: Waiting for module online. 2-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 SUCCESS Extracting "loader" version from image bootflash:///m9500-sf1ek9-kickstart...disruptive 6 yes non-disruptive Install-type rolling rolling rolling rolling reset reset Reason ------ SUCCESS Syncing image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin to mdsfeedback-doc@cisco.com 100% -- Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades Chapter 2 Troubleshooting Installs, Upgrades...
Troubleshooting Guide
Page 74
... automatically from the problem. A clock module failed. See the "Recoverable System Restarts" section on page 2-27 and the "Switch or Process Resets" section on a process in the system. Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to determine the cause. Possible Cause Solution A recoverable...
... automatically from the problem. A clock module failed. See the "Recoverable System Restarts" section on page 2-27 and the "Switch or Process Resets" section on a process in the system. Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to determine the cause. Possible Cause Solution A recoverable...
Troubleshooting Guide
Page 79
... 5 and slot 6 are displayed. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to an unrecoverable reset, see the "Troubleshooting Cisco SAN-OS Software System Reboots" section on page 2-13. If a module is absent, the reset-reason codes for that was running at the time of a process...
... 5 and slot 6 are displayed. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to an unrecoverable reset, see the "Troubleshooting Cisco SAN-OS Software System Reboots" section on page 2-13. If a module is absent, the reset-reason codes for that was running at the time of a process...
Troubleshooting Guide
Page 80
...Cannot Create, Modify, or Delete Admin Role, page 2-34 • FC IDs Change after Link Reset, page 2-34 • Switch Displays Wrong User, page 2-34 2-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Table 2-8 Recovering Administrator Password Problem You ... in Table 2-8. Recovering the Administrator Password Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to mdsfeedback-doc@cisco.com Recovering the Administrator Password You can recover the password using a local console connection. Solution You can access the switch...
...Cannot Create, Modify, or Delete Admin Role, page 2-34 • FC IDs Change after Link Reset, page 2-34 • Switch Displays Wrong User, page 2-34 2-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Table 2-8 Recovering Administrator Password Problem You ... in Table 2-8. Recovering the Administrator Password Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to mdsfeedback-doc@cisco.com Recovering the Administrator Password You can recover the password using a local console connection. Solution You can access the switch...
Troubleshooting Guide
Page 81
...Solution This symptom may reboot. Symptom The system console reports that the module's ports are down because of a system health failure. Resetting the module will clear the problem, but the problem could reoccur unless you are Down Because of a System Health Failure. Table ... the MPS-14/2 module using a SAN-OS version with operational FCIP PortChannels is a specific module. Downgrade to Cisco SAN-OS Release 2.1.2 or 2.1(1b). Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Miscellaneous Software Image Issues Send documentation comments to mdsfeedback-doc...
...Solution This symptom may reboot. Symptom The system console reports that the module's ports are down because of a system health failure. Resetting the module will clear the problem, but the problem could reoccur unless you are Down Because of a System Health Failure. Table ... the MPS-14/2 module using a SAN-OS version with operational FCIP PortChannels is a specific module. Downgrade to Cisco SAN-OS Release 2.1.2 or 2.1(1b). Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Miscellaneous Software Image Issues Send documentation comments to mdsfeedback-doc...
Troubleshooting Guide
Page 82
...-config command, the switch displays the wrong user. FC IDs Change after Link Reset Symptom FC IDs change after a link resets. Possible Cause Solution Following an upgrade from Cisco SAN-OS Release 1.3(x) to Cisco SAN-OS Release 2.0(x) and then issue the show running -config CLI command....The user shown after a link resets. Table 2-13 FC IDs Change After a Link Reset Symptom FC IDs change after the nondisruptive upgrade is different from the user shown when you perform a nondisruptive upgrade from Cisco SAN-OS Release 1.1 to mdsfeedback-doc@cisco.com Cannot Create, Modify, or...
...-config command, the switch displays the wrong user. FC IDs Change after Link Reset Symptom FC IDs change after a link resets. Possible Cause Solution Following an upgrade from Cisco SAN-OS Release 1.3(x) to Cisco SAN-OS Release 2.0(x) and then issue the show running -config CLI command....The user shown after a link resets. Table 2-13 FC IDs Change After a Link Reset Symptom FC IDs change after the nondisruptive upgrade is different from the user shown when you perform a nondisruptive upgrade from Cisco SAN-OS Release 1.1 to mdsfeedback-doc@cisco.com Cannot Create, Modify, or...
Troubleshooting Guide
Page 108
...shutdown in [dec] seconds. Otherwise, the switch power supplies are printed every five seconds during the next maintenance window. 4-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Explanation Module contains a faulty temperature sensor. The following syslog message ...ok/active B DS-C9500-CL 0.0 ok/standby On a clock module failure, the system switches over to the redundant clock module automatically. System will be reset. The following system message: Error Message PLATFORM-5-MOD_TEMPFAIL: Module [dec] temperature sensor failed. If Cisco SAN-OS ...
...shutdown in [dec] seconds. Otherwise, the switch power supplies are printed every five seconds during the next maintenance window. 4-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Explanation Module contains a faulty temperature sensor. The following syslog message ...ok/active B DS-C9500-CL 0.0 ok/standby On a clock module failure, the system switches over to the redundant clock module automatically. System will be reset. The following system message: Error Message PLATFORM-5-MOD_TEMPFAIL: Module [dec] temperature sensor failed. If Cisco SAN-OS ...
Troubleshooting Guide
Page 110
... up . The active supervisor initialization differs from the show up will default to initialize properly. After all critical information is lost. Cisco SAN-OS maintains debug information during runtime. When a supervisor reboots, much of Supervisors is up first will default to reconstruct the ...are declared as faulty. Note the following CLI commands to view the persistent log: • show logging nvram • show system reset-reason • show hardware internal errors command output. However, all components on whether or not you have a threshold before the ...
... up . The active supervisor initialization differs from the show up will default to initialize properly. After all critical information is lost. Cisco SAN-OS maintains debug information during runtime. When a supervisor reboots, much of Supervisors is up first will default to reconstruct the ...are declared as faulty. Note the following CLI commands to view the persistent log: • show logging nvram • show system reset-reason • show hardware internal errors command output. However, all components on whether or not you have a threshold before the ...
Troubleshooting Guide
Page 111
...caused the reboot Version: 2.1(2) Example 4-7 displays the system messages on the standby supervisor to view a list of the reset after a process crash. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-15 Runtime diagnostics failure detected. Example 4-7 System Messages for module 6 ----1)... information. Example 4-6 displays the reason for Supervisor Reboot Caused by Failed Process Switch# show system reset-reason CLI command to mdsfeedback-doc@cisco.com Active Supervisor Reboots Symptom Active supervisor reboots. Use the show logging 2005 Sep 27 18:58...
...caused the reboot Version: 2.1(2) Example 4-7 displays the system messages on the standby supervisor to view a list of the reset after a process crash. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-15 Runtime diagnostics failure detected. Example 4-7 System Messages for module 6 ----1)... information. Example 4-6 displays the reason for Supervisor Reboot Caused by Failed Process Switch# show system reset-reason CLI command to mdsfeedback-doc@cisco.com Active Supervisor Reboots Symptom Active supervisor reboots. Use the show logging 2005 Sep 27 18:58...
Troubleshooting Guide
Page 116
...supervisor. Troubleshooting Supervisor Issues Chapter 4 Troubleshooting Hardware Send documentation comments to mdsfeedback-doc@cisco.com Verifying That a Standby Supervisor Is in the Powered-Up State Using Device ...module Command Output switch# show module Mod Ports Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 powered-up state using the CLI, follow these steps:... situation is PoweredUp. Right-click the standby supervisor and select Reset from attempting to fail over to an unavailable module. Use the reload module command...
...supervisor. Troubleshooting Supervisor Issues Chapter 4 Troubleshooting Hardware Send documentation comments to mdsfeedback-doc@cisco.com Verifying That a Standby Supervisor Is in the Powered-Up State Using Device ...module Command Output switch# show module Mod Ports Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 powered-up state using the CLI, follow these steps:... situation is PoweredUp. Right-click the standby supervisor and select Reset from attempting to fail over to an unavailable module. Use the reload module command...
Troubleshooting Guide
Page 117
... Output switch# show module 8 Mod Ports Module-Type Model Status 8 8 IP Storage Services Module DS-X9308-SMIP ok Mod Sw Hw World-Wide-Name(s) (WWN) 8 2.1(2) 0.206 21:c1:00...page 4-36 • Reinitializing a Failed Module Using the CLI, page 4-37 • Module Resets, page 4-38 Overview of code changed, or the running configuration on the active supervisor was not...Chapter 4 Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to mdsfeedback-doc@cisco.com In this error, the current Flash images were unable to be copied from...
... Output switch# show module 8 Mod Ports Module-Type Model Status 8 8 IP Storage Services Module DS-X9308-SMIP ok Mod Sw Hw World-Wide-Name(s) (WWN) 8 2.1(2) 0.206 21:c1:00...page 4-36 • Reinitializing a Failed Module Using the CLI, page 4-37 • Module Resets, page 4-38 Overview of code changed, or the running configuration on the active supervisor was not...Chapter 4 Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to mdsfeedback-doc@cisco.com In this error, the current Flash images were unable to be copied from...
Troubleshooting Guide
Page 118
... Transient The module reloaded. Otherwise, the module was configured. The chassis does not have enough remaining power to mdsfeedback-doc@cisco.com The module status indicates the state of the module related failures (such as powered-down err-pwd-dn pwr-denied ... failure Description Module Status Condition The module is up and running -config | include poweroff CLI command to show system reset-reason module 4-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Troubleshooting Switching and Services Modules Chapter 4 Troubleshooting Hardware Send ...
... Transient The module reloaded. Otherwise, the module was configured. The chassis does not have enough remaining power to mdsfeedback-doc@cisco.com The module status indicates the state of the module related failures (such as powered-down err-pwd-dn pwr-denied ... failure Description Module Status Condition The module is up and running -config | include poweroff CLI command to show system reset-reason module 4-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Troubleshooting Switching and Services Modules Chapter 4 Troubleshooting Hardware Send ...
Troubleshooting Guide
Page 124
...problems. Right-click the module in powered-down state. Right-click the module in Device Manager and select Reset or use the reload module CLI command to verify the status of the module. Right-click the module in Device Manager and... Mod Ports Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 8 8 IP Storage Services Module powered-dn Mod Sw Hw World-Wide-Name(s) (WWN) 5 2.1(2) 1.1 -- 6 2.1(2) 0.602 -- 4-28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Verify...
...problems. Right-click the module in powered-down state. Right-click the module in Device Manager and select Reset or use the reload module CLI command to verify the status of the module. Right-click the module in Device Manager and... Mod Ports Module-Type Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 8 8 IP Storage Services Module powered-dn Mod Sw Hw World-Wide-Name(s) (WWN) 5 2.1(2) 1.1 -- 6 2.1(2) 0.602 -- 4-28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Verify...
Troubleshooting Guide
Page 128
...Chapter 4 Troubleshooting Hardware Send documentation comments to [chars] in device [dec] (error [hex]). The module manager will reset the module. Explanation Module reported a failure in the runtime diagnostic because of a failure in the runtime diagnostic. Recommended Action...failure. Error Message SYSTEMHEALTH-2-OHMS_MOD_SNAKE_TEST_FAILED: Module [dec] has failed snake loopback tests. Explanation The module is required. 4-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Recommended Action No action is required. Error Message MODULE-2-MOD_DIAG_FAIL:...
...Chapter 4 Troubleshooting Hardware Send documentation comments to [chars] in device [dec] (error [hex]). The module manager will reset the module. Explanation Module reported a failure in the runtime diagnostic because of a failure in the runtime diagnostic. Recommended Action...failure. Error Message SYSTEMHEALTH-2-OHMS_MOD_SNAKE_TEST_FAILED: Module [dec] has failed snake loopback tests. Explanation The module is required. 4-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Recommended Action No action is required. Error Message MODULE-2-MOD_DIAG_FAIL:...
Troubleshooting Guide
Page 129
... event: [LCM_EV_LCP_ALIVE_TIMEOUT] to verify that the module lost synchronize with the fabric. Right-click the module in Device Manager and select Reset or use the reload module CLI command to heartbeat requests. switch# show module internal event-history module CLI command and look for ... failure. Use the show module CLI command to mdsfeedback-doc@cisco.com Table 4-12 Module is Automatically Reloaded Symptom Module is automatically reloaded. Right-click the module in Device Manager and select Reset or use the show system internal xbar internal event-history errors...
... event: [LCM_EV_LCP_ALIVE_TIMEOUT] to verify that the module lost synchronize with the fabric. Right-click the module in Device Manager and select Reset or use the reload module CLI command to heartbeat requests. switch# show module internal event-history module CLI command and look for ... failure. Use the show module CLI command to mdsfeedback-doc@cisco.com Table 4-12 Module is Automatically Reloaded Symptom Module is automatically reloaded. Right-click the module in Device Manager and select Reset or use the show system internal xbar internal event-history errors...