Troubleshooting Guide
Page 3
... Hardware Installation i-xxx Cisco Fabric Manager i-xxxi Command-Line Interface i-xxxi Troubleshooting and Reference i-xxxi Installation and Configuration Note i-xxxi Obtaining Documentation, Obtaining Support, and Security Guidelines i-xxxii Troubleshooting Overview 1-1 Overview of the Troubleshooting Process 1-1 Best Practices 1-2 Troubleshooting Basics 1-2 General Steps 1-2 Gathering Information Using Common Fabric Manager Tools and CLI Commands 1-3 Common Fabric Manager Tools 1-3 Common CLI Commands 1-4 Verifying Basic Connectivity 1-4 Verifying SAN Element Registration 1-5 Fibre Channel...
... Hardware Installation i-xxx Cisco Fabric Manager i-xxxi Command-Line Interface i-xxxi Troubleshooting and Reference i-xxxi Installation and Configuration Note i-xxxi Obtaining Documentation, Obtaining Support, and Security Guidelines i-xxxii Troubleshooting Overview 1-1 Overview of the Troubleshooting Process 1-1 Best Practices 1-2 Troubleshooting Basics 1-2 General Steps 1-2 Gathering Information Using Common Fabric Manager Tools and CLI Commands 1-3 Common Fabric Manager Tools 1-3 Common CLI Commands 1-4 Verifying Basic Connectivity 1-4 Verifying SAN Element Registration 1-5 Fibre Channel...
Troubleshooting Guide
Page 4
...documentation comments to mdsfeedback-doc@cisco.com Implementing Syslog with the CLI 1-12 Troubleshooting with Logs 1-13 Viewing Logs with Fabric Manager 1-13 Viewing Logs with the CLI 1-13 Viewing the Log from the Supervisor 1-14 Viewing NVRAM logs 1-14 Contacting Customer Support 1-15 2 C H A P T E R Troubleshooting Installs, Upgrades, and Reboots 2-1 Overview 2-1 Guidelines 2-2 Guidelines for Installations 2-2 Guidelines for Upgrading 2-2 Guidelines for Reboots 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager...
...documentation comments to mdsfeedback-doc@cisco.com Implementing Syslog with the CLI 1-12 Troubleshooting with Logs 1-13 Viewing Logs with Fabric Manager 1-13 Viewing Logs with the CLI 1-13 Viewing the Log from the Supervisor 1-14 Viewing NVRAM logs 1-14 Contacting Customer Support 1-15 2 C H A P T E R Troubleshooting Installs, Upgrades, and Reboots 2-1 Overview 2-1 Guidelines 2-2 Guidelines for Installations 2-2 Guidelines for Upgrading 2-2 Guidelines for Reboots 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager...
Troubleshooting Guide
Page 26
..., "Troubleshooting IP Access Lists" Added troubleshooting users and roles based 2.x access. New and Changed Information Send documentation comments to mdsfeedback-doc@cisco.com Table 1 New and Changed Features for Release 3.x (continued) Feature FICON N-Port Virtualization SANTap Device Alias Services IP Network Simulator Nondisruptive upgrades on the Cisco MDS 9124 Fabric Switch. Chapter 2, "Troubleshooting Installs, Upgrades, and Reboots" Added troubleshooting information for SAN device virtualization. 3.1(2) Chapter 12, "Troubleshooting SAN Device Virtualization" Added SNMP test...
..., "Troubleshooting IP Access Lists" Added troubleshooting users and roles based 2.x access. New and Changed Information Send documentation comments to mdsfeedback-doc@cisco.com Table 1 New and Changed Features for Release 3.x (continued) Feature FICON N-Port Virtualization SANTap Device Alias Services IP Network Simulator Nondisruptive upgrades on the Cisco MDS 9124 Fabric Switch. Chapter 2, "Troubleshooting Installs, Upgrades, and Reboots" Added troubleshooting information for SAN device virtualization. 3.1(2) Chapter 12, "Troubleshooting SAN Device Virtualization" Added SNMP test...
Troubleshooting Guide
Page 30
...-doc@cisco.com Related Documentation The documentation set for the Cisco MDS 9000 Family Hardware Installation • Cisco MDS 9124 Multilayer Fabric Switch Quick Start Guide • Cisco MDS 9500 Series Hardware Installation Guide • Cisco MDS 9200 Series Hardware Installation Guide • Cisco MDS 9100 Series Hardware Installation Guide • Cisco MDS 9020 Fabric Switch Hardware Installation Guide Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxx OL-9285-05 Related Documentation Preface Send documentation comments to the IBM TotalStorage Support website...
...-doc@cisco.com Related Documentation The documentation set for the Cisco MDS 9000 Family Hardware Installation • Cisco MDS 9124 Multilayer Fabric Switch Quick Start Guide • Cisco MDS 9500 Series Hardware Installation Guide • Cisco MDS 9200 Series Hardware Installation Guide • Cisco MDS 9100 Series Hardware Installation Guide • Cisco MDS 9020 Fabric Switch Hardware Installation Guide Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxx OL-9285-05 Related Documentation Preface Send documentation comments to the IBM TotalStorage Support website...
Troubleshooting Guide
Page 31
...Command-Line Interface • Cisco MDS 9000 Family Software Upgrade and Downgrade Guide • Cisco MDS 9000 Family Storage Services Module Software Installation and Upgrade Guide • Cisco MDS 9000 Family CLI Quick Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide • Cisco MDS 9000 Family Command Reference • Cisco MDS 9000 Family Quick Command Reference • Cisco MDS 9020 Fabric Switch Configuration Guide and Command Reference • Cisco MDS 9000 Family SAN Volume Controller Configuration Guide Troubleshooting and Reference • Cisco MDS...
...Command-Line Interface • Cisco MDS 9000 Family Software Upgrade and Downgrade Guide • Cisco MDS 9000 Family Storage Services Module Software Installation and Upgrade Guide • Cisco MDS 9000 Family CLI Quick Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide • Cisco MDS 9000 Family Command Reference • Cisco MDS 9000 Family Quick Command Reference • Cisco MDS 9020 Fabric Switch Configuration Guide and Command Reference • Cisco MDS 9000 Family SAN Volume Controller Configuration Guide Troubleshooting and Reference • Cisco MDS...
Troubleshooting Guide
Page 51
... other switching modules. • If you see this command on the intended changes to your upgrade will be installing. - For example, if a switching module fails to be nondisruptive. • Establish a PC serial connection to each supervisor console to record upgrade activity to a file. You have the option to permit or deny. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Guidelines Send documentation comments to mdsfeedback-doc@cisco.com • Copy the startup-config to a snapshot config...
... other switching modules. • If you see this command on the intended changes to your upgrade will be installing. - For example, if a switching module fails to be nondisruptive. • Establish a PC serial connection to each supervisor console to record upgrade activity to a file. You have the option to permit or deny. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Guidelines Send documentation comments to mdsfeedback-doc@cisco.com • Copy the startup-config to a snapshot config...
Troubleshooting Guide
Page 57
... in progress. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades Send documentation comments to accept the updated image. The standby supervisor module bootflash: file system does not have sufficient space to mdsfeedback-doc@cisco.com Software Installation Ends with Error Symptom The software installation ends with an error. In Device Manager, choose Admin > Flash Files and delete unnecessary files. Restart the installation. See the "Installing SAN-OS Software Using Fabric Manager" section on page 2-11...
... in progress. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades Send documentation comments to accept the updated image. The standby supervisor module bootflash: file system does not have sufficient space to mdsfeedback-doc@cisco.com Software Installation Ends with Error Symptom The software installation ends with an error. In Device Manager, choose Admin > Flash Files and delete unnecessary files. Restart the installation. See the "Installing SAN-OS Software Using Fabric Manager" section on page 2-11...
Troubleshooting Guide
Page 61
... standby terminated. Interrupt the boot process and reconfigure the BIOS through the console port to load a new kickstart image that updates to view the upgraded supervisor module using the show module command. Please check the standby supervisor. -- hangs for Switches with Dual Supervisor Modules, page 2-22 • Recognizing Error States, page 2-25 Power On or Switch Reboot Hangs Symptom Power on or switch reboot hangs. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation...
... standby terminated. Interrupt the boot process and reconfigure the BIOS through the console port to load a new kickstart image that updates to view the upgraded supervisor module using the show module command. Please check the standby supervisor. -- hangs for Switches with Dual Supervisor Modules, page 2-22 • Recognizing Error States, page 2-25 Power On or Switch Reboot Hangs Symptom Power on or switch reboot hangs. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation...
Troubleshooting Guide
Page 65
... file name must be entered exactly as shown on your TFTP server. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-17 Basic CMOS Configuration screen. (See Figure 2-5.) Figure 2-5 BIOS Setup Configuration (CMOS) Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Change the Boot 1st: field to select the Basic CMOS Configuration. Enter the IP address of the default gateway, and press the Tab key...
... file name must be entered exactly as shown on your TFTP server. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-17 Basic CMOS Configuration screen. (See Figure 2-5.) Figure 2-5 BIOS Setup Configuration (CMOS) Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Change the Boot 1st: field to select the Basic CMOS Configuration. Enter the IP address of the default gateway, and press the Tab key...
Troubleshooting Guide
Page 75
...-5-IF_DOWN: Interface fc1/1 is down (Link failure o r not-connected) switch# Enter the following command to the Cisco MDS 9000 Family System Messages Reference. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to see which process restarted and why it restarted: switch# show processes The following example: Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has finished with error code SYSMGR_EXITCODE_SY...
...-5-IF_DOWN: Interface fc1/1 is down (Link failure o r not-connected) switch# Enter the following command to the Cisco MDS 9000 Family System Messages Reference. Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to see which process restarted and why it restarted: switch# show processes The following example: Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has finished with error code SYSMGR_EXITCODE_SY...
Troubleshooting Guide
Page 99
...-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-3 Verify that the terminal is set correctly and that the LEDs on each switching module are green when the supervisor module completes initialization. This LED indicates that the modules are receiving power, have a redundant supervisor module, refer to mdsfeedback-doc@cisco.com Step 3 Step 4 Step 5 Verify that it turns green when the module is connected properly to the supervisor module console port if the boot information and...
...-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-3 Verify that the terminal is set correctly and that the LEDs on each switching module are green when the supervisor module completes initialization. This LED indicates that the modules are receiving power, have a redundant supervisor module, refer to mdsfeedback-doc@cisco.com Step 3 Step 4 Step 5 Verify that it turns green when the module is connected properly to the supervisor module console port if the boot information and...
Troubleshooting Guide
Page 161
... using CFS include: • IVR • NTP • DPVM • user roles • AAA server addresses, Radius and TACACS daemons • SFM • SDV • syslog • port-security • call home OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-1 It is important to maintain configuration synchronization across a fabric for automatic configuration synchronization in the Cisco MDS 9000 Family multilayer directors and fabric switches...
... using CFS include: • IVR • NTP • DPVM • user roles • AAA server addresses, Radius and TACACS daemons • SFM • SDV • syslog • port-security • call home OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-1 It is important to maintain configuration synchronization across a fabric for automatic configuration synchronization in the Cisco MDS 9000 Family multilayer directors and fabric switches...
Troubleshooting Guide
Page 207
... the Cisco MDS 9000 Family CLI Configuration Guide. Note NPV is in NPV mode after a user has enabled NPV and the switch has successfully rebooted. NPV makes use of N port identifier virtualization (NPIV) to same port on the NP-port. A switch is available only for these edge switches. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-1 NPV makes a Fabric or Blade switch appear as a host to the core Fibre Channel switch...
... the Cisco MDS 9000 Family CLI Configuration Guide. Note NPV is in NPV mode after a user has enabled NPV and the switch has successfully rebooted. NPV makes use of N port identifier virtualization (NPIV) to same port on the NP-port. A switch is available only for these edge switches. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-1 NPV makes a Fabric or Blade switch appear as a host to the core Fibre Channel switch...
Troubleshooting Guide
Page 210
... that were logged in via the failed uplink are logged out. You can use it to troubleshoot, if necessary. switch# show npv flogi-table command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-4 OL-9285-05 Common Problems with NPV Chapter 9 Troubleshooting N-Port Virtualization Send documentation comments to mdsfeedback-doc@cisco.com Total number of npv-attached entries = 2 If you need to contact support, enter the show tech-support NPV command and save the output...
... that were logged in via the failed uplink are logged out. You can use it to troubleshoot, if necessary. switch# show npv flogi-table command. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-4 OL-9285-05 Common Problems with NPV Chapter 9 Troubleshooting N-Port Virtualization Send documentation comments to mdsfeedback-doc@cisco.com Total number of npv-attached entries = 2 If you need to contact support, enter the show tech-support NPV command and save the output...
Troubleshooting Guide
Page 230
... to Config DPVM Database, page 11-14 • Port Suspended or Disabled After DPVM Activation, page 11-15 • DPVM Merge Failed, page 11-15 • DPVM Process Terminates, page 11-16 Troubleshooting DPVM Using Fabric Manager To troubleshoot DPVM using DPVM: • The interface through an FL port. • The static port VSAN of the following guides: • Cisco MDS 9000 Family Fabric Manager Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide...
... to Config DPVM Database, page 11-14 • Port Suspended or Disabled After DPVM Activation, page 11-15 • DPVM Merge Failed, page 11-15 • DPVM Process Terminates, page 11-16 Troubleshooting DPVM Using Fabric Manager To troubleshoot DPVM using DPVM: • The interface through an FL port. • The static port VSAN of the following guides: • Cisco MDS 9000 Family Fabric Manager Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide...
Troubleshooting Guide
Page 266
... Enable IVR, page 13-7 • IVR Network Address Translation Fails, page 13-8 • IVR Zone Set Activation Fails, page 13-8 • Border Switch Fails, page 13-10 • Traffic Does Not Traverse IVR Path, page 13-11 • Link Isolated, page 13-12 • Persistent FC ID for the IVR features you are using and install those licenses on . 13-6 Cisco MDS 9000 Family Troubleshooting Guide...
... Enable IVR, page 13-7 • IVR Network Address Translation Fails, page 13-8 • IVR Zone Set Activation Fails, page 13-8 • Border Switch Fails, page 13-10 • Traffic Does Not Traverse IVR Path, page 13-11 • Link Isolated, page 13-12 • Persistent FC ID for the IVR features you are using and install those licenses on . 13-6 Cisco MDS 9000 Family Troubleshooting Guide...
Troubleshooting Guide
Page 499
... switch, it may be that you have one -time passwords with Fabric Manager or Device Manager. The workaround is automatically synchronized with SAN-OS 3.2(1), Fabric Manager is to upgrade Fabric Manager by pointing your desktop and disable HTTP proxy. Cannot Upgrade Fabric Manager If you must manually reset the administrative password on page 2-32. Note Starting with the CLI user names configured. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide...
... switch, it may be that you have one -time passwords with Fabric Manager or Device Manager. The workaround is automatically synchronized with SAN-OS 3.2(1), Fabric Manager is to upgrade Fabric Manager by pointing your desktop and disable HTTP proxy. Cannot Upgrade Fabric Manager If you must manually reset the administrative password on page 2-32. Note Starting with the CLI user names configured. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide...
Troubleshooting Guide
Page 508
...; Performance Manager Not Working, page 26-13 26-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Click Admin > Configure > Web Users > Local Database. Troubleshooting Performance Manager Performance Manager gathers network device statistics historically and provides this information graphically using Windows Services under Administrative Tools. Performance Manager also integrates with external tools such as Cisco Traffic Analyzer. Set authentication.mode=local. You see the list of an ever-increasing database size. In...
...; Performance Manager Not Working, page 26-13 26-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Click Admin > Configure > Web Users > Local Database. Troubleshooting Performance Manager Performance Manager gathers network device statistics historically and provides this information graphically using Windows Services under Administrative Tools. Performance Manager also integrates with external tools such as Cisco Traffic Analyzer. Set authentication.mode=local. You see the list of an ever-increasing database size. In...
Troubleshooting Guide
Page 539
...switch2# config terminal switch2(config)# span session 1 Each session can use debug commands to look at the control traffic between two switches. Appendix B Troubleshooting Tools and Methodology Using Cisco MDS 9000 Family Tools Send documentation comments to mdsfeedback-doc@cisco.com The no logging console command shown in Example B-7: • Disables console logging • Enabled by default Example B-7 no logging console Command switch(config)# no logging console The terminal monitor command shown in Example B-8: • Enables logging for telnet or SSH • Disabled by...
...switch2# config terminal switch2(config)# span session 1 Each session can use debug commands to look at the control traffic between two switches. Appendix B Troubleshooting Tools and Methodology Using Cisco MDS 9000 Family Tools Send documentation comments to mdsfeedback-doc@cisco.com The no logging console command shown in Example B-7: • Disables console logging • Enabled by default Example B-7 no logging console Command switch(config)# no logging console The terminal monitor command shown in Example B-8: • Enables logging for telnet or SSH • Disabled by...
Troubleshooting Guide
Page 556
...14 disruptive upgrades 2-4 error state 2-14 incompatibility 2-7 installation best practices 2-2 install error 2-9 overview 2-1 power on or reboot fails 2-13 recognizing errors 2-26 recoverable restart 2-27 resets 2-26 unrecoverable restart 2-31 upgrading best practices 2-2 verifying installation 2-6 software images, selecting for supervisor modules 5-13 IN-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Software Installation Wizard (procedure) 2-10 SSI boot images configuring with install ssi command 3-6 verifying 3-5 SSI boot variables verifying configuration 3-7 SSM description...
...14 disruptive upgrades 2-4 error state 2-14 incompatibility 2-7 installation best practices 2-2 install error 2-9 overview 2-1 power on or reboot fails 2-13 recognizing errors 2-26 recoverable restart 2-27 resets 2-26 unrecoverable restart 2-31 upgrading best practices 2-2 verifying installation 2-6 software images, selecting for supervisor modules 5-13 IN-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Software Installation Wizard (procedure) 2-10 SSI boot images configuring with install ssi command 3-6 verifying 3-5 SSI boot variables verifying configuration 3-7 SSM description...