Troubleshooting Guide
Page 1
Send documentation comments to mdsfeedback-doc@cisco.com Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Cisco MDS SAN-OS for Release 3.3(1) May 2008 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 Text Part Number: OL-9285-05
Send documentation comments to mdsfeedback-doc@cisco.com Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Cisco MDS SAN-OS for Release 3.3(1) May 2008 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 Text Part Number: OL-9285-05
Troubleshooting Guide
Page 2
... TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x © 2009 Cisco Systems, Inc. Send documentation comments to the Human Network are the property of California. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University...
... TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x © 2009 Cisco Systems, Inc. Send documentation comments to the Human Network are the property of California. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University...
Troubleshooting Guide
Page 3
... Document Conventions i-xxix Related Documentation i-xxx Release Notes i-xxx Compatibility Information i-xxx Regulatory Compliance and Safety Information i-xxx Hardware Installation i-xxx Cisco Fabric Manager i-xxxi Command-Line Interface i-xxxi Troubleshooting and Reference i-xxxi Installation and Configuration Note i-xxxi Obtaining Documentation, Obtaining Support, and ... Primary Troubleshooting Flowchart 1-9 System Messages 1-10 System Message Text 1-10 Syslog Server Implementation 1-10 Implementing Syslog with Fabric Manager 1-11 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iii
... Document Conventions i-xxix Related Documentation i-xxx Release Notes i-xxx Compatibility Information i-xxx Regulatory Compliance and Safety Information i-xxx Hardware Installation i-xxx Cisco Fabric Manager i-xxxi Command-Line Interface i-xxxi Troubleshooting and Reference i-xxxi Installation and Configuration Note i-xxxi Obtaining Documentation, Obtaining Support, and ... Primary Troubleshooting Flowchart 1-9 System Messages 1-10 System Message Text 1-10 Syslog Server Implementation 1-10 Implementing Syslog with Fabric Manager 1-11 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iii
Troubleshooting Guide
Page 4
... Guidelines for Reboots 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager Installations 2-5 Verifying Cisco SAN-OS Software Installations 2-6 Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-7 Software Installation Reports an Incompatibility 2-7 Diagnosing Compatibility Issues 2-7 Software Installation Ends with Error 2-9... States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-05
... Guidelines for Reboots 2-4 Disruptive Module Upgrades 2-4 Troubleshooting a Nondisruptive Upgrade on a Fabric Switch 2-4 Troubleshooting Fabric Manager Installations 2-5 Verifying Cisco SAN-OS Software Installations 2-6 Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-7 Software Installation Reports an Incompatibility 2-7 Diagnosing Compatibility Issues 2-7 Software Installation Ends with Error 2-9... States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-05
Troubleshooting Guide
Page 5
Fan LED is Red 4-6 Troubleshooting the Power Supplies 4-7 Troubleshooting Fan Issues 4-8 Fan Is Not Spinning 4-8 Fan Is Spinning; Contents Send documentation comments to mdsfeedback-doc@cisco.com Recovering the Administrator Password 2-32 Miscellaneous Software Image Issues 2-32 All Ports Down Because of System Health Failure 2-33 Switch Reboots after FCIP Reload 2-... is On 4-6 Power Supply Fan Ok LED is Red 4-8 Troubleshooting a Fan Failure Using Device Manager 4-9 Troubleshooting a Fan Failure Using the CLI 4-10 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x v
Fan LED is Red 4-6 Troubleshooting the Power Supplies 4-7 Troubleshooting Fan Issues 4-8 Fan Is Not Spinning 4-8 Fan Is Spinning; Contents Send documentation comments to mdsfeedback-doc@cisco.com Recovering the Administrator Password 2-32 Miscellaneous Software Image Issues 2-32 All Ports Down Because of System Health Failure 2-33 Switch Reboots after FCIP Reload 2-... is On 4-6 Power Supply Fan Ok LED is Red 4-8 Troubleshooting a Fan Failure Using Device Manager 4-9 Troubleshooting a Fan Failure Using the CLI 4-10 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x v
Troubleshooting Guide
Page 6
Contents Send documentation comments to mdsfeedback-doc@cisco.com Temperature Threshold Violations 4-11 Troubleshooting Clock Module Issues 4-12 Troubleshooting Other Hardware Issues 4-13 Troubleshooting Supervisor Issues 4-14 Active Supervisor Reboots 4-15 Standby Supervisor ... Module Using the CLI 4-37 Module Resets 4-38 5 C H A P T E R Troubleshooting Mixed Generation Hardware 5-1 Overview 5-1 Port Groups 5-2 Port Speed Mode 5-3 Dynamic Bandwidth Management 5-3 Out-of-Service Interfaces 5-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x vi OL-9285-05
Contents Send documentation comments to mdsfeedback-doc@cisco.com Temperature Threshold Violations 4-11 Troubleshooting Clock Module Issues 4-12 Troubleshooting Other Hardware Issues 4-13 Troubleshooting Supervisor Issues 4-14 Active Supervisor Reboots 4-15 Standby Supervisor ... Module Using the CLI 4-37 Module Resets 4-38 5 C H A P T E R Troubleshooting Mixed Generation Hardware 5-1 Overview 5-1 Port Groups 5-2 Port Speed Mode 5-3 Dynamic Bandwidth Management 5-3 Out-of-Service Interfaces 5-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x vi OL-9285-05
Troubleshooting Guide
Page 7
Contents Send documentation comments to mdsfeedback-doc@cisco.com Port Index Availability 5-4 Combining Modules and Supervisors 5-7 Initial Troubleshooting Checklist 5-7 Generation 1 and Generation 2 Issues 5-7 Module Does Not Come Online 5-8 Verifying Port ... Multiple Modules 6-8 Grace Period Alerts 6-9 Checking in the Fabric Manager Server License From Device Manager 6-10 License Listed as Missing 6-11 7 C H A P T E R Troubleshooting Cisco Fabric Services 7-1 Overview 7-1 Initial Troubleshooting Checklist 7-2 Verifying CFS Using Fabric Manager 7-3 Verifying CFS Using the CLI 7-3 OL-9285-05...
Contents Send documentation comments to mdsfeedback-doc@cisco.com Port Index Availability 5-4 Combining Modules and Supervisors 5-7 Initial Troubleshooting Checklist 5-7 Generation 1 and Generation 2 Issues 5-7 Module Does Not Come Online 5-8 Verifying Port ... Multiple Modules 6-8 Grace Period Alerts 6-9 Checking in the Fabric Manager Server License From Device Manager 6-10 License Listed as Missing 6-11 7 C H A P T E R Troubleshooting Cisco Fabric Services 7-1 Overview 7-1 Initial Troubleshooting Checklist 7-2 Verifying CFS Using Fabric Manager 7-3 Verifying CFS Using the CLI 7-3 OL-9285-05...
Troubleshooting Guide
Page 8
Contents Send documentation comments to mdsfeedback-doc@cisco.com Merge Failure Troubleshooting 7-5 Recovering from a Merge Failure with Fabric Manager 7-6 Recovering from a Merge Failure with the CLI 7-6 Lock Failure Troubleshooting 7-7 Resolving Lock Failure Issues ... State 8-13 Troubleshooting Port Problems 8-15 Port Remains in Initializing State 8-16 Troubleshooting Port Registration Issues Using the CLI 8-17 Unexpected Link Flapping Occurs 8-21 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x viii OL-9285-05
Contents Send documentation comments to mdsfeedback-doc@cisco.com Merge Failure Troubleshooting 7-5 Recovering from a Merge Failure with Fabric Manager 7-6 Recovering from a Merge Failure with the CLI 7-6 Lock Failure Troubleshooting 7-7 Resolving Lock Failure Issues ... State 8-13 Troubleshooting Port Problems 8-15 Port Remains in Initializing State 8-16 Troubleshooting Port Registration Issues Using the CLI 8-17 Unexpected Link Flapping Occurs 8-21 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x viii OL-9285-05
Troubleshooting Guide
Page 9
Contents Send documentation comments to mdsfeedback-doc@cisco.com Link Initialization Flow 8-23 Viewing Port Counters 8-25 Port Bounces Between Initializing and Offline States 8-26 Troubleshooting ELP Issues Using the CLI 8-27 E Port ...-3 Cannot Configure a PortChannel 10-3 Newly Added Interface Does Not Come Online In a PortChannel 10-4 Configuring Port Channel Modes Using Fabric Manager 10-4 Trunking Issues 10-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x ix
Contents Send documentation comments to mdsfeedback-doc@cisco.com Link Initialization Flow 8-23 Viewing Port Counters 8-25 Port Bounces Between Initializing and Offline States 8-26 Troubleshooting ELP Issues Using the CLI 8-27 E Port ...-3 Cannot Configure a PortChannel 10-3 Newly Added Interface Does Not Come Online In a PortChannel 10-4 Configuring Port Channel Modes Using Fabric Manager 10-4 Trunking Issues 10-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x ix
Troubleshooting Guide
Page 10
Contents Send documentation comments to mdsfeedback-doc@cisco.com Cannot Configure Trunking 10-5 VSAN Traffic Does Not Traverse Trunk 10-5 11 C H A P T E R Troubleshooting VSANs, Domains, and FSPF 11-1 Overview 11-1 Initial Troubleshooting Checklist 11-1 ...-17 Switch Cannot See Other Switches in a VSAN 11-18 FC Domain ID Overlap 11-18 Assigning a New Domain ID Using Fabric Manager 11-19 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x x OL-9285-05
Contents Send documentation comments to mdsfeedback-doc@cisco.com Cannot Configure Trunking 10-5 VSAN Traffic Does Not Traverse Trunk 10-5 11 C H A P T E R Troubleshooting VSANs, Domains, and FSPF 11-1 Overview 11-1 Initial Troubleshooting Checklist 11-1 ...-17 Switch Cannot See Other Switches in a VSAN 11-18 FC Domain ID Overlap 11-18 Assigning a New Domain ID Using Fabric Manager 11-19 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x x OL-9285-05
Troubleshooting Guide
Page 11
Contents Send documentation comments to mdsfeedback-doc@cisco.com Assigning a New Domain ID Using the CLI 11-19 Using Fabric Reconfiguration for Domain ID Assignments 11-20 CFS Distribution of Domain ID List ... 13-1 Configuration Guidelines 13-1 Transit VSANs 13-2 Border Switches 13-2 Limitations and Restrictions 13-2 Initial Troubleshooting Checklist 13-3 Verifying IVR Configuration Using Fabric Manager 13-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xi
Contents Send documentation comments to mdsfeedback-doc@cisco.com Assigning a New Domain ID Using the CLI 11-19 Using Fabric Reconfiguration for Domain ID Assignments 11-20 CFS Distribution of Domain ID List ... 13-1 Configuration Guidelines 13-1 Transit VSANs 13-2 Border Switches 13-2 Limitations and Restrictions 13-2 Initial Troubleshooting Checklist 13-3 Verifying IVR Configuration Using Fabric Manager 13-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xi
Troubleshooting Guide
Page 12
... 14-12 Resolving Mismatched Default Zone Policies Using Fabric Manager 14-12 Resolving Mismatched Default Zone Policies Using the CLI 14-13 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xii OL-9285-05 Contents Send documentation comments to mdsfeedback-doc...@cisco.com Verifying IVR Configuration Using the CLI 13-4 IVR Enhancements by Cisco SAN-OS Release 13-5 IVR Issues 13-6 IVR Licensing Issues 13-6 Cannot Enable IVR 13-7 IVR Network Address Translation Fails 13-8...
... 14-12 Resolving Mismatched Default Zone Policies Using Fabric Manager 14-12 Resolving Mismatched Default Zone Policies Using the CLI 14-13 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xii OL-9285-05 Contents Send documentation comments to mdsfeedback-doc...@cisco.com Verifying IVR Configuration Using the CLI 13-4 IVR Enhancements by Cisco SAN-OS Release 13-5 IVR Issues 13-6 IVR Licensing Issues 13-6 Cannot Enable IVR 13-7 IVR Network Address Translation Fails 13-8...
Troubleshooting Guide
Page 13
Contents Send documentation comments to mdsfeedback-doc@cisco.com Zone Merge Failure 14-13 Recovering from Link Isolation 14-15 Resolving a Link Isolation Because of a Failed Zone Merge Using Fabric Manager 14-15 ... C H A P T E R Troubleshooting FICON 16-1 FICON Overview 16-1 FICON Port Numbering 16-2 Default FICON Port Numbering Scheme 16-2 Reserved FICON Port Numbering Scheme 16-4 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xiii
Contents Send documentation comments to mdsfeedback-doc@cisco.com Zone Merge Failure 14-13 Recovering from Link Isolation 14-15 Resolving a Link Isolation Because of a Failed Zone Merge Using Fabric Manager 14-15 ... C H A P T E R Troubleshooting FICON 16-1 FICON Overview 16-1 FICON Port Numbering 16-2 Default FICON Port Numbering Scheme 16-2 Reserved FICON Port Numbering Scheme 16-4 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xiii
Troubleshooting Guide
Page 14
Contents Send documentation comments to mdsfeedback-doc@cisco.com FICON Configuration Files 16-5 CUP In-Band Management 16-6 Fabric Binding 16-6 FICON Configuration Requirements 16-6 Initial Troubleshooting Checklist 16-7 Common Troubleshooting Tools in ...-9 Verifying RADIUS Server Groups Using the CLI 17-9 Verifying TACACS+ Server Groups Using Fabric Manager 17-9 Verifying TACACS+ Server Groups Using the CLI 17-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xiv OL-9285-05
Contents Send documentation comments to mdsfeedback-doc@cisco.com FICON Configuration Files 16-5 CUP In-Band Management 16-6 Fabric Binding 16-6 FICON Configuration Requirements 16-6 Initial Troubleshooting Checklist 16-7 Common Troubleshooting Tools in ...-9 Verifying RADIUS Server Groups Using the CLI 17-9 Verifying TACACS+ Server Groups Using Fabric Manager 17-9 Verifying TACACS+ Server Groups Using the CLI 17-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xiv OL-9285-05
Troubleshooting Guide
Page 15
... User Is Not in Any Configured Role 17-10 User Cannot Access Certain Features 17-11 Troubleshooting RADIUS and TACACS+ With Cisco ACS 17-11 18 C H A P T E R Troubleshooting Users and Roles 18-1 Overview 18-1 User Accounts 18...Using the CLI 18-11 User Cannot Configure E Ports 18-11 Unexpected User Displayed in Logs 18-12 Troubleshooting Users and Roles with Cisco ACS 18-12 19 C H A P T E R Troubleshooting FC-SP, Port Security, and Fabric Binding 19-1 FC-SP ...FC-SP Issues 19-4 Switch or Host Blocked from Fabric 19-4 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xv
... User Is Not in Any Configured Role 17-10 User Cannot Access Certain Features 17-11 Troubleshooting RADIUS and TACACS+ With Cisco ACS 17-11 18 C H A P T E R Troubleshooting Users and Roles 18-1 Overview 18-1 User Accounts 18...Using the CLI 18-11 User Cannot Configure E Ports 18-11 Unexpected User Displayed in Logs 18-12 Troubleshooting Users and Roles with Cisco ACS 18-12 19 C H A P T E R Troubleshooting FC-SP, Port Security, and Fabric Binding 19-1 FC-SP ...FC-SP Issues 19-4 Switch or Host Blocked from Fabric 19-4 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xv
Troubleshooting Guide
Page 16
... the CLI 19-5 Verifying Local FC-SP Database Using Fabric Manager 19-5 Verifying Local FC-SP Database Using the CLI 19-6 Authentication Fails When Using Cisco ACS 19-7 Port Security Issues 19-7 Device Does Not Log into a Switch When AutoLearn Is Disabled 19-8 Device Does Not Log into a Switch When Autolearn... 20-1 iSCSI Restrictions 20-2 iSLB Restrictions 20-2 Initial Troubleshooting Checklist 20-3 Common Troubleshooting Tools in Fabric Manager 20-3 Common Troubleshooting Commands in the CLI 20-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xvi OL-9285-05
... the CLI 19-5 Verifying Local FC-SP Database Using Fabric Manager 19-5 Verifying Local FC-SP Database Using the CLI 19-6 Authentication Fails When Using Cisco ACS 19-7 Port Security Issues 19-7 Device Does Not Log into a Switch When AutoLearn Is Disabled 19-8 Device Does Not Log into a Switch When Autolearn... 20-1 iSCSI Restrictions 20-2 iSLB Restrictions 20-2 Initial Troubleshooting Checklist 20-3 Common Troubleshooting Tools in Fabric Manager 20-3 Common Troubleshooting Commands in the CLI 20-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xvi OL-9285-05
Troubleshooting Guide
Page 17
Contents Send documentation comments to mdsfeedback-doc@cisco.com IP Issues 20-5 Verifying Basic Connectivity 20-5 Verifying Basic Connectivity Using Device Manager 20-5 Verifying Basic Connectivity Using the CLI 20-5 Verification of Switch ... Special Frame Tunnel Creation and Monitoring 20-30 Configuring and Displaying an FCIP Tunnel with Special Frame Using the CLI 20-31 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xvii
Contents Send documentation comments to mdsfeedback-doc@cisco.com IP Issues 20-5 Verifying Basic Connectivity 20-5 Verifying Basic Connectivity Using Device Manager 20-5 Verifying Basic Connectivity Using the CLI 20-5 Verification of Switch ... Special Frame Tunnel Creation and Monitoring 20-30 Configuring and Displaying an FCIP Tunnel with Special Frame Using the CLI 20-31 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xvii
Troubleshooting Guide
Page 18
Large Iscsi Config Present..." 20-61 xviii Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Contents Send documentation comments to mdsfeedback-doc@cisco.com Special Frame Misconfiguration Example 20-33 Troubleshooting FCIP Link Flaps 20-34 Troubleshooting FCIP ISL Link Failures 20-34 Troubleshooting FCIP and Compression 20-...
Large Iscsi Config Present..." 20-61 xviii Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Contents Send documentation comments to mdsfeedback-doc@cisco.com Special Frame Misconfiguration Example 20-33 Troubleshooting FCIP Link Flaps 20-34 Troubleshooting FCIP ISL Link Failures 20-34 Troubleshooting FCIP and Compression 20-...
Troubleshooting Guide
Page 19
Contents Send documentation comments to mdsfeedback-doc@cisco.com iSLB Commit Timeout 20-61 Session Down-"pWWN in Use At Remote Switch" 20-62 Redirected Session Does Not Come Up 20-62 iSLB ...-4 Common Troubleshooting Tools in Fabric Manager 22-4 Common Troubleshooting Commands in the CLI 22-5 IPsec Issues 22-5 Verifying IKE Configuration Compatibility 22-6 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xix
Contents Send documentation comments to mdsfeedback-doc@cisco.com iSLB Commit Timeout 20-61 Session Down-"pWWN in Use At Remote Switch" 20-62 Redirected Session Does Not Come Up 20-62 iSLB ...-4 Common Troubleshooting Tools in Fabric Manager 22-4 Common Troubleshooting Commands in the CLI 22-5 IPsec Issues 22-5 Verifying IKE Configuration Compatibility 22-6 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xix
Troubleshooting Guide
Page 20
Contents Send documentation comments to mdsfeedback-doc@cisco.com Verifying IPsec Configuration Compatibility Using Fabric Manager 22-6 Verifying IPsec Configuration Compatibility Using the CLI 22-7 Verifying Security Policy Databases Compatibility 22-8 Verifying Interface ...-2 PKI Enrollment Support 24-2 Maximum Limits 24-3 Initial Troubleshooting Checklist 24-3 Common Troubleshooting Tools in Fabric Manager 24-3 Common Troubleshooting Commands in the CLI 24-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xx OL-9285-05
Contents Send documentation comments to mdsfeedback-doc@cisco.com Verifying IPsec Configuration Compatibility Using Fabric Manager 22-6 Verifying IPsec Configuration Compatibility Using the CLI 22-7 Verifying Security Policy Databases Compatibility 22-8 Verifying Interface ...-2 PKI Enrollment Support 24-2 Maximum Limits 24-3 Initial Troubleshooting Checklist 24-3 Common Troubleshooting Tools in Fabric Manager 24-3 Common Troubleshooting Commands in the CLI 24-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xx OL-9285-05