Database Management Services, Oracle Database Management Solution, Oracle Databases, Oracle Exadata

Exadata Hardware Generation at a Glance (V1 to X8)

By Netsoftmate on

April 5, 2019

- Database Management Services, Oracle Database Management Solution, Oracle Databases, Oracle Exadata

An engineered system comprising of Compute Nodes, Storage Cells and Infiniband  – all of it packaged inside a single physical cabinet called “Exadata Rack”

Exadata Hardware Generation At A Glance

 

Exadata Database Machine X8-2

0

---

Uncategorized

Quick Overview of Exadata Database Machine X8-2

By Netsoftmate on

April 3, 2019

- Uncategorized

Oracle announced the next-generation Oracle Exadata X8 with significant hardware and software enhancements in overall performance, storage capacity, network bandwidth, and automation. Exadata X7 delivers extreme performance and reliability to run the largest, most business-critical database workloads.


Oracle Exadata X8-2 quick glance:

• – Latest Intel Xeon (8260) Processors (2.4GHz) 2*24 cores per database server (384 core in a Full Rack)
•  
• – Exadata X8 Capacity-on-Demand enables at least 14 cores per server
•  
• – Delivers up to 60 percent faster throughput than previous models
•  
• – NO changes in Database server Physical memory (upto 12TB in a full Rack)
•  
• – Latest Intel Xeon (5218) Processors (2.3GHz) 2*16 cores per Storage server (448 core in a Full Rack)
•  
• – NO increase in the capacity of Extreme Flash storage (716.8TB in a full Rack)
•  
• – 40% increase in disk capacity (2352TB in a full Rack)
•  
• – Exadata X8 introduces new Exadata storage option – Extended (XT) Storage Server
•  
• – Each Exadata XT Storage Server includes twelve 14 TB SAS disk drives
•  
• – A full rack Exadata X8-2 system has:
  •  
  • – Raw capacity of 2.3 petabytes of disk storage & 358.4TB Flash
  •  
  • – 720 terabytes of NVMe all-Flash storage
  •  
  • – Raw capacity of 2.3 petabytes of disk storage & No flash
•  
• – Additional network 2x 10/25 Gb optical Ethernet (client – optional)
•  
• – Available in Oracle Public Cloud – Oracle Database Exadata Cloud Service

For more information please visit –

https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadata-x8-2-ds.pdf
 
https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadata-x8-8-ds.pdf

0

---

Step by Step Exadata Infiniband Switch Patching 18c

By Netsoftmate on

February 28, 2019

- Uncategorized

Overview

• The Exadata network grid consists of multiple Sun QDR InfiniBand switches.
• IB Switches are used for the storage network as well as the Oracle RAC interconnect.
• Exadata compute nodes and storage cells are configured with dual-port InfiniBand ports and connect to each of the two leaf switches.
• You can access IB Switches using command line and Web ILOM
• IB Switches run Linux operating system.

In this article I will demonstrate how to patch or upgrade Oracle Exadata IB Switches.

About Infiniband Switch Patching

• Starting with release 11.2.3.3.0, the patchmgr utility is used to upgrade and downgrade the InfiniBand switches.
• IB Switch patch is delievered with Exadata storage patch.
• IB Switch patches are released semi annually to annually.
• IB Switch can be patched in Rolling fashion only.

Environment

• Exadata Half Rack X4-2
• 4 Compute nodes, 7 Storage cells and 2 IB Switches
• Current IB Switch Version 2.2.7-1

Step by Step Infiniband Switch Patching

 

• Identify the number of switches in clusters.

[root@dm01dbadm01 ~]# ibswitches
Switch  : 0x002128469b8aa0a0 ports 36 “SUN DCS 36P QDR dm01sw-iba01 10.209.41.246” enhanced port 0 lid 5 lmc 0
Switch  : 0x002128469b97a0a0 ports 36 “SUN DCS 36P QDR dm01sw-ibb01 10.209.41.247” enhanced port 0 lid 4 lmc 0

• Identify the current IB switch software version on all the Switches

[root@dm01db01 patch_18.1.12.0.0.190111]# ssh dm01sw-iba01 version
SUN DCS 36p version: 2.2.7-1
Build time: Aug  4 2017 12:20:53
SP board info:
Manufacturing Date: 2014.05.20
Serial Number: “NCDFxxxxx”
Hardware Revision: 0x0107
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

• Log in to Exadata Compute node 1 as root user and navigate the Exadata Storage Software staging area

[root@dm01dbadm01 ESS_121220]# cd /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111
 

[root@dm01dbadm01 patch_18.1.12.0.0.190111]# pwd
/u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111

• Create a file named ibswitches.lst and enter IB switch names one per line as follows:

[root@dm01dbadm01 patch_18.1.12.0.0.190111]# vi ~/ibswitch_group
dm01sw-ibb01
dm01sw-iba01

[root@dm01dbadm01 patch_18.1.12.0.0.190111]# cat ~/ibswitch_group
dm01sw-ibb01
dm01sw-iba01

• Execute the following to perform the IB Switch precheck

[root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -ibswitches ~/ibswitch_group -upgrade -ibswitch_precheck

2019-02-10 03:02:46 -0600 1 of 1 :Working: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
 —– InfiniBand switch update process started 2019-02-10 03:02:47 -0600 —–
[NOTE     ] Log file at /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/upgradeIBSwitch.log

[INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ibb01 dm01sw-iba01 )
[SUCCESS  ] Verifying Network connectivity to dm01sw-ibb01
[SUCCESS  ] Verifying Network connectivity to dm01sw-iba01
[SUCCESS  ] Validating verify-topology output
[INFO     ] Master Subnet Manager is set to “dm01sw-ibb01” in all Switches
[INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-ibb01 first to 2.2.7-2

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
[INFO     ] Starting pre-update validation on dm01sw-ibb01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 26M
[SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:03:03
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
[INFO     ] Finished pre-update validation on dm01sw-ibb01
[SUCCESS  ] Pre-update validation on dm01sw-ibb01

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
[INFO     ] Starting pre-update validation on dm01sw-ibb01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 26M
[SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:03:34
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
[INFO     ] Finished pre-update validation on dm01sw-ibb01
[SUCCESS  ] Pre-update validation on dm01sw-ibb01
[SUCCESS  ] Prereq check on dm01sw-ibb01
[INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-iba01 first to 2.2.7-2

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
[INFO     ] Starting pre-update validation on dm01sw-iba01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 26M
[SUCCESS  ] NTP daemon is running on dm01sw-iba01.
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:04:06
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
[INFO     ] Finished pre-update validation on dm01sw-iba01
[SUCCESS  ] Pre-update validation on dm01sw-iba01

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
[INFO     ] Starting pre-update validation on dm01sw-iba01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 26M
[SUCCESS  ] NTP daemon is running on dm01sw-iba01.
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:04:36
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
[INFO     ] Finished pre-update validation on dm01sw-iba01
[SUCCESS  ] Pre-update validation on dm01sw-iba01
[SUCCESS  ] Prereq check on dm01sw-iba01
[SUCCESS  ] Overall status

 —– InfiniBand switch update process ended 2019-02-10 03:05:00 -0600 —–
2019-02-10 03:05:00 -0600 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).

• Upgrade the IB Switches using the following command:

[root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -ibswitches ~/ibswitch_group -upgrade

2019-02-10 03:07:26 -0600 1 of 1 :Working: DO: Initiate upgrade of InfiniBand switches to 2.2.11-2. Expect up to 40 minutes for each switch
                                                  
 —– InfiniBand switch update process started 2019-02-10 03:07:27 -0600 —–
[NOTE     ] Log file at /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/upgradeIBSwitch.log

[INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ibb01 dm01sw-iba01 )
[SUCCESS  ] Verifying Network connectivity to dm01sw-ibb01
[SUCCESS  ] Verifying Network connectivity to dm01sw-iba01
[SUCCESS  ] Validating verify-topology output
[INFO     ] Proceeding with upgrade of InfiniBand switches to version 2.2.11_2
[INFO     ] Master Subnet Manager is set to “dm01sw-ibb01” in all Switches
[INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-ibb01 first to 2.2.7-2

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
[INFO     ] Starting pre-update validation on dm01sw-ibb01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 26M
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
[SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:07:43
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
[INFO     ] Finished pre-update validation on dm01sw-ibb01
[SUCCESS  ] Pre-update validation on dm01sw-ibb01
[INFO     ] Package will be downloaded at firmware update time via scp
[SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
[SUCCESS  ] Execute plugin check for Patching on dm01sw-ibb01
[INFO     ] Starting upgrade on dm01sw-ibb01 to 2.2.7_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO     ] Rebooting dm01sw-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS  ] Load firmware 2.2.7_2 onto dm01sw-ibb01
[SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
[SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-ibb01
[INFO     ] Set SMPriority to 5 on dm01sw-ibb01
[INFO     ] Rebooting dm01sw-ibb01. Wait for 4 minutes before continuing
[SUCCESS  ] Reboot dm01sw-ibb01
[SUCCESS  ] SUCCESS
[INFO     ] Starting post-update validation on dm01sw-ibb01
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
[SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:29:11
[INFO     ] /conf/configvalid is 1
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
[SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ibb01
[INFO     ] Finished post-update validation on dm01sw-ibb01
[SUCCESS  ] Post-update validation on dm01sw-ibb01

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
[INFO     ] Starting pre-update validation on dm01sw-ibb01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 28M
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
[SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:29:39
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
[INFO     ] Finished pre-update validation on dm01sw-ibb01
[SUCCESS  ] Pre-update validation on dm01sw-ibb01
[INFO     ] Package will be downloaded at firmware update time via scp
[SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
[SUCCESS  ] Execute plugin check for Patching on dm01sw-ibb01
[INFO     ] Starting upgrade on dm01sw-ibb01 to 2.2.11_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO     ] Rebooting dm01sw-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS  ] Load firmware 2.2.11_2 onto dm01sw-ibb01
[SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
[SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-ibb01
[INFO     ] Set SMPriority to 5 on dm01sw-ibb01
[INFO     ] Rebooting dm01sw-ibb01. Wait for 4 minutes before continuing
[SUCCESS  ] Reboot dm01sw-ibb01
[SUCCESS  ] SUCCESS
[INFO     ] Starting post-update validation on dm01sw-ibb01
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
[SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:51:03
[INFO     ] /conf/configvalid is 1
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
[SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ibb01
[INFO     ] Finished post-update validation on dm01sw-ibb01
[SUCCESS  ] Post-update validation on dm01sw-ibb01
[SUCCESS  ] Update InfiniBand switch dm01sw-ibb01 to 2.2.11_2
[INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-iba01 first to 2.2.7-2
[INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
[INFO     ] Starting pre-update validation on dm01sw-iba01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 26M
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
[SUCCESS  ] NTP daemon is running on dm01sw-iba01.
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:51:38
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
[INFO     ] Finished pre-update validation on dm01sw-iba01
[SUCCESS  ] Pre-update validation on dm01sw-iba01
[INFO     ] Package will be downloaded at firmware update time via scp
[SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
[SUCCESS  ] Execute plugin check for Patching on dm01sw-iba01
[INFO     ] Starting upgrade on dm01sw-iba01 to 2.2.7_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO     ] Rebooting dm01sw-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS  ] Load firmware 2.2.7_2 onto dm01sw-iba01
[SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
[SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-iba01
[INFO     ] Set SMPriority to 5 on dm01sw-iba01
[INFO     ] Rebooting dm01sw-iba01. Wait for 4 minutes before continuing
[SUCCESS  ] Reboot dm01sw-iba01
[SUCCESS  ] SUCCESS
[INFO     ] Starting post-update validation on dm01sw-iba01
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
[SUCCESS  ] NTP daemon is running on dm01sw-iba01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 04:13:06
[INFO     ] /conf/configvalid is 1
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
[SUCCESS  ] Execute plugin check for Post Patch on dm01sw-iba01
[INFO     ] Finished post-update validation on dm01sw-iba01
[SUCCESS  ] Post-update validation on dm01sw-iba01

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
     To downgrade to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
     – Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
[INFO     ] Starting pre-update validation on dm01sw-iba01
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
[SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 28M
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
[SUCCESS  ] NTP daemon is running on dm01sw-iba01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 04:13:35
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
[SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
[INFO     ] Finished pre-update validation on dm01sw-iba01
[SUCCESS  ] Pre-update validation on dm01sw-iba01
[INFO     ] Package will be downloaded at firmware update time via scp
[SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
[SUCCESS  ] Execute plugin check for Patching on dm01sw-iba01
[INFO     ] Starting upgrade on dm01sw-iba01 to 2.2.11_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO     ] Rebooting dm01sw-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS  ] Load firmware 2.2.11_2 onto dm01sw-iba01
[SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
[SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-iba01
[INFO     ] Set SMPriority to 5 on dm01sw-iba01
[INFO     ] Rebooting dm01sw-iba01. Wait for 4 minutes before continuing
[SUCCESS  ] Reboot dm01sw-iba01
[SUCCESS  ] SUCCESS
[INFO     ] Starting post-update validation on dm01sw-iba01
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
[SUCCESS  ] NTP daemon is running on dm01sw-iba01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 04:35:00
[INFO     ] /conf/configvalid is 1
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
[SUCCESS  ] Execute plugin check for Post Patch on dm01sw-iba01
[INFO     ] Finished post-update validation on dm01sw-iba01
[SUCCESS  ] Post-update validation on dm01sw-iba01
[SUCCESS  ] Update InfiniBand switch dm01sw-iba01 to 2.2.11_2
[INFO     ] InfiniBand Switches ( dm01sw-ibb01 dm01sw-iba01 ) updated to 2.2.11_2
[SUCCESS  ] Overall status

 —– InfiniBand switch update process ended 2019-02-10 04:35:25 -0600 —–
2019-02-10 04:35:25 -0600 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.2.11-2.

• Verify that all the IB Switches are upgraded to latest version.

[root@dm01db01 ~]# ssh dm01sw-ibb01 version
SUN DCS 36p version: 2.2.11-2
Build time: Aug 27 2018 11:18:39
SP board info:
Manufacturing Date: 2014.05.19
Serial Number: “NCDFxxxxx”
Hardware Revision: 0x0107
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010
[root@dm01db01 ~]#

[root@dm01db01 ~]# ssh dm01sw-iba01 version
SUN DCS 36p version: 2.2.11-2
Build time: Aug 27 2018 11:18:39
SP board info:
Manufacturing Date: 2014.05.20
Serial Number: “NCDFxxxxx”
Hardware Revision: 0x0107
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

 

Conclusion
In this article we have demonstrated how to patch Exadata IB Switches using patchmgr utility. Patching an Exadata IB switch is very straight forward and can be done in rolling fashion without any downtime.

1

---

Uncategorized

Change root User Password on Exadata Infiniband Switch

By Netsoftmate on

February 18, 2019

- Uncategorized

I was working on changing password for the administrative user accounts on all Exadata Components. I encountered a strange issue while changing the root password on Infiniband Switch. We were unable to change the root password on IB Siwtch using command line method. We used couple different command line methods to change the root password on IB switches but all of them failed. This could be a BUG, firmware issue or something else.

In this article we demonstrate how to change the root password on an Exadata infiniband switch using Browser User Interface.

Issue 1: Using passwd command

Tried to change the root user password using passwd command using dcli. This method assumes you are have ssh equivalence setup from compute node 1. As you can see the command failed saying to use the ILOM shell. In the past I have used the same command successfully to change the root password on IB Switches.

[root@dm01db01 ~]#  dcli -g ibswitch_group -l root “echo welcome1 | passwd –stdin root”
dm01sw-ibb01: This command should not be used for ILOM users.
dm01sw-ibb01: Please use ILOM shell to handle password for this user.
dm01sw-ibb01: Example:
dm01sw-ibb01: -> set /SP/users/root password
dm01sw-ibb01:
dm01sw-iba01: This command should not be used for ILOM users.
dm01sw-iba01: Please use ILOM shell to handle password for this user.
dm01sw-iba01: Example:
dm01sw-iba01: -> set /SP/users/root password
dm01sw-iba01:

So I decided to login to the IB switch directly and use the passwd command instead of running from dcli. The passwd command fail again with the same error.

[root@dm01sw-iba01 ~]# ssh dm01sw-ibb01
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use “help” at linux prompt.

[root@dm01sw-ibb01 ~]# hostname
dm01sw-ibb01

[root@dm01sw-iba01 ~]# passwd root
This command should not be used for ILOM users.
Please use ILOM shell to handle password for this user.
Example:
   -> set /SP/users/root password




Issue 2: Using ILOM Shell

As the passwd command failed asking to use the ILOM shell, I login to the IB switch as ilom-admin and executed the change password command. What I see is, the password change command failed at ILOM prompt as well.

[root@dm01sw-iba01 ~]# su – ilom-admin

Oracle(R) Integrated Lights Out Manager
Version 2.2.7-1 ILOM 3.2.6 r118629
Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
Warning: HTTPS certificate is set to factory default.
 
Hostname: dm01sw-iba01

-> set /SP/users/root welcome1
set: Invalid command syntax
Usage: set [-script] [target] <property>=<value> [<property>=<value>…]


 
Solution: Using Browser User Interface

I have decided to use the BUI to change the password.

Steps:

• Open a Browser and enter the IB Switch hostname or IP address

https://dm01sw-ibb01.netsoftmate.com

• Accept the security warning and proceed to connect to the IB Switch

• Enter the username and password to connect to the IB Switch

• This show the summary page

• On the left Pan, expand ILOM administration and select User Management

• Click on  User Accounts, Select root user and click on edit button

• Enter the new password and confirm and Finally click on the Save button to change the password.

• To Verify the new password, open a Putty session and ssh to IB Switch using new password.

[root@dm01db01 ~]# ssh dm01sw-ibb01
Password:
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use “help” at linux prompt.

[root@dm01sw-ibb01 ~]# hostname
dm01sw-ibb01


Conclusion

In this article we have learned how to change the root password on Infiniband Switch using Browser User Interface when the command line option doesn’t work.

1

---

How To Clear Hardware Fault on Exadata Infiniband Switch Manually

By Netsoftmate on

February 15, 2019

- Uncategorized

Introduction

We had a FAN failure on Exadata Infiniband Switch (FAN2). Scheduled the faulty hardware replacement with Oracle. The Oracle Feild Engineer came to the Customer Data Center and replaced the faulty FAN on Infiniband Switch. The FAN replacement was successful however the fault was not cleared automatically. We can still see the FAN was marked faulted from Infiniband BUI and CLI.

From Infiniband Browser User Interface

In this article we will demonstrate how to clear the fault on Infiniband Switch after hardware replacement.

• Login to the Infiniband switch using Putty as root user and check the Infiniband health. From the output below we can see the FANs are all good.

[root@dm01sw-iba01 ~]# env_test
Environment test started:
Starting Environment Daemon test:
Environment daemon running
Environment Daemon test returned OK
Starting Voltage test:
Voltage ECB OK
Measured 3.3V Main = 3.28 V
Measured 3.3V Standby = 3.39 V
Measured 12V = 11.97 V
Measured 5V = 5.02 V
Measured VBAT = 3.14 V
Measured 2.5V = 2.49 V
Measured 1.8V = 1.79 V
Measured I4 1.2V = 1.22 V
Voltage test returned OK
Starting PSU test:
PSU 0 present OK
PSU 1 present OK
PSU test returned OK
Starting Temperature test:
Back temperature 40
Front temperature 41
SP temperature 57
Switch temperature 55, maxtemperature 59
Temperature test returned OK
Starting FAN test:
Fan 0 not present
Fan 1 running at rpm 17004
Fan 2 running at rpm 15696
Fan 3 running at rpm 17004
Fan 4 not present
FAN test returned OK
Starting Connector test:
Connector test returned OK
Starting Onboard ibdevice test:
Switch OK
All Internal ibdevices OK
Onboard ibdevice test returned OK
Starting SSD test:
SSD test returned OK
Starting Auto-link-disable test:
Auto-link-disable test returned OK
Environment test PASSED

• Check the FAN Speed. FAN looks good.

[root@dm01sw-iba01 ~]# getfanspeed
Fan 0 not present
Fan 1 running at rpm 17004
Fan 2 running at rpm 15478
Fan 3 running at rpm 17004
Fan 4 not present

• Switch to the ilom-admin user

[root@dm01sw-iba01 ~]# su – ilom-admin

Oracle(R) Integrated Lights Out Manager

Version 2.2.9-3 ILOM 3.2.11 r124039

Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: dm01sw-iba01.netsoftmate.com

->

• Now check the fault table for any faulty components. Now we can see the FAN2 is Faulted though the FAN was replaced with a new FAN.

-> show / -a -l 4 -o table fault_state
Target                                  | Property                                     | Value
—————————————-+———————————————-+——————————————————————–
/SYS                                    | fault_state                                  | OK
/SYS/MB                                 | fault_state                                  | OK
/SYS/PSU0                               | fault_state                                  | OK
/SYS/PSU1                               | fault_state                                  | OK
/SYS/FAN1                               | fault_state                                  | OK
/SYS/FAN2                               | fault_state                                  | Faulted /SYS/FAN3                               | fault_state                                  | OK

->

• You can also execute the below command to identify the fault

-> show -d targets /SP/faultmgmt

 /SP/faultmgmt
    Targets:
        shell
        0 (/SYS/FAN2)

• Clear the Fault as show below

-> set /SYS/FAN2 clear_fault_action=true
Are you sure you want to clear /SYS/FAN2 (y/n)? y
Set ‘clear_fault_action’ to ‘true’

• Verify the fault is cleared

-> show / -a -l 4 -o table fault_state
Target                                  | Property                                     | Value
—————————————-+———————————————-+——————————————————————–
/SYS                                    | fault_state                                  | OK
/SYS/MB                                 | fault_state                                  | OK
/SYS/PSU0                               | fault_state                                  | OK
/SYS/PSU1                               | fault_state                                  | OK
/SYS/FAN1                               | fault_state                                  | OK
/SYS/FAN2                               | fault_state                                  | OK
/SYS/FAN3                               | fault_state                                  | OK

-> show -d targets /SP/faultmgmt

 /SP/faultmgmt
    Targets:
        shell

• Verify from the Infiniband Band BUI

Conclusion

In this article we have learned how to identify the fault and clear it manually on an Exadata Infiniband Switch. The ILOM commands comes handy for clearing the fault. You can also clear the fault using the Browser User Interface (BUI).

2

---

Patching Exadata Infiniband Switch

By Netsoftmate on

February 12, 2017

- Uncategorized

Overview

• The Exadata network grid consists of multiple Sun QDR InfiniBand switches.
• IB Switches are used for the storage network as well as the Oracle RAC interconnect.
• Exadata compute nodes and storage cells are configured with dual-port InfiniBand ports and connect to each of the two leaf switches.
• You can access IB Switches using command line and Web ILOM
• IB Switches run Linux operating system.

In this article I will demonstrate how to patch or upgrade Oracle Exadata IB Switches.

About Infiniband Switch Patching

• Starting with release 11.2.3.3.0, the patchmgr utility is used to upgrade and downgrade the InfiniBand switches.
• IB Switch patch is delievered with Exadata storage patch.
• IB Switch patches are released semi annually to annually.
• IB Switch can be patched in Rolling and Non-Rolling fashion.

Environment

• Exadata Full Rack X2-2
• 8 Compute nodes, 14 Storage cells and 3 IB Switches
• Current IB Switch Version 2.1.5-1

Infiniband Switch Upgrade Steps

• Identify the number of switches in clusters.

[root@dm01dbadm01 ~]# ibswitches
Switch  : 0x002128469b8aa0a0 ports 36 “SUN DCS 36P QDR dm01sw-ib1 10.209.41.246” enhanced port 0 lid 5 lmc 0
Switch  : 0x002128469b97a0a0 ports 36 “SUN DCS 36P QDR dm01sw-ib2 10.209.41.247” enhanced port 0 lid 4 lmc 0
Switch  : 0x002128469b82a0a0 ports 36 “SUN DCS 36P QDR dm01sw-ib3 10.209.41.248” enhanced port 0 lid 3 lmc 0

• Identify the current IB switch software version on all the Switches

[root@dm01dbadm01 patch_12.1.2.2.0.150917]# ssh dm01sw-ib1 version
SUN DCS 36p version: 2.1.5-1
Build time: Oct  6 2014 10:35:15
SP board info:
Manufacturing Date: 2011.01.12
Serial Number: “NCD680421”
Hardware Revision: 0x0006
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

• Log in to Exadata Compute node 1 as root user and navigate the Exadata Storage Software staging area

[root@dm01dbadm01 ESS_121220]# cd /u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917/
[root@dm01dbadm01 patch_12.1.2.2.0.150917]# pwd
/u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917

• Create a file named ibswitches.lst and enter IB switch names one per line as follows:

[root@dm01dbadm01 patch_12.1.2.2.0.150917]# vi ibswitches.lst
dm01sw-ib1
dm01sw-ib2
dm01sw-ib3

[root@dm01dbadm01 patch_12.1.2.2.0.150917]# cat ibswitches.lst
dm01sw-ib1
dm01sw-ib2
dm01sw-ib3

• Execute the following to perform the IB Switch precheck

[root@dm01dbadm01 patch_12.1.2.2.0.150917]# ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck -force

2015-11-04 11:31:44 -0600 1 of 1 :SUCCESS: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
 —– InfiniBand switch update process started 2015-11-04 11:31:44 -0600 —–
[NOTE     ] Log file at /u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917/upgradeIBSwitch.log

[INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ib1 dm01sw-ib2 dm01sw-ib3 )

[SUCCESS  ] Verifying Network connectivity to dm01sw-ib1
[SUCCESS  ] Verifying Network connectivity to dm01sw-ib2
[SUCCESS  ] Verifying Network connectivity to dm01sw-ib3
[FAIL     ] Validating verify-topology output
[INFO     ] Master Subnet Manager is set to “dm01sw-ib3” in all Switches

[INFO     ] ———- Starting with IBSwitch dm01sw-ib1
[WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
     To rollback to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
     – Run patchmgr command to initiate rollback.
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib1, found 137M
[SUCCESS  ] Verifying that / has 80M in dm01sw-ib1, found 197M
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib1
[SUCCESS  ] Verifying that dm01sw-ib1 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:41:20
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib1
[SUCCESS  ] Pre-update validation on dm01sw-ib1

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib2
[WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
     To rollback to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
     – Run patchmgr command to initiate rollback.
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib2, found 249M
[SUCCESS  ] Verifying that / has 80M in dm01sw-ib2, found 199M
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib2
[SUCCESS  ] Verifying that dm01sw-ib2 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:41:33
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib2
[SUCCESS  ] Pre-update validation on dm01sw-ib2

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib3
[WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
     To rollback to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
     – Run patchmgr command to initiate rollback.
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib3, found 248M
[SUCCESS  ] Verifying that / has 80M in dm01sw-ib3, found 196M
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib3
[SUCCESS  ] Verifying that dm01sw-ib3 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:41:42
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib3
[SUCCESS  ] Pre-update validation on dm01sw-ib3
[SUCCESS  ] Overall status
 —– InfiniBand switch update process ended 2015-11-04 11:41:42 -0600 —–
2015-11-04 11:41:42 -0600 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).

• Upgrade the IB Switches using the following command:

[root@dm01dbadm01 patch_12.1.2.2.0.150917]# ./patchmgr -ibswitches ibswitches.lst -upgrade -force

2015-11-04 11:42:51 -0600 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.6-2. Expect up to 15 minutes for each switch
 —– InfiniBand switch update process started 2015-11-04 11:42:51 -0600 —–
[NOTE     ] Log file at /u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917/upgradeIBSwitch.log

[INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ib1 dm01sw-ib2 dm01sw-ib3 )

[SUCCESS  ] Verifying Network connectivity to dm01sw-ib1
[SUCCESS  ] Verifying Network connectivity to dm01sw-ib2
[SUCCESS  ] Verifying Network connectivity to dm01sw-ib3
[FAIL     ] Validating verify-topology output
[INFO     ] Proceeding with upgrade of InfiniBand switches to version 2.1.6_2
[INFO     ] Master Subnet Manager is set to “aibdalab1sw-ib3” in all Switches

[INFO     ] ———- Starting with IBSwitch dm01sw-ib1
[WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
     To rollback to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
     – Run patchmgr command to initiate rollback.
[SUCCESS  ] Copy firmware packages to dm01sw-ib1
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib1, found 137M
[SUCCESS  ] Verifying that / has 80M in dm01sw-ib1, found 197M
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib1
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib1
[SUCCESS  ] Verifying that dm01sw-ib1 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:44:09
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib1
[SUCCESS  ] Pre-update validation on dm01sw-ib1
[SUCCESS  ] Disable Subnet Manager on dm01sw-ib1
[INFO     ] Starting upgrade on dm01sw-ib1 to 2.1.6_2. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS  ] Execute plugin check for Patching on dm01sw-ib1
[SUCCESS  ] Load firmware 2.1.6_2 onto dm01sw-ib1
[SUCCESS  ] Disable Subnet Manager on dm01sw-ib1
[SUCCESS  ] Set SMPriority to 8 on dm01sw-ib1
[INFO     ] Rebooting dm01sw-ib1. Wait for 240 secs before continuing
[SUCCESS  ] Reboot dm01sw-ib1
[SUCCESS  ] SUCCESS
[INFO     ] Starting post-update validation on dm01sw-ib1
[SUCCESS  ] Inifiniband switch dm01sw-ib1 is at target patching level
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib1
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib1
[SUCCESS  ] Verifying that dm01sw-ib1 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:01:21
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ib1
[SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ib1
[SUCCESS  ] Post-check validation on IBSwitch dm01sw-ib1
[SUCCESS  ] Update switch dm01sw-ib1 to 2.1.6_2

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib2
[WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
     To rollback to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
     – Run patchmgr command to initiate rollback.
[SUCCESS  ] Copy firmware packages to dm01sw-ib2
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib2, found 138M
[SUCCESS  ] Verifying that / has 80M in dm01sw-ib2, found 199M
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib2
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib2
[SUCCESS  ] Verifying that dm01sw-ib2 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:06:23
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib2
[SUCCESS  ] Pre-update validation on dm01sw-ib2
[SUCCESS  ] Disable Subnet Manager on dm01sw-ib2
[INFO     ] Starting upgrade on dm01sw-ib2 to 2.1.6_2. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS  ] Execute plugin check for Patching on dm01sw-ib2
[SUCCESS  ] Load firmware 2.1.6_2 onto dm01sw-ib2
[SUCCESS  ] Disable Subnet Manager on dm01sw-ib2
[SUCCESS  ] Set SMPriority to 2 on dm01sw-ib2
[INFO     ] Rebooting dm01sw-ib2. Wait for 240 secs before continuing
[SUCCESS  ] Reboot dm01sw-ib2
[SUCCESS  ] SUCCESS
[INFO     ] Starting post-update validation on dm01sw-ib2
[SUCCESS  ] Inifiniband switch dm01sw-ib2 is at target patching level
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib2
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib2
[SUCCESS  ] Verifying that dm01sw-ib2 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:23:30
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ib2
[SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ib2
[SUCCESS  ] Post-check validation on IBSwitch dm01sw-ib2
[SUCCESS  ] Update switch dm01sw-ib2 to 2.1.6_2

[INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib3
[WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
     To rollback to other versions:
     – Manually download the InfiniBand switch firmware package to the patch directory
     – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
     – Run patchmgr command to initiate rollback.
[SUCCESS  ] Copy firmware packages to dm01sw-ib3
[SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib3, found 137M
[SUCCESS  ] Verifying that / has 80M in dm01sw-ib3, found 196M
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib3
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib3
[SUCCESS  ] Verifying that dm01sw-ib3 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:28:26
[SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib3
[SUCCESS  ] Pre-update validation on dm01sw-ib3
[SUCCESS  ] Disable Subnet Manager on dm01sw-ib3
[INFO     ] Starting upgrade on dm01sw-ib3 to 2.1.6_2. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[SUCCESS  ] Execute plugin check for Patching on dm01sw-ib3
[SUCCESS  ] Load firmware 2.1.6_2 onto dm01sw-ib3
[SUCCESS  ] Disable Subnet Manager on dm01sw-ib3
[SUCCESS  ] Set SMPriority to 2 on dm01sw-ib3
[INFO     ] Rebooting dm01sw-ib3. Wait for 240 secs before continuing
[SUCCESS  ] Reboot dm01sw-ib3
[SUCCESS  ] SUCCESS
[INFO     ] Starting post-update validation on dm01sw-ib3
[SUCCESS  ] Inifiniband switch dm01sw-ib3 is at target patching level
[SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib3
[SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib3
[SUCCESS  ] Verifying that dm01sw-ib3 has at least 1 NTP Server, found 1
[SUCCESS  ] opensm.conf passed all validations
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:46:04
[SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ib3
[SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ib3
[SUCCESS  ] Post-check validation on IBSwitch dm01sw-ib3
[SUCCESS  ] Update switch dm01sw-ib3 to 2.1.6_2
[INFO     ] InfiniBand Switches ( dm01sw-ib1 dm01sw-ib2 dm01sw-ib3 ) updated to 2.1.6_2
[SUCCESS  ] Overall status
 —– InfiniBand switch update process ended 2015-11-04 12:50:22 -0600 —–
2015-11-04 12:50:22 -0600 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.6-2.

• Verify that all the IB Switches are upgraded to latest version.

[root@dm01dbadm01 ~]# ssh dm01sw-ib1 version
SUN DCS 36p version: 2.1.6-2
Build time: Dec  8 2014 10:36:50
SP board info:
Manufacturing Date: 2011.01.12
Serial Number: “NCD680421”
Hardware Revision: 0x0006
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010
[root@dm01dbadm01 ~]#
 
Conclusion
In this article we have demonstrated how to patch Exadata IB Switches using patchmgr utility. Patching an Exadata IB switch is very straight forward and can be done in rolling and non-rolling fashion.

0

---