Netsoftmate

Tag: Exadata

  • Step by Step Exadata Infiniband Switch Patching 18c

    Overview

    • The Exadata network grid consists of multiple Sun QDR InfiniBand switches.
    • IB Switches are used for the storage network as well as the Oracle RAC interconnect.
    • Exadata compute nodes and storage cells are configured with dual-port InfiniBand ports and connect to each of the two leaf switches.
    • You can access IB Switches using command line and Web ILOM
    • IB Switches run Linux operating system.

    In this article I will demonstrate how to patch or upgrade Oracle Exadata IB Switches.

    About Infiniband Switch Patching

    • Starting with release 11.2.3.3.0, the patchmgr utility is used to upgrade and downgrade the InfiniBand switches.
    • IB Switch patch is delievered with Exadata storage patch.
    • IB Switch patches are released semi annually to annually.
    • IB Switch can be patched in Rolling fashion only.

    Environment
    • Exadata Half Rack X4-2
    • 4 Compute nodes, 7 Storage cells and 2 IB Switches
    • Current IB Switch Version 2.2.7-1

    Step by Step Infiniband Switch Patching
     

    • Identify the number of switches in clusters.
    [root@dm01dbadm01 ~]# ibswitches
    Switch  : 0x002128469b8aa0a0 ports 36 “SUN DCS 36P QDR dm01sw-iba01 10.209.41.246” enhanced port 0 lid 5 lmc 0
    Switch  : 0x002128469b97a0a0 ports 36 “SUN DCS 36P QDR dm01sw-ibb01 10.209.41.247” enhanced port 0 lid 4 lmc 0


    • Identify the current IB switch software version on all the Switches
    [root@dm01db01 patch_18.1.12.0.0.190111]# ssh dm01sw-iba01 version
    SUN DCS 36p version: 2.2.7-1
    Build time: Aug  4 2017 12:20:53
    SP board info:
    Manufacturing Date: 2014.05.20
    Serial Number: “NCDFxxxxx”
    Hardware Revision: 0x0107
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010


    • Log in to Exadata Compute node 1 as root user and navigate the Exadata Storage Software staging area
    [root@dm01dbadm01 ESS_121220]# cd /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111
     
    [root@dm01dbadm01 patch_18.1.12.0.0.190111]# pwd
    /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111


    • Create a file named ibswitches.lst and enter IB switch names one per line as follows:
    [root@dm01dbadm01 patch_18.1.12.0.0.190111]# vi ~/ibswitch_group
    dm01sw-ibb01
    dm01sw-iba01

    [root@dm01dbadm01 patch_18.1.12.0.0.190111]# cat ~/ibswitch_group
    dm01sw-ibb01
    dm01sw-iba01



    • Execute the following to perform the IB Switch precheck
    [root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -ibswitches ~/ibswitch_group -upgrade -ibswitch_precheck

    2019-02-10 03:02:46 -0600 1 of 1 :Working: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
     —– InfiniBand switch update process started 2019-02-10 03:02:47 -0600 —–
    [NOTE     ] Log file at /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/upgradeIBSwitch.log

    [INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ibb01 dm01sw-iba01 )
    [SUCCESS  ] Verifying Network connectivity to dm01sw-ibb01
    [SUCCESS  ] Verifying Network connectivity to dm01sw-iba01
    [SUCCESS  ] Validating verify-topology output
    [INFO     ] Master Subnet Manager is set to “dm01sw-ibb01” in all Switches
    [INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-ibb01 first to 2.2.7-2

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
    [INFO     ] Starting pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 26M
    [SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:03:03
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
    [INFO     ] Finished pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Pre-update validation on dm01sw-ibb01

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
    [INFO     ] Starting pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 26M
    [SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:03:34
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
    [INFO     ] Finished pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Prereq check on dm01sw-ibb01
    [INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-iba01 first to 2.2.7-2

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
    [INFO     ] Starting pre-update validation on dm01sw-iba01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 26M
    [SUCCESS  ] NTP daemon is running on dm01sw-iba01.
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:04:06
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
    [INFO     ] Finished pre-update validation on dm01sw-iba01
    [SUCCESS  ] Pre-update validation on dm01sw-iba01

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
    [INFO     ] Starting pre-update validation on dm01sw-iba01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 26M
    [SUCCESS  ] NTP daemon is running on dm01sw-iba01.
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:04:36
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
    [INFO     ] Finished pre-update validation on dm01sw-iba01
    [SUCCESS  ] Pre-update validation on dm01sw-iba01
    [SUCCESS  ] Prereq check on dm01sw-iba01
    [SUCCESS  ] Overall status

     —– InfiniBand switch update process ended 2019-02-10 03:05:00 -0600 —–
    2019-02-10 03:05:00 -0600 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).



    • Upgrade the IB Switches using the following command:
    [root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -ibswitches ~/ibswitch_group -upgrade

    2019-02-10 03:07:26 -0600 1 of 1 :Working: DO: Initiate upgrade of InfiniBand switches to 2.2.11-2. Expect up to 40 minutes for each switch
                                                      
     —– InfiniBand switch update process started 2019-02-10 03:07:27 -0600 —–
    [NOTE     ] Log file at /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/upgradeIBSwitch.log

    [INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ibb01 dm01sw-iba01 )
    [SUCCESS  ] Verifying Network connectivity to dm01sw-ibb01
    [SUCCESS  ] Verifying Network connectivity to dm01sw-iba01
    [SUCCESS  ] Validating verify-topology output
    [INFO     ] Proceeding with upgrade of InfiniBand switches to version 2.2.11_2
    [INFO     ] Master Subnet Manager is set to “dm01sw-ibb01” in all Switches
    [INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-ibb01 first to 2.2.7-2

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
    [INFO     ] Starting pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 26M
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
    [SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:07:43
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
    [INFO     ] Finished pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Pre-update validation on dm01sw-ibb01
    [INFO     ] Package will be downloaded at firmware update time via scp
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
    [SUCCESS  ] Execute plugin check for Patching on dm01sw-ibb01
    [INFO     ] Starting upgrade on dm01sw-ibb01 to 2.2.7_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [INFO     ] Rebooting dm01sw-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
    [SUCCESS  ] Load firmware 2.2.7_2 onto dm01sw-ibb01
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
    [SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-ibb01
    [INFO     ] Set SMPriority to 5 on dm01sw-ibb01
    [INFO     ] Rebooting dm01sw-ibb01. Wait for 4 minutes before continuing
    [SUCCESS  ] Reboot dm01sw-ibb01
    [SUCCESS  ] SUCCESS
    [INFO     ] Starting post-update validation on dm01sw-ibb01
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
    [SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:29:11
    [INFO     ] /conf/configvalid is 1
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
    [SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ibb01
    [INFO     ] Finished post-update validation on dm01sw-ibb01
    [SUCCESS  ] Post-update validation on dm01sw-ibb01

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ibb01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-ibb01.
    [INFO     ] Starting pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ibb01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-ibb01, found 28M
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
    [SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:29:39
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-ibb01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ibb01
    [INFO     ] Finished pre-update validation on dm01sw-ibb01
    [SUCCESS  ] Pre-update validation on dm01sw-ibb01
    [INFO     ] Package will be downloaded at firmware update time via scp
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
    [SUCCESS  ] Execute plugin check for Patching on dm01sw-ibb01
    [INFO     ] Starting upgrade on dm01sw-ibb01 to 2.2.11_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [INFO     ] Rebooting dm01sw-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
    [SUCCESS  ] Load firmware 2.2.11_2 onto dm01sw-ibb01
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ibb01
    [SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-ibb01
    [INFO     ] Set SMPriority to 5 on dm01sw-ibb01
    [INFO     ] Rebooting dm01sw-ibb01. Wait for 4 minutes before continuing
    [SUCCESS  ] Reboot dm01sw-ibb01
    [SUCCESS  ] SUCCESS
    [INFO     ] Starting post-update validation on dm01sw-ibb01
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ibb01
    [SUCCESS  ] NTP daemon is running on dm01sw-ibb01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:51:03
    [INFO     ] /conf/configvalid is 1
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ibb01
    [SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ibb01
    [INFO     ] Finished post-update validation on dm01sw-ibb01
    [SUCCESS  ] Post-update validation on dm01sw-ibb01
    [SUCCESS  ] Update InfiniBand switch dm01sw-ibb01 to 2.2.11_2
    [INFO     ] Upgrade to 2.2.11_2 requires that the InfiniBand switch be at 2.2.7-2. Upgrading dm01sw-iba01 first to 2.2.7-2
    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
    [INFO     ] Starting pre-update validation on dm01sw-iba01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 26M
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
    [SUCCESS  ] NTP daemon is running on dm01sw-iba01.
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 03:51:38
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
    [INFO     ] Finished pre-update validation on dm01sw-iba01
    [SUCCESS  ] Pre-update validation on dm01sw-iba01
    [INFO     ] Package will be downloaded at firmware update time via scp
    [SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
    [SUCCESS  ] Execute plugin check for Patching on dm01sw-iba01
    [INFO     ] Starting upgrade on dm01sw-iba01 to 2.2.7_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [INFO     ] Rebooting dm01sw-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
    [SUCCESS  ] Load firmware 2.2.7_2 onto dm01sw-iba01
    [SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
    [SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-iba01
    [INFO     ] Set SMPriority to 5 on dm01sw-iba01
    [INFO     ] Rebooting dm01sw-iba01. Wait for 4 minutes before continuing
    [SUCCESS  ] Reboot dm01sw-iba01
    [SUCCESS  ] SUCCESS
    [INFO     ] Starting post-update validation on dm01sw-iba01
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
    [SUCCESS  ] NTP daemon is running on dm01sw-iba01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 04:13:06
    [INFO     ] /conf/configvalid is 1
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
    [SUCCESS  ] Execute plugin check for Post Patch on dm01sw-iba01
    [INFO     ] Finished post-update validation on dm01sw-iba01
    [SUCCESS  ] Post-update validation on dm01sw-iba01

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-iba01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.9-3 with the current package.
         To downgrade to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION” to the appropriate version
         – Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host dm01db01.netsoftmate.com from the InfiniBand Switch dm01sw-iba01.
    [INFO     ] Starting pre-update validation on dm01sw-iba01
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-iba01, found 246M
    [SUCCESS  ] Verifying that / has 20M in dm01sw-iba01, found 28M
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
    [SUCCESS  ] NTP daemon is running on dm01sw-iba01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 04:13:35
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
    [SUCCESS  ] Verifying that the patchmgr host dm01db01.netsoftmate.com is recognized on the InfiniBand Switch dm01sw-iba01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-iba01
    [INFO     ] Finished pre-update validation on dm01sw-iba01
    [SUCCESS  ] Pre-update validation on dm01sw-iba01
    [INFO     ] Package will be downloaded at firmware update time via scp
    [SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
    [SUCCESS  ] Execute plugin check for Patching on dm01sw-iba01
    [INFO     ] Starting upgrade on dm01sw-iba01 to 2.2.11_2. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [INFO     ] Rebooting dm01sw-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
    [SUCCESS  ] Load firmware 2.2.11_2 onto dm01sw-iba01
    [SUCCESS  ] Disable Subnet Manager on dm01sw-iba01
    [SUCCESS  ] Verify that /conf/configvalid is set to 1 on dm01sw-iba01
    [INFO     ] Set SMPriority to 5 on dm01sw-iba01
    [INFO     ] Rebooting dm01sw-iba01. Wait for 4 minutes before continuing
    [SUCCESS  ] Reboot dm01sw-iba01
    [SUCCESS  ] SUCCESS
    [INFO     ] Starting post-update validation on dm01sw-iba01
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-iba01
    [SUCCESS  ] NTP daemon is running on dm01sw-iba01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2019-02-10 Time:(HH:MM:SS) 04:35:00
    [INFO     ] /conf/configvalid is 1
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-iba01
    [SUCCESS  ] Execute plugin check for Post Patch on dm01sw-iba01
    [INFO     ] Finished post-update validation on dm01sw-iba01
    [SUCCESS  ] Post-update validation on dm01sw-iba01
    [SUCCESS  ] Update InfiniBand switch dm01sw-iba01 to 2.2.11_2
    [INFO     ] InfiniBand Switches ( dm01sw-ibb01 dm01sw-iba01 ) updated to 2.2.11_2
    [SUCCESS  ] Overall status

     —– InfiniBand switch update process ended 2019-02-10 04:35:25 -0600 —–
    2019-02-10 04:35:25 -0600 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.2.11-2.



    • Verify that all the IB Switches are upgraded to latest version.
    [root@dm01db01 ~]# ssh dm01sw-ibb01 version
    SUN DCS 36p version: 2.2.11-2
    Build time: Aug 27 2018 11:18:39
    SP board info:
    Manufacturing Date: 2014.05.19
    Serial Number: “NCDFxxxxx”
    Hardware Revision: 0x0107
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010
    [root@dm01db01 ~]#

    [root@dm01db01 ~]# ssh dm01sw-iba01 version
    SUN DCS 36p version: 2.2.11-2
    Build time: Aug 27 2018 11:18:39
    SP board info:
    Manufacturing Date: 2014.05.20
    Serial Number: “NCDFxxxxx”
    Hardware Revision: 0x0107
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010

     

    Conclusion
    In this article we have demonstrated how to patch Exadata IB Switches using patchmgr utility. Patching an Exadata IB switch is very straight forward and can be done in rolling fashion without any downtime.

  • Step by Step Exadata Storage Server Patching 18c

    The patchmgr utility can be used for upgrading, rollback and backup Exadata Storage cells. patchmgr utility can be used for upgrading Storage cells in a rolling or non-rolling fashion. Non-Rolling is default. Storage server patches apply operating system, firmware, and driver updates.

    Launch patchmgr from the compute node that is node 1 that has user equivalence setup to all the storage cells.

    In this article I will demonstrate how to perform upgrade Exadata Storage cells using patchmgr utility.

    MOS Notes
    Read the following MOS notes carefully.

    • Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)
    • Exadata 18.1.12.0.0 release and patch (29194095) (Doc ID 2492012.1)   
    • Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)   


    Software Download

    • Download the following patches required for Upgrading Storage cells.
    • Patch 29194095 – Storage server (18.1.12.0.0.190111) and InfiniBand switch software (2.2.11-2)


    Current Environment

    • Exadata X4-2 Half Rack (4 Compute nodes, 7 Storage Cells and 2 IB Switches) running ESS version 12.2.1.1.6


    Current Image version

    • Execute the “imageinfo” command on one of the Compute nodes to identify the current Exadata Image version

    [root@dm01cel01 ~]# imageinfo

    Kernel version: 4.1.12-94.7.8.el6uek.x86_64 #2 SMP Thu Jan 11 20:41:01 PST 2018 x86_64
    Cell version: OSS_12.2.1.1.6_LINUX.X64_180125.1
    Cell rpm version: cell-12.2.1.1.6_LINUX.X64_180125.1-1.x86_64

    Active image version: 12.2.1.1.6.180125.1
    Active image kernel version: 4.1.12-94.7.8.el6uek
    Active image activated: 2018-05-08 00:42:57 -0500
    Active image status: success
    Active system partition on device: /dev/md6
    Active software partition on device: /dev/md8

    Cell boot usb partition: /dev/sdac1
    Cell boot usb version: 12.2.1.1.6.180125.1

    Inactive image version: 12.1.2.3.6.170713
    Inactive image activated: 2017-10-03 00:57:25 -0500
    Inactive image status: success
    Inactive system partition on device: /dev/md5
    Inactive software partition on device: /dev/md7

    Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
    Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
    Inactive kernel version for the rollback: 2.6.39-400.297.1.el6uek.x86_64
    Rollback to the inactive partitions: Possible

    Prerequisites

    • Install and configure VNC Server on Exadata compute node 1. It is recommended to use VNC or screen utility for patching to avoid disconnections due to network issues.


    • Enable blackout (OEM, crontab and so on)


    • Verify disk space on storage cells

    [root@dm01db01 ~]# dcli -g ~/cell_group -l root ‘df -h /’
    dm01cel01: Filesystem      Size  Used Avail Use% Mounted on
    dm01cel01: /dev/md6        9.8G  4.4G  4.9G  48% /
    dm01cel02: Filesystem      Size  Used Avail Use% Mounted on
    dm01cel02: /dev/md6        9.8G  4.5G  4.8G  49% /
    dm01cel03: Filesystem      Size  Used Avail Use% Mounted on
    dm01cel03: /dev/md6        9.8G  4.5G  4.8G  49% /
    dm01cel04: Filesystem      Size  Used Avail Use% Mounted on
    dm01cel04: /dev/md6        9.8G  4.5G  4.8G  49% /
    dm01cel05: Filesystem      Size  Used Avail Use% Mounted on
    dm01cel05: /dev/md6        9.8G  4.5G  4.8G  49% /
    dm01cel06: Filesystem      Size  Used Avail Use% Mounted on
    dm01cel06: /dev/md6        9.8G  4.6G  4.7G  50% /
    dm01cel07: Filesystem      Size  Used Avail Use% Mounted on
    dm01cel07: /dev/md6        9.8G  4.5G  4.8G  48% /

    • Run Exachk before starting the actual patching. Correct any Critical issues and Failure that can conflict with patching.


    • Verify hardware failure. Make sure there are no hardware failures before patching

    [root@dm01db01 ~]# dcli -g ~/cell_group -l root ‘cellcli -e list physicaldisk where status!=normal’
    [root@dm01db01 ~]# dcli -l root -g ~/cell_group “cellcli -e list physicaldisk where diskType=FlashDisk and status not = normal”
    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ‘dbmcli -e list physicaldisk where status!=normal’

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ‘ipmitool sunoem cli “show -d properties -level all /SYS fault_state==Faulted”‘
    [root@dm01db01 ~]# dcli -g ~/cell_group -l root ‘ipmitool sunoem cli “show -d properties -level all /SYS fault_state==Faulted”‘

    • Clear or acknowledge alerts on db and cell nodes

    [root@dm01db01 ~]# dcli -l root -g ~/cell_group “cellcli -e drop alerthistory all”
    [root@dm01db01 ~]# dcli -l root -g ~/dbs_group “dbmcli -e  drop alerthistory all”

    • Download patches and copy them to the compute node 1 under staging directory

    Patch 29194095 – Storage server software (18.1.12.0.0.190111) and InfiniBand switch software (2.2.11-2)

    • Copy the patches to compute node 1 under staging aread and unzip the patches

    [root@dm01db01 ~]# cd /u01/app/oracle/software/exa_patches
    [root@dm01db01 ~]# unzip p29194095_181000_Linux-x86-64.zip

    • Read the readme file and document the steps for storage cell patching.


    Steps to perform Storage Cell Patching

    • Open VNC Session and login as root user

    • Login as root user

    [root@dm01db01 ~]# id
    uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

    • Check SSH user equivalence

    [root@dm01db01 ~]# dcli -g cell_group -l root uptime
    dm01cel01: 01:46:18 up 194 days, 40 min,  0 users,  load average: 0.17, 0.50, 0.61
    dm01cel02: 01:46:18 up 194 days, 40 min,  0 users,  load average: 0.05, 0.29, 0.45
    dm01cel03: 01:46:18 up 194 days, 40 min,  0 users,  load average: 0.25, 0.64, 0.63
    dm01cel04: 01:46:18 up 194 days, 40 min,  0 users,  load average: 0.12, 0.44, 0.53
    dm01cel05: 01:46:18 up 194 days, 40 min,  0 users,  load average: 0.15, 0.55, 0.65
    dm01cel06: 01:46:18 up 194 days, 40 min,  0 users,  load average: 0.33, 0.48, 0.55
    dm01cel07: 01:46:18 up 194 days, 40 min,  0 users,  load average: 0.09, 0.37, 0.52

    • Adjust the disk_repair_time for Oracle ASM.

    SQL> col value for a40
    SQL> select dg.name,a.value from v$asm_diskgroup dg, v$asm_attribute a where dg.group_number=a.group_number and a.name=’disk_repair_time’;

    NAME                           VALUE
    —————————— ————————–
    DATA_DM01                      3.6H
    RECO_DM01                      3.6h

    • Shut down and stop the Oracle components on each database server using the following commands:

    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘/u01/app/11.2.0.4/grid/bin/crsctl stop cluster -all’
    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘/u01/app/11.2.0.4/grid/bin/crsctl stop crs’

    • Get the current Cell Exadata Storage software version

    [root@dm01cel01 ~]# imageinfo

    Kernel version: 4.1.12-94.7.8.el6uek.x86_64 #2 SMP Thu Jan 11 20:41:01 PST 2018 x86_64
    Cell version: OSS_12.2.1.1.6_LINUX.X64_180125.1
    Cell rpm version: cell-12.2.1.1.6_LINUX.X64_180125.1-1.x86_64

    Active image version: 12.2.1.1.6.180125.1
    Active image kernel version: 4.1.12-94.7.8.el6uek
    Active image activated: 2018-05-08 00:42:57 -0500
    Active image status: success
    Active system partition on device: /dev/md6
    Active software partition on device: /dev/md8

    Cell boot usb partition: /dev/sdac1
    Cell boot usb version: 12.2.1.1.6.180125.1

    Inactive image version: 12.1.2.3.6.170713
    Inactive image activated: 2017-10-03 00:57:25 -0500
    Inactive image status: success
    Inactive system partition on device: /dev/md5
    Inactive software partition on device: /dev/md7

    Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
    Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
    Inactive kernel version for the rollback: 2.6.39-400.297.1.el6uek.x86_64
    Rollback to the inactive partitions: Possible

    • Shut down all cell services on all cells to be updated. Use dcli command to do all cells at the same time:

    [root@dm01db01 ~]# dcli -g cell_group -l root “cellcli -e alter cell shutdown services all”
    dm01cel01:
    dm01cel01: Stopping the RS, CELLSRV, and MS services…
    dm01cel01: The SHUTDOWN of services was successful.
    dm01cel02:
    dm01cel02: Stopping the RS, CELLSRV, and MS services…
    dm01cel02: The SHUTDOWN of services was successful.
    dm01cel03:
    dm01cel03: Stopping the RS, CELLSRV, and MS services…
    dm01cel03: The SHUTDOWN of services was successful.
    dm01cel04:
    dm01cel04: Stopping the RS, CELLSRV, and MS services…
    dm01cel04: The SHUTDOWN of services was successful.
    dm01cel05:
    dm01cel05: Stopping the RS, CELLSRV, and MS services…
    dm01cel05: The SHUTDOWN of services was successful.
    dm01cel06:
    dm01cel06: Stopping the RS, CELLSRV, and MS services…
    dm01cel06: The SHUTDOWN of services was successful.
    dm01cel07:
    dm01cel07: Stopping the RS, CELLSRV, and MS services…
    dm01cel07: The SHUTDOWN of services was successful.

    • Reset the patchmgr state to a known state using the following command:

    [root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -cells ~/cell_group -reset_force

    2019-02-10 01:56:19 -0600        :Working: DO: Force Cleanup
    2019-02-10 01:56:21 -0600        :SUCCESS: DONE: Force Cleanup

    • Clean up any previous patchmgr utility runs using the following command:

    [root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -cells ~/cell_group -cleanup

    2019-02-10 01:57:39 -0600        :Working: DO: Cleanup
    2019-02-10 01:57:40 -0600        :SUCCESS: DONE: Cleanup

    • Verify that the cells meet prerequisite checks using the following command.

    [root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -cells ~/cell_group -patch_check_prereq

    2019-02-10 02:01:53 -0600        :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell …
    2019-02-10 02:01:55 -0600        :SUCCESS: DONE: Check cells have ssh equivalence for root user.
    2019-02-10 02:02:00 -0600        :Working: DO: Initialize files. Up to 1 minute …
    2019-02-10 02:02:01 -0600        :Working: DO: Setup work directory
    2019-02-10 02:02:02 -0600        :SUCCESS: DONE: Setup work directory
    2019-02-10 02:02:04 -0600        :SUCCESS: DONE: Initialize files.
    2019-02-10 02:02:04 -0600        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes …
    2019-02-10 02:02:17 -0600        :INFO   : Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
    2019-02-10 02:02:18 -0600        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
    2019-02-10 02:02:18 -0600        :Working: DO: Check space and state of cell services. Up to 20 minutes …
    2019-02-10 02:03:40 -0600        :SUCCESS: DONE: Check space and state of cell services.
    2019-02-10 02:03:40 -0600        :Working: DO: Check prerequisites on all cells. Up to 2 minutes …
    2019-02-10 02:03:49 -0600        :SUCCESS: DONE: Check prerequisites on all cells.
    2019-02-10 02:03:49 -0600        :Working: DO: Execute plugin check for Patch Check Prereq …
    2019-02-10 02:03:49 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 22909764 v1.0.
    2019-02-10 02:03:49 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:03:49 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3.
    2019-02-10 02:03:49 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:03:49 -0600        :SUCCESS: No exposure to bug 17854520 with non-rolling patching
    2019-02-10 02:03:49 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 22468216 v1.0.
    2019-02-10 02:03:49 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:03:49 -0600        :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22468216
    2019-02-10 02:03:49 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 24625612 v1.0.
    2019-02-10 02:03:49 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:03:49 -0600        :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 24625612
    2019-02-10 02:03:49 -0600        :SUCCESS: No exposure to bug  with non-rolling patching
    2019-02-10 02:03:49 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 22651315 v1.0.
    2019-02-10 02:03:49 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:03:51 -0600        :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22651315
    2019-02-10 02:03:51 -0600        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
    2019-02-10 02:03:51 -0600        :Working: DO: Check ASM deactivation outcome. Up to 1 minute …
    2019-02-10 02:04:02 -0600        :SUCCESS: DONE: Check ASM deactivation outcome.

    • If the prerequisite checks pass, then start the update process.

    [root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -cells ~/cell_group -patch
    ********************************************************************************
    NOTE Cells will reboot during the patch or rollback process.
    NOTE For non-rolling patch or rollback, ensure all ASM instances using
    NOTE the cells are shut down for the duration of the patch or rollback.
    NOTE For rolling patch or rollback, ensure all ASM instances using
    NOTE the cells are up for the duration of the patch or rollback.

    WARNING Do not interrupt the patchmgr session.
    WARNING Do not alter state of ASM instances during patch or rollback.
    WARNING Do not resize the screen. It may disturb the screen layout.
    WARNING Do not reboot cells or alter cell services during patch or rollback.
    WARNING Do not open log files in editor in write mode or try to alter them.

    NOTE All time estimates are approximate.
    ********************************************************************************

    2019-02-10 02:08:27 -0600        :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell …
    2019-02-10 02:08:28 -0600        :SUCCESS: DONE: Check cells have ssh equivalence for root user.
    2019-02-10 02:08:33 -0600        :Working: DO: Initialize files. Up to 1 minute …
    2019-02-10 02:08:34 -0600        :Working: DO: Setup work directory
    2019-02-10 02:09:13 -0600        :SUCCESS: DONE: Setup work directory
    2019-02-10 02:09:15 -0600        :SUCCESS: DONE: Initialize files.
    2019-02-10 02:09:15 -0600        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes …
    2019-02-10 02:09:28 -0600        :INFO   : Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
    2019-02-10 02:09:30 -0600        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
    2019-02-10 02:09:30 -0600        :Working: DO: Check space and state of cell services. Up to 20 minutes …
    2019-02-10 02:10:05 -0600        :SUCCESS: DONE: Check space and state of cell services.
    2019-02-10 02:10:05 -0600        :Working: DO: Check prerequisites on all cells. Up to 2 minutes …
    2019-02-10 02:10:13 -0600        :SUCCESS: DONE: Check prerequisites on all cells.
    2019-02-10 02:10:13 -0600        :Working: DO: Copy the patch to all cells. Up to 3 minutes …
    2019-02-10 02:12:01 -0600        :SUCCESS: DONE: Copy the patch to all cells.
    2019-02-10 02:12:03 -0600        :Working: DO: Execute plugin check for Patch Check Prereq …
    2019-02-10 02:12:03 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 22909764 v1.0.
    2019-02-10 02:12:03 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:12:03 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3.
    2019-02-10 02:12:03 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:12:03 -0600        :SUCCESS: No exposure to bug 17854520 with non-rolling patching
    2019-02-10 02:12:03 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 22468216 v1.0.
    2019-02-10 02:12:03 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:12:03 -0600        :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22468216
    2019-02-10 02:12:03 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 24625612 v1.0.
    2019-02-10 02:12:03 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:12:03 -0600        :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 24625612
    2019-02-10 02:12:03 -0600        :SUCCESS: No exposure to bug  with non-rolling patching
    2019-02-10 02:12:03 -0600        :INFO   : Patchmgr plugin start: Prereq check for exposure to bug 22651315 v1.0.
    2019-02-10 02:12:03 -0600        :INFO   : Details in logfile /u01/app/oracle/software/exa_patches/patch_18.1.12.0.0.190111/patchmgr.stdout.
    2019-02-10 02:12:05 -0600        :SUCCESS: Patchmgr plugin complete: Prereq check passed for the bug 22651315
    2019-02-10 02:12:06 -0600        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
    2019-02-10 02:12:12 -0600 1 of 5 :Working: DO: Initiate patch on cells. Cells will remain up. Up to 5 minutes …
    2019-02-10 02:12:16 -0600 1 of 5 :SUCCESS: DONE: Initiate patch on cells.
    2019-02-10 02:12:16 -0600 2 of 5 :Working: DO: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes …
    2019-02-10 02:13:16 -0600        :INFO   : Wait for patch pre-reboot procedures
    2019-02-10 02:14:34 -0600 2 of 5 :SUCCESS: DONE: Waiting to finish pre-reboot patch actions.
    2019-02-10 02:14:34 -0600        :Working: DO: Execute plugin check for Patching …
    2019-02-10 02:14:34 -0600        :SUCCESS: DONE: Execute plugin check for Patching.
    2019-02-10 02:14:35 -0600 3 of 5 :Working: DO: Finalize patch on cells. Cells will reboot. Up to 5 minutes …
    2019-02-10 02:14:39 -0600 3 of 5 :SUCCESS: DONE: Finalize patch on cells.
    2019-02-10 02:15:41 -0600 4 of 5 :Working: DO: Wait for cells to reboot and come online. Up to 120 minutes …
    2019-02-10 02:16:41 -0600        :INFO   : Wait for patch finalization and reboot
    2019-02-10 02:44:33 -0600 4 of 5 :SUCCESS: DONE: Wait for cells to reboot and come online.
    2019-02-10 02:44:33 -0600 5 of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes …
    2019-02-10 02:44:52 -0600 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.
    2019-02-10 02:44:52 -0600        :Working: DO: Execute plugin check for Pre Disk Activation …
    2019-02-10 02:44:53 -0600        :SUCCESS: DONE: Execute plugin check for Pre Disk Activation.
    2019-02-10 02:44:53 -0600        :Working: DO: Activate grid disks…
    2019-02-10 02:44:54 -0600        :INFO   : Wait for checking and activating grid disks
    2019-02-10 02:45:00 -0600        :SUCCESS: DONE: Activate grid disks.
    2019-02-10 02:45:03 -0600        :Working: DO: Execute plugin check for Post Patch …
    2019-02-10 02:45:03 -0600        :SUCCESS: DONE: Execute plugin check for Post Patch.
    2019-02-10 02:45:04 -0600        :Working: DO: Cleanup
    2019-02-10 02:45:56 -0600        :SUCCESS: DONE: Cleanup

    • Monitor the log files and cells being updated when e-mail alerts are not setup. open a new session and do a tail on the log file as shown below

    [root@dm01db01 patch_18.1.12.0.0.190111]# tail -f patchmgr.stdout

    • Verify the update status after the patchmgr utility completes as follows:

    [root@dm01cel01 ~]# imageinfo

    Kernel version: 4.1.12-94.8.10.el6uek.x86_64 #2 SMP Sat Dec 22 21:26:11 PST 2018 x86_64
    Cell version: OSS_18.1.12.0.0_LINUX.X64_190111
    Cell rpm version: cell-18.1.12.0.0_LINUX.X64_190111-1.x86_64

    Active image version: 18.1.12.0.0.190111
    Active image kernel version: 4.1.12-94.8.10.el6uek
    Active image activated: 2019-02-10 02:43:36 -0600
    Active image status: success
    Active system partition on device: /dev/md5
    Active software partition on device: /dev/md7

    Cell boot usb partition: /dev/sdac1
    Cell boot usb version: 18.1.12.0.0.190111

    Inactive image version: 12.2.1.1.6.180125.1
    Inactive image activated: 2018-05-16 00:58:24 -0500
    Inactive image status: success
    Inactive system partition on device: /dev/md6
    Inactive software partition on device: /dev/md8

    Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
    Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
    Inactive usb grub config for the rollback: /boot/grub/grub.conf.usb.inactive
    Inactive kernel version for the rollback: 4.1.12-94.7.8.el6uek.x86_64
    Rollback to the inactive partitions: Possible

    • Check the imagehistory

    [root@dm01cel01 ~]# imagehistory
    Version                              : 12.1.1.1.1.140712
    Image activation date                : 2014-11-23 00:34:06 -0800
    Imaging mode                         : fresh
    Imaging status                       : success

    Version                              : 12.1.1.1.2.150411
    Image activation date                : 2015-05-28 21:40:16 -0500
    Imaging mode                         : out of partition upgrade
    Imaging status                       : success

    Version                              : 12.1.2.3.2.160721
    Image activation date                : 2016-10-14 02:45:04 -0500
    Imaging mode                         : out of partition upgrade
    Imaging status                       : success

    Version                              : 12.1.2.3.4.170111
    Image activation date                : 2017-04-04 00:25:08 -0500
    Imaging mode                         : out of partition upgrade
    Imaging status                       : success

    Version                              : 12.1.2.3.6.170713
    Image activation date                : 2017-10-19 03:40:28 -0500
    Imaging mode                         : out of partition upgrade
    Imaging status                       : success

    Version                              : 12.2.1.1.6.180125.1
    Image activation date                : 2018-05-16 00:58:24 -0500
    Imaging mode                         : out of partition upgrade
    Imaging status                       : success

    Version                              : 18.1.12.0.0.190111
    Image activation date                : 2019-02-10 02:43:36 -0600
    Imaging mode                         : out of partition upgrade
    Imaging status                       : success

    • Verify the image on all cells

    [root@dm01db01 ~]# dcli -g cell_group -l root ‘imageinfo | grep “Active image version”‘
    dm01cel01: Active image version: 18.1.12.0.0.190111
    dm01cel02: Active image version: 18.1.12.0.0.190111
    dm01cel03: Active image version: 18.1.12.0.0.190111
    dm01cel04: Active image version: 18.1.12.0.0.190111
    dm01cel05: Active image version: 18.1.12.0.0.190111
    dm01cel06: Active image version: 18.1.12.0.0.190111
    dm01cel07: Active image version: 18.1.12.0.0.190111

    • Clean up the cells using the -cleanup option to clean up all the temporary update or rollback files on the cells.

    [root@dm01db01 patch_18.1.12.0.0.190111]# ./patchmgr -cells ~/cell_group -cleanup

    2019-02-10 02:58:37 -0600        :Working: DO: Cleanup
    2019-02-10 02:58:39 -0600        :SUCCESS: DONE: Cleanup

    • Start Clusterware and databases

    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/crsctl check crs
    CRS-4639: Could not contact Oracle High Availability Services

    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘/u01/app/11.2.0.4/grid/bin/crsctl start crs’
    dm01db01: CRS-4123: Oracle High Availability Services has been started.
    dm01db02: CRS-4123: Oracle High Availability Services has been started.
    dm01db03: CRS-4123: Oracle High Availability Services has been started.
    dm01db04: CRS-4123: Oracle High Availability Services has been started.

    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘/u01/app/11.2.0.4/grid/bin/crsctl check crs’
    dm01db01: CRS-4638: Oracle High Availability Services is online
    dm01db01: CRS-4537: Cluster Ready Services is online
    dm01db01: CRS-4529: Cluster Synchronization Services is online
    dm01db01: CRS-4533: Event Manager is online
    dm01db02: CRS-4638: Oracle High Availability Services is online
    dm01db02: CRS-4537: Cluster Ready Services is online
    dm01db02: CRS-4529: Cluster Synchronization Services is online
    dm01db02: CRS-4533: Event Manager is online
    dm01db03: CRS-4638: Oracle High Availability Services is online
    dm01db03: CRS-4537: Cluster Ready Services is online
    dm01db03: CRS-4529: Cluster Synchronization Services is online
    dm01db03: CRS-4533: Event Manager is online
    dm01db04: CRS-4638: Oracle High Availability Services is online
    dm01db04: CRS-4537: Cluster Ready Services is online
    dm01db04: CRS-4529: Cluster Synchronization Services is online
    dm01db04: CRS-4533: Event Manager is online

    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/crsctl stat res -t | more
    ——————————————————————————–
    NAME           TARGET  STATE        SERVER                   STATE_DETAILS
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.DATA_dm01.dg
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
    ora.DBFS_DG.dg
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
    ora.LISTENER.lsnr
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
    ora.RECO_dm01.dg
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
    ora.asm
                   ONLINE  ONLINE       dm01db01                 Started
                   ONLINE  ONLINE       dm01db02                 Started
                   ONLINE  ONLINE       dm01db03                 Started
                   ONLINE  ONLINE       dm01db04                 Started
    ora.gsd
                   OFFLINE OFFLINE      dm01db01
                   OFFLINE OFFLINE      dm01db02
                   OFFLINE OFFLINE      dm01db03
                   OFFLINE OFFLINE      dm01db04
    ora.net1.network
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
    ora.ons
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
    ora.registry.acfs
                   ONLINE  OFFLINE      dm01db01
                   ONLINE  OFFLINE      dm01db02
                   ONLINE  OFFLINE      dm01db03
                   ONLINE  OFFLINE      dm01db04
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.LISTENER_SCAN1.lsnr
          1        ONLINE  ONLINE       dm01db04
    ora.LISTENER_SCAN2.lsnr
          1        ONLINE  ONLINE       dm01db03
    ora.LISTENER_SCAN3.lsnr
          1        ONLINE  ONLINE       dm01db01
    ora.cvu
          1        ONLINE  ONLINE       dm01db02
    ora.dbm01.db
          1        OFFLINE OFFLINE
          2        OFFLINE OFFLINE
          3        OFFLINE OFFLINE
          4        OFFLINE OFFLINE
    ora.dm01db01.vip
          1        ONLINE  ONLINE       dm01db01
    ora.dm01db02.vip
          1        ONLINE  ONLINE       dm01db02
    ora.dm01db03.vip
          1        ONLINE  ONLINE       dm01db03
    ora.dm01db04.vip
          1        ONLINE  ONLINE       dm01db04
    ora.oc4j
          1        ONLINE  ONLINE       dm01db02
    ora.orcldb.db
          1        ONLINE  ONLINE       dm01db01                 Open
          2        ONLINE  ONLINE       dm01db02                 Open
          3        ONLINE  ONLINE       dm01db03                 Open
          4        ONLINE  ONLINE       dm01db04                 Open
    ora.nsmdb.db
          1        ONLINE  ONLINE       dm01db01                 Open
          2        ONLINE  ONLINE       dm01db02                 Open
          3        ONLINE  ONLINE       dm01db03                 Open
          4        ONLINE  ONLINE       dm01db04                 Open
    ora.scan1.vip
          1        ONLINE  ONLINE       dm01db04
    ora.scan2.vip
          1        ONLINE  ONLINE       dm01db03
    ora.scan3.vip
          1        ONLINE  ONLINE       dm01db01

    • Verify the databases and start them if needed

    $ srvctl status database -d orcldb
    $ srvctl status database -d nsmdb
     

    Conclusion

    In this article we have learned how to perform upgrade Exadata Storage cells using patchmgr utility. The patchmgr utility can be used for upgrading, rollback and backup Exadata Storage cells. patchmgr utility can be used for upgrading Storage cells in a rolling or non-rolling fashion. Non-Rolling is default. Storage server patches apply operating system, firmware, and driver updates. Launch patchmgr from the compute node that is node 1 that has user equivalence setup to all the storage cells.

  • Change root User Password on Exadata Infiniband Switch

    Change root User Password on Exadata Infiniband Switch

    I was working on changing password for the administrative user accounts on all Exadata Components. I encountered a strange issue while changing the root password on Infiniband Switch. We were unable to change the root password on IB Siwtch using command line method. We used couple different command line methods to change the root password on IB switches but all of them failed. This could be a BUG, firmware issue or something else.

    In this article we demonstrate how to change the root password on an Exadata infiniband switch using Browser User Interface.

    Issue 1: Using passwd command

    Tried to change the root user password using passwd command using dcli. This method assumes you are have ssh equivalence setup from compute node 1. As you can see the command failed saying to use the ILOM shell. In the past I have used the same command successfully to change the root password on IB Switches.

    [root@dm01db01 ~]#  dcli -g ibswitch_group -l root “echo welcome1 | passwd –stdin root”
    dm01sw-ibb01: This command should not be used for ILOM users.
    dm01sw-ibb01: Please use ILOM shell to handle password for this user.
    dm01sw-ibb01: Example:
    dm01sw-ibb01: -> set /SP/users/root password
    dm01sw-ibb01:
    dm01sw-iba01: This command should not be used for ILOM users.
    dm01sw-iba01: Please use ILOM shell to handle password for this user.
    dm01sw-iba01: Example:
    dm01sw-iba01: -> set /SP/users/root password
    dm01sw-iba01:

    So I decided to login to the IB switch directly and use the passwd command instead of running from dcli. The passwd command fail again with the same error.

    [root@dm01sw-iba01 ~]# ssh dm01sw-ibb01
    You are now logged in to the root shell.
    It is recommended to use ILOM shell instead of root shell.
    All usage should be restricted to documented commands and documented
    config files.
    To view the list of documented commands, use “help” at linux prompt.

    [root@dm01sw-ibb01 ~]# hostname
    dm01sw-ibb01

    [root@dm01sw-iba01 ~]# passwd root
    This command should not be used for ILOM users.
    Please use ILOM shell to handle password for this user.
    Example:
       -> set /SP/users/root password

    eBook - Oracle Exadata X8M Patching Recipes | Netsoftmate

    Issue 2: Using ILOM Shell

    As the passwd command failed asking to use the ILOM shell, I login to the IB switch as ilom-admin and executed the change password command. What I see is, the password change command failed at ILOM prompt as well.

    [root@dm01sw-iba01 ~]# su – ilom-admin

    Oracle(R) Integrated Lights Out Manager
    Version 2.2.7-1 ILOM 3.2.6 r118629
    Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
    Warning: HTTPS certificate is set to factory default.
     
    Hostname: dm01sw-iba01

    -> set /SP/users/root welcome1
    set: Invalid command syntax
    Usage: set [-script] [target] <property>=<value> [<property>=<value>…]


     
    Solution: Using Browser User Interface

    I have decided to use the BUI to change the password.

    Steps:

    • Open a Browser and enter the IB Switch hostname or IP address

    https://dm01sw-ibb01.netsoftmate.com

    • Accept the security warning and proceed to connect to the IB Switch
    • Enter the username and password to connect to the IB Switch

    • This show the summary page


    • On the left Pan, expand ILOM administration and select User Management


    • Click on  User Accounts, Select root user and click on edit button


    • Enter the new password and confirm and Finally click on the Save button to change the password.


    • To Verify the new password, open a Putty session and ssh to IB Switch using new password.

    [root@dm01db01 ~]# ssh dm01sw-ibb01
    Password:
    You are now logged in to the root shell.
    It is recommended to use ILOM shell instead of root shell.
    All usage should be restricted to documented commands and documented
    config files.
    To view the list of documented commands, use “help” at linux prompt.

    [root@dm01sw-ibb01 ~]# hostname
    dm01sw-ibb01

    Conclusion

    In this article we have learned how to change the root password on Infiniband Switch using Browser User Interface when the command line option doesn’t work.

  • How To Clear Hardware Fault on Exadata Infiniband Switch Manually

    Introduction

    We had a FAN failure on Exadata Infiniband Switch (FAN2). Scheduled the faulty hardware replacement with Oracle. The Oracle Feild Engineer came to the Customer Data Center and replaced the faulty FAN on Infiniband Switch. The FAN replacement was successful however the fault was not cleared automatically. We can still see the FAN was marked faulted from Infiniband BUI and CLI.

    From Infiniband Browser User Interface


    In this article we will demonstrate how to clear the fault on Infiniband Switch after hardware replacement.

    • Login to the Infiniband switch using Putty as root user and check the Infiniband health. From the output below we can see the FANs are all good.

    [root@dm01sw-iba01 ~]# env_test
    Environment test started:
    Starting Environment Daemon test:
    Environment daemon running
    Environment Daemon test returned OK
    Starting Voltage test:
    Voltage ECB OK
    Measured 3.3V Main = 3.28 V
    Measured 3.3V Standby = 3.39 V
    Measured 12V = 11.97 V
    Measured 5V = 5.02 V
    Measured VBAT = 3.14 V
    Measured 2.5V = 2.49 V
    Measured 1.8V = 1.79 V
    Measured I4 1.2V = 1.22 V
    Voltage test returned OK
    Starting PSU test:
    PSU 0 present OK
    PSU 1 present OK
    PSU test returned OK
    Starting Temperature test:
    Back temperature 40
    Front temperature 41
    SP temperature 57
    Switch temperature 55, maxtemperature 59
    Temperature test returned OK
    Starting FAN test:
    Fan 0 not present
    Fan 1 running at rpm 17004
    Fan 2 running at rpm 15696
    Fan 3 running at rpm 17004
    Fan 4 not present
    FAN test returned OK
    Starting Connector test:
    Connector test returned OK
    Starting Onboard ibdevice test:
    Switch OK
    All Internal ibdevices OK
    Onboard ibdevice test returned OK
    Starting SSD test:
    SSD test returned OK
    Starting Auto-link-disable test:
    Auto-link-disable test returned OK
    Environment test PASSED

    • Check the FAN Speed. FAN looks good.

    [root@dm01sw-iba01 ~]# getfanspeed
    Fan 0 not present
    Fan 1 running at rpm 17004
    Fan 2 running at rpm 15478
    Fan 3 running at rpm 17004
    Fan 4 not present

    • Switch to the ilom-admin user

    [root@dm01sw-iba01 ~]# su – ilom-admin

    Oracle(R) Integrated Lights Out Manager

    Version 2.2.9-3 ILOM 3.2.11 r124039

    Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.

    Warning: HTTPS certificate is set to factory default.

    Hostname: dm01sw-iba01.netsoftmate.com

    ->

    • Now check the fault table for any faulty components. Now we can see the FAN2 is Faulted though the FAN was replaced with a new FAN.

    -> show / -a -l 4 -o table fault_state
    Target                                  | Property                                     | Value
    —————————————-+———————————————-+——————————————————————–
    /SYS                                    | fault_state                                  | OK
    /SYS/MB                                 | fault_state                                  | OK
    /SYS/PSU0                               | fault_state                                  | OK
    /SYS/PSU1                               | fault_state                                  | OK
    /SYS/FAN1                               | fault_state                                  | OK
    /SYS/FAN2                               | fault_state                                  | Faulted /SYS/FAN3                               | fault_state                                  | OK

    ->

    • You can also execute the below command to identify the fault

    -> show -d targets /SP/faultmgmt

     /SP/faultmgmt
        Targets:
            shell
            0 (/SYS/FAN2)

    • Clear the Fault as show below

    -> set /SYS/FAN2 clear_fault_action=true
    Are you sure you want to clear /SYS/FAN2 (y/n)? y
    Set ‘clear_fault_action’ to ‘true’

    • Verify the fault is cleared

    -> show / -a -l 4 -o table fault_state
    Target                                  | Property                                     | Value
    —————————————-+———————————————-+——————————————————————–
    /SYS                                    | fault_state                                  | OK
    /SYS/MB                                 | fault_state                                  | OK
    /SYS/PSU0                               | fault_state                                  | OK
    /SYS/PSU1                               | fault_state                                  | OK
    /SYS/FAN1                               | fault_state                                  | OK
    /SYS/FAN2                               | fault_state                                      | OK
    /SYS/FAN3                               | fault_state                                  | OK

    -> show -d targets /SP/faultmgmt

     /SP/faultmgmt
        Targets:
            shell

    • Verify from the Infiniband Band BUI

    Conclusion

    In this article we have learned how to identify the fault and clear it manually on an Exadata Infiniband Switch. The ILOM commands comes handy for clearing the fault. You can also clear the fault using the Browser User Interface (BUI).

  • Update PDU firmware on Exadata Database Machine

    In this article we will demonstrate how to update the Power Distribution Unit (PDU) firmware on Exadata Database Machine

    You may update the firmware on a PDU if you are running an older version in some cases you need to update the firmware after replacing a hardware component.


    Get the current PDU Firmware version

    • Using a web browser login to the PDU using the hostname or IP address

    https://dm01sw-pdua01.netsoftmate.com/

     

    • Click on “Module Info”

    Under “System Info”, look for “Firmware Version”.
    In my case the PDU firmware version is 2.01.



    Steps to Upgrade PDU firmware version

    • Login to My Oracle Support and open the MOS note 888828.1

    • Under the MOS note 888828.1 go to the section “Power Distribution Unit (PDU) metering unit”

    • Download the latest PDU patch available. In this case it is 28529456

    • Unzip the patch zip file in a location on your desktop or laptop

    • You will see 2 files after unzipping the patch

    • Using a web browser login to the PDU using the hostname or IP address

    https://dm01sw-pdua01.netsoftmate.com/

    • Click on “Net Configuration”

    • Enter the username and password. Default username and password is admin/admin

    Note: In some cases, the default password could be admin or adm1n or welcome1

    • Click on “Firmware Update”

    • Click on the Browse button and select the MKAppl_V2.09.DL file that was created after unzipping the patch above

    • Click on the Submit button to update the metering unit firmware

    • We can now see the file is being uploaded

    • After Firmware Update is complete & PDU Restart. Wait for couple of minutes.

    • Click on the Module Info link to verify the firmware updated version and confirm that the firmware updated successfully.

    • Click on Logout

    • Repeat the above steps on second PDU in the Exadata Rack.



    Conclusion
    In this article we have learned how to update the PDU firmware on an Exadata Database Machine.

  • Creating A New File System on Exadata Compute Node

    Creating A New File System on Exadata Compute Node

    When the Exadata Database Machine is installed by Oracle ACS, you will see the following default file system created:

    • /
    • /dev/shm
    • /boot
    • /u01

    There is a plenty of space available in the volume group and this can be used to increase the existing
    file system size or you can create a new file system.


    In this article we will demonstrate how to create a new file system (named /u02) on Exadata Compute
    node.

    • Connect to the compute node as root user

    login as: rootroot@dm01db01’s password:
    Last login: Mon Nov 19 13:11:39 2018 from 10.xx.xxx.xxx

    • List the exisint file system on Exadata Compute node

    [root@dm01db01 ~]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VGExaDb-LVDbSys1
                           59G   38G   19G  67% /
    tmpfs                 252G  6.0M  252G   1% /dev/shm
    /dev/sda1             480M   63M  393M  14% /boot
    /dev/mapper/VGExaDb-LVDbOra1
                          197G   97G   91G  52% /u01

    • Get the free space available in the volume group

    [root@dm01db01 ~]# vgdisplay | grep Free
      Free  PE / Size       337428 / 1.29 TiB

     
    • List the physical volumes and logical volumes

    [root@dm01db01 ~]# pvs
     PV         VG      Fmt  Attr PSize   PFree
      /dev/sda2  VGExaDb lvm2 a–u 557.36g 202.36g
      /dev/sda3  VGExaDb lvm2 a–u   1.09t   1.09t

    [root@dm01db01 ~]# lvs
      LV                 VG      Attr       LSize   Pool Origin   Data%  Meta%  Move Log Cpy%Sync Convert
      LVDbOra1           VGExaDb owi-aos— 200.00g
      LVDbSwap1          VGExaDb -wi-ao—-  24.00g
      LVDbSys1           VGExaDb owi-aos—  60.00g
      LVDbSys2           VGExaDb -wi-a—–  60.00g
      LVDoNotRemoveOrUse VGExaDb -wi-a—–   1.00g
      root_snap          VGExaDb swi-I-s—   5.00g      LVDbSys1 100.00
      u01_snap           VGExaDb swi-I-s—   5.00g      LVDbOra1 100.00

     
    • Create a new logical volume of your desired size. Here we are creating a logical volume of 100GB size

    [root@dm01db01 ~]# lvcreate -L100GB -n LVDbOra2 VGExaDb
      Logical volume “LVDbOra2” created.

    • List the logical volumes and ensure our new logical volume is displayed

    [root@dm01db01 ~]# lvs
      LV                 VG      Attr       LSize   Pool Origin   Data%  Meta%  Move Log Cpy%Sync Convert
      LVDbOra1           VGExaDb owi-aos— 200.00g
      LVDbOra2           VGExaDb -wi-a—– 100.00g
      LVDbSwap1          VGExaDb -wi-ao—-  24.00g
      LVDbSys1           VGExaDb owi-aos—  60.00g
      LVDbSys2           VGExaDb -wi-a—–  60.00g
      LVDoNotRemoveOrUse VGExaDb -wi-a—–   1.00g
      root_snap          VGExaDb swi-I-s—   5.00g      LVDbSys1 100.00
      u01_snap           VGExaDb swi-I-s—   5.00g      LVDbOra1 100.00

     


    • Now create the new file system as shown below

    [root@dm01db01 ~]# mkfs.ext3 -j -L u02 /dev/VGExaDb/LVDbOra2
    mke2fs 1.43-WIP (20-Jun-2013)
    Filesystem label=u02
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    Stride=0 blocks, Stripe width=0 blocks
    6553600 inodes, 26214400 blocks
    1310720 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=4294967296
    800 block groups
    32768 blocks per group, 32768 fragments per group
    8192 inodes per group
    Superblock backups stored on blocks:
            32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
            4096000, 7962624, 11239424, 20480000, 23887872
    Allocating group tables: done
    Writing inode tables: done
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done

    • Mount the new file system

    [root@dm01db01 ~]# mkdir /u02

    [root@dm01db01 ~]# mount -t ext3 /dev/VGExaDb/LVDbOra2 /u02

     


    • Verify that the new file system is mounted and accessible

    [root@dm01db01 ~]# df -kh
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VGExaDb-LVDbSys1
                           59G   38G   19G  67% /
    tmpfs                 252G  6.0M  252G   1% /dev/shm
    /dev/sda1             480M   63M  393M  14% /boot
    /dev/mapper/VGExaDb-LVDbOra1
                          197G   97G   91G  52% /u01
    /dev/mapper/VGExaDb-LVDbOra2
                           99G   60M   94G   1% /u02


    Conclusion
    In this article we have learned how to create a new file system on Exadata Compute node using the free space available in the volume group.

    eBook - Oracle Exadata X8M Patching Recipes | Netsoftmate

  • Oracle Transparent Data Encryption (TDE) On Exadata

    Oracle Transparent Data Encryption (TDE) enables the organizations to encrypt sensitive application data on storage media completely transparent to the application. TDE protects the data at rest. TDE encrypts the data in the datafiles so that in case they are obtained by hacker or theft it will not be possible to access the clear text data. In the databases where TDE is configured any user who has access on an encrypted table will be able to see the data in clear text because Oracle will transparently decrypt the data for any user having the necessary privileges. Oracle TDE is available by default in Oracle RDBMS Enteprise Edition. To be able to use this it is necessary to purchase an Oracle Advanced Security license.




    In this article we will demonstrate how to implement TDE on Exadata Database Machine to protect sensitive data. Here we are setting up TDE for multiple database running on the same cluster. However same steps are valid for one single database as well.




    Steps to configure or implement TDE




    Step 1: Create Directories and Grant File Permissions


    # dcli -g ~/dbs_group -l root mkdir -p /etc/oracle/wallets/


    #dcli -g ~/dbs_group -l root -s “-q” ‘ls -l /etc/oracle/wallets’


    #dcli -g ~/dbs_group -l root ‘cd /etc/oracle/wallets;ls -l’


    #dcli -g ~/dbs_group -l root ‘cd /etc;chown -R oracle:oinstall oracle’


    #dcli -g ~/dbs_group -l root ‘cd /etc;chmod -R 700 oracle’


    #dcli -g ~/dbs_group -l oracle ‘cd /etc/oracle/wallets;mkdir orcldb’


    #dcli -g ~/dbs_group -l oracle ‘cd /etc/oracle/wallets;mkdir nsmdb’




    Step 2: Set the ENCRYPTION_WALLET_LOCATION parameter in sqlnet.ora to the newly created directory.


    – backup the sqlnet.ora file on all the nodes


    $ dcli -g ~/dbs_group -l oracle ‘cd /u01/app/oracle/product/11.2.0.4/dbhome_1/network/admin;cp sqlnet.ora sqlnet.ora_bkp’


    $ dcli -g ~/dbs_group -l oracle ‘cd /u01/app/oracle/product/11.2.0.4/dbhome_1/network/admin;ls -l sqlnet.ora*’


    – Update the sqlnet.ora on node 1 add the below line to the sqlnet.ora file


    $ cd /u01/app/oracle/product/11.2.0.4/dbhome_1/network/admin
    $ vi sqlnet.ora


    ENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /etc/oracle/wallets/$ORACLE_UNQNAME/)))




    Step 3: Copy sqlnet.ora to all the nodes from node 1


    dm01db01-nsmdb1 {/home/oracle}:scp sqlnet.ora oracle@dm01db02:/u01/app/oracle/product/11.2.0.4/dbhome_1/network/admin/


    dm01db01-nsmdb1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘cd /u01/app/oracle/product/11.2.0.4/dbhome_1/network/admin;ls -l sqlnet.ora’


    – Verify the contents of sqlnet.ora on all the nodes:


    dm01db01-nsmdb1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘cd /u01/app/oracle/product/11.2.0.4/dbhome_1/network/admin;cat sqlnet.ora’




    Step 4: Initialize the wallet and add the master encryption key using the SQL*Plus command line interface:


     **************************VERY VERY IMPORTANT **************************


    AT THE OS PROMPT SET THE ENVIRONMENT VARIABLE export ORACLE_UNQNAME=orcldb


    dm01db01-nsmdb1 {/home/oracle}:export ORACLE_UNQNAME=orcldb


    dm01db01-nsmdb1 {/home/oracle}: sqlplus / as sysdba


    SQL> col WRL_PARAMETER for a60
    SQL> set lines 200
    SQL> select * from gv$encryption_wallet order by 1;


       INST_ID WRL_TYPE             WRL_PARAMETER                                                STATUS
    ———- ——————– ———————————————————— ——————
             1 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             2 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             3 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             4 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             5 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             6 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             7 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             8 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED


    8 rows selected.


    SQL> alter system set encryption key identified by “welcom1”;


    System altered.


    SQL> select * from gv$encryption_wallet order by 1;


       INST_ID WRL_TYPE             WRL_PARAMETER                                                STATUS
    ———- ——————– ———————————————————— ——————
             1 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             2 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             3 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             4 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             5 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             6 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             7 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             8 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED




    Step 5: Copy the wallet file ewallet.p12 to all the nodes:


    dm01db01-orcldb1 {/home/oracle}:cd /etc/oracle/wallets/orcldb/


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:ls -ltr
    total 8
    -rw——- 1 oracle oinstall 1837 Feb 22 08:28 ewallet.p12_20130226
    -rw-r–r– 1 oracle oinstall 1309 Feb 26 05:14 ewallet.p12


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:scp ewallet.p12 oracle@dm01db02:/etc/oracle/wallets/orcldb/
    ewallet.p12                                                 100% 1309     1.3KB/s   00:00




    Step 6: verify at the wallet is opened on all the nodes. This is the default behavior after you copy the wallet file to all the nodes


    dm01db01-nsmdb1 {/home/oracle}: sqlplus / as sysdba


    SQL> col WRL_PARAMETER for a60
    SQL> set lines 200
    SQL> select * from gv$encryption_wallet order by 1;


       INST_ID WRL_TYPE             WRL_PARAMETER                                                STATUS
    ———- ——————– ———————————————————— ——————
             1 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             2 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             3 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             4 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             5 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             6 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             7 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             8 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN


    8 rows selected.




    – IF FOR SOME REASON IF THE WALLET IS NOT OPENED, FOLLOW THE BELOW STEPS


    A) set the variable
    export ORACLE_UNQNAME=orcldb


    B) login to the database
    sqlplus / as sysdba


    c) open the wallet
    ALTER SYSTEM SET WALLET OPEN IDENTIFIED BY “welcome1”;


    D) Repeat the A,B and C for all the instances in the cluster.




    Step 7: Whenever you bounce the database you must open the wallet manually as follows:


    dm01db02-orcldb2 {/home/oracle}:srvctl stop database -d orcldb
    dm01db01-orcldb1 {/home/oracle}:srvctl start database -d orcldb


    dm01db01-orcldb1 {/home/oracle}:export ORACLE_UNQNAME=orcldb
    dm01db01-orcldb1 {/home/oracle}:sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Tue Feb 26 05:23:02 2013


    Copyright (c) 1982, 2011, Oracle.  All rights reserved.


    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options


    SQL> select name,open_mode,database_role from gv$database;


    NAME      OPEN_MODE            DATABASE_ROLE
    ——— ——————– —————-
    orcldb   READ WRITE           PRIMARY
    orcldb   READ WRITE           PRIMARY
    orcldb   READ WRITE           PRIMARY
    orcldb   READ WRITE           PRIMARY
    orcldb   READ WRITE           PRIMARY
    orcldb   READ WRITE           PRIMARY
    orcldb   READ WRITE           PRIMARY
    orcldb   READ WRITE           PRIMARY


    8 rows selected.


    SQL> col WRL_PARAMETER for a60
    SQL> set lines 200
    SQL> select * from gv$encryption_wallet order by 1;


       INST_ID WRL_TYPE             WRL_PARAMETER                                                STATUS
    ———- ——————– ———————————————————— ——————
             1 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             2 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             3 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             4 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             5 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             6 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             7 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED
             8 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         CLOSED


    8 rows selected.




    SQL> ALTER SYSTEM SET WALLET OPEN IDENTIFIED BY “welcome1”;


    System altered.


    SQL> select * from gv$encryption_wallet order by 1;


       INST_ID WRL_TYPE             WRL_PARAMETER                                                STATUS
    ———- ——————– ———————————————————— ——————
             1 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             2 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             3 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             4 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             5 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             6 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             7 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             8 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN


    8 rows selected.




    Step 8:  Making the wallet Auto-Login. If you have a auto-login wallet you don’t need to open the wallet after a database bounce. This is done automatic for you.


    dm01db01-orcldb1 {/home/oracle} export ORACLE_UNQNAME=orcldb
       
    dm01db01-orcldb1 {/home/oracle}:ls -l /etc/oracle/wallets/$ORACLE_UNQNAME
    total 8
    -rw-r–r– 1 oracle oinstall 1309 Feb 26 05:14 ewallet.p12


    dm01db01-orcldb1 {/home/oracle}:orapki wallet create -wallet /etc/oracle/wallets/$ORACLE_UNQNAME -auto_login
    Oracle PKI Tool : Version 11.2.0.4.0 – Production
    Copyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved.


    Enter wallet password:


    dm01db01-orcldb1 {/home/oracle}:ls -l /etc/oracle/wallets/$ORACLE_UNQNAME
    total 12
    -rw——- 1 oracle oinstall 1387 Feb 26 05:26 cwallet.sso
    -rw-r–r– 1 oracle oinstall 1309 Feb 26 05:14 ewallet.p12


    dm01db01-orcldb1 {/home/oracle}:cd /etc/oracle/wallets/orcldb/
    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:ls -ltr
    total 12
    -rw-r–r– 1 oracle oinstall 1309 Feb 26 05:14 ewallet.p12
    -rw——- 1 oracle oinstall 1387 Feb 26 05:26 cwallet.sso


    – Copy the cwallet.sso to all the nodes:


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:scp cwallet.sso oracle@dm01db02:/etc/oracle/wallets/orcldb/


    cwallet.sso                                  100% 1387     1.4KB/s   00:00


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Tue Feb 26 05:28:26 2013


    Copyright (c) 1982, 2011, Oracle.  All rights reserved.


    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options


    SQL> col WRL_PARAMETER for a60
    SQL> set lines 200
    SQL> select * from gv$encryption_wallet order by 1;


       INST_ID WRL_TYPE             WRL_PARAMETER                                                STATUS
    ———- ——————– ———————————————————— ——————
             1 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             2 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             3 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             4 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             5 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             6 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             7 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             8 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN


    8 rows selected.


    SQL> exit
    Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options


    – Verify that the wallet is opened following a database bounce:


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:srvctl stop database -d orcldb


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:srvctl start database -d orcldb


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:srvctl status database -d orcldb
    Instance orcldb1 is running on node dm01db01
    Instance orcldb2 is running on node dm01db02
    Instance orcldb3 is running on node dm01db03
    Instance orcldb4 is running on node dm01db04
    Instance orcldb5 is running on node dm01db05
    Instance orcldb6 is running on node dm01db06
    Instance orcldb7 is running on node dm01db07
    Instance orcldb8 is running on node dm01db08


    dm01db01-orcldb1 {/etc/oracle/wallets/orcldb}:sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Tue Feb 26 05:32:19 2013


    Copyright (c) 1982, 2011, Oracle.  All rights reserved.


    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options


    SQL> col WRL_PARAMETER for a60
    SQL> set lines 200
    SQL> select * from gv$encryption_wallet order by 1;


       INST_ID WRL_TYPE             WRL_PARAMETER                                                STATUS
    ———- ——————– ———————————————————— ——————
             1 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             2 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             3 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             4 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             5 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             6 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             7 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN
             8 file                 /etc/oracle/wallets/$ORACLE_UNQNAME/                         OPEN


    8 rows selected.




    ############ REAPEAT THE SAME STEPS FOR ALL THE DATABASES #############



    Testing Tablespace Encryption (TSE)



    Step 1: Create an Encrypted tablespace and grant the users permissions on the tablespace


    SQL> CREATE TABLESPACE enc_tbs DATAFILE ‘+DATA’ SIZE 1000M ENCRYPTION USING ‘AES256’ DEFAULT STORAGE(ENCRYPT);


    Tablespace created.


    SQL> select TABLESPACE_NAME, ENCRYPTED from dba_tablespaces where tablespace_name in (‘USERS’,’ENC_TBS’) order by 1;


    TABLESPACE_NAME                ENC
    —————————— —
    ENC_TBS                        YES
    USERS                          NO




    SQL> select TABLESPACE_NAME, FILE_NAME from dba_data_files where TABLESPACE_NAME in (‘USERS’,’ENC_TBS’);


    TABLESPACE_NAME FILE_NAME
    —————————— ——————————
    USERS +DATA/nsmdb/datafile/users.529.762113429
    ENC_TBS +DATA/nsmdb/datafile/enc_tbs.540.804301513




    SQL> alter user test quota unlimited on  ENC_TBS;


    User altered.


    SQL> alter user test quota unlimited on  USERS;


    User altered.




    Step 2: Create tables in encypted and non-encrypted tablespaces


    SQL> create table test.nonenc(data varchar2(100)) tablespace users;


    Table created.


    SQL> insert into test.nonenc values (‘ABCDEFGHIJKLMNOPQRSTUVWZYZ’);


    1 row created.


    SQL> commit;


    Commit complete.




    SQL> create table test.encrypted(data varchar2(100)) tablespace enc_tbs;


    Table created.


    SQL> insert into test.encrypted values (‘ABCDEFGHIJKLMNOPQRSTUVWZYZ’);


    1 row created.


    SQL> commit;


    Commit complete.


    SQL> alter system checkpoint;


    System altered.




    Step 3: Verify that the data is encrypted


    dm01db01-nsmdb1 {/u01/dba}:asm


    dm01db01-+ASM1 {/u01/dba}:asmcmd -p
    ASMCMD [+] > cd +DATA/NSMDB/DATA*
    ASMCMD [+DATA/NSMDB/DATAFILE] > ls -l


    ASMCMD [+DATA/NSMDB/DATAFILE] > cp USERS.529.762113429 /u01/dba/
    copying +DATA/NSMDB/DATAFILE/USERS.529.762113429 -> /u01/dba//USERS.529.762113429


    ASMCMD [+DATA/NSMDB/DATAFILE] > cp ENC_TBS.540.804301513 /u01/dba/
    copying +DATA/NSMDB/DATAFILE/ENC_TBS.540.804301513 -> /u01/dba//ENC_TBS.540.804301513


    dm01db01-+ASM1 {/u01/dba}:ls -ltr
    -rw-r—– 1 oracle oinstall  703602688 Jan 10 01:30 USERS.529.762113429
    -rw-r—– 1 oracle oinstall 1048584192 Jan 10 01:31 ENC_TBS.540.804301513




    dm01db01-nsmdb1 {/u01/dba}:strings /u01/dba/USERS.529.762113429 | grep ABCD
    89:;<=>?@ABCDEFGHIJKLMNOPQR|STUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz
    ABCDEFGHIJKLMNOPQRSTUVWZYZ


    dm01db01-nsmdb1 {/u01/dba}:strings /u01/dba/ENC_TBS.540.804301513 | grep ABCD


    – From the above 2 string commands it is clear that the tablespace encryption is indeed working for us.




    Conclusion


    In this article we have learned how to encrypt sensitive data using Transparent Data Encryption (TDE). TDE protects the data at rest. In the databases where TDE is configured any user who has access on an encrypted table will be able to see the data in clear text because Oracle will transparently decrypt the data for any user having the necessary privileges.

  • Move Database from DATA to RECO ASM Disk Group On Exadata

    Exadata Database machine consists of 3 ASM Disk Groups:
    +DATA for Database Files
    +RECO for Online Redo log and Archive log files
    +DBFS_DG for Cluster configuration files such as OCR and Voting disks


    In a Customized environment Customers can choose to have more than 3 Disk Groups. But it is recommended to have 3 Disk Groups. The DATA and RECO disk groups can be sized 80%-20% or 40%-60% respectively of over all storage capacity. Sometimes it is possible that +DATA disk group can be filled very fast if you have several databases.


    In this article we will demostrate how to move a Database from +DATA disk group to +RECO disk group.




    Steps to move a database from +DATA to +RECO ASM Disk Group:




    Step 1: Get the ASM Disk Information


    SQL> select state,name from v$asm_diskgroup;


    STATE       NAME
    ———– ——————————
    MOUNTED     RECO
    MOUNTED     DBFS_DG
    MOUNTED     DATA




    Step 2: Get the Database files details


    SQL> select name, open_mode,database_role from gv$database;


    NAME      OPEN_MODE            DATABASE_ROLE
    ——— ——————– —————-
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY


    SQL> select name from v$controlfile;


    NAME
    ——————————————————————————–
    +DATA/dbm01/controlfile/current.256.976374731


    SQL> select name from v$datafile;


    NAME
    ——————————————————————————–
    +DATA/dbm01/datafile/system.259.976374739
    +DATA/dbm01/datafile/sysaux.260.976374743
    +DATA/dbm01/datafile/undotbs1.261.976374745
    +DATA/dbm01/datafile/undotbs2.263.976374753
    +DATA/dbm01/datafile/undotbs3.264.976374755
    +DATA/dbm01/datafile/undotbs4.265.976374757
    +DATA/dbm01/datafile/users.266.976374757


    7 rows selected.


    SQL> select name from v$tempfile;


    NAME
    ——————————————————————————–
    +DATAC1/dbm01/tempfile/temp.262.976375229


    SQL>


    SQL> select member from v$logfile;


    MEMBER
    ——————————————————————————–
    +DATA/dbm01/onlinelog/group_1.257.976374733
    +DATA/dbm01/onlinelog/group_2.258.976374735
    +DATA/dbm01/onlinelog/group_7.267.976375073
    +DATA/dbm01/onlinelog/group_8.268.976375075
    +DATA/dbm01/onlinelog/group_5.269.976375079
    +DATA/dbm01/onlinelog/group_6.270.976375083
    +DATA/dbm01/onlinelog/group_3.271.976375085
    +DATA/dbm01/onlinelog/group_4.272.976375087
    +DATA/dbm01/onlinelog/group_9.274.976375205
    +DATA/dbm01/onlinelog/group_10.275.976375209
    +DATA/dbm01/onlinelog/group_11.276.976375211
    +DATA/dbm01/onlinelog/group_12.277.976375215
    +DATA/dbm01/onlinelog/group_13.278.976375217
    +DATA/dbm01/onlinelog/group_14.279.976375219
    +DATA/dbm01/onlinelog/group_15.280.976375223
    +DATA/dbm01/onlinelog/group_16.281.976375225


    16 rows selected.


    SQL> select filename from v$block_change_tracking;


    FILENAME
    ——————————————————————–
    +DATA/dbm01/changetracking/ctf.282.976375227




    Step 3: Backup Database using RMAN copy command as shown below. Here we are moving database to +RECO ASM Disk Group.


    RMAN> run {
    allocate channel c1 device type disk;
    allocate channel c2 device type disk;
    allocate channel c3 device type disk;
    allocate channel c4 device type disk;
    allocate channel c5 device type disk;
    allocate channel c6 device type disk;
    allocate channel c7 device type disk;
    allocate channel c8 device type disk;
    backup as copy database include current controlfile format ‘+RECO’;
    release channel c1;
    release channel c2;
    release channel c3;
    release channel c4;
    release channel c5;
    release channel c6;
    release channel c7;
    release channel c8;
    }


    released channel: ORA_DISK_1
    allocated channel: c1
    channel c1: SID=1189 instance=dbm011 device type=DISK


    allocated channel: c2
    channel c2: SID=1321 instance=dbm011 device type=DISK


    allocated channel: c3
    channel c3: SID=1343 instance=dbm011 device type=DISK


    allocated channel: c4
    channel c4: SID=1387 instance=dbm011 device type=DISK


    allocated channel: c5
    channel c5: SID=1497 instance=dbm011 device type=DISK


    allocated channel: c6
    channel c6: SID=1519 instance=dbm011 device type=DISK


    allocated channel: c7
    channel c7: SID=1541 instance=dbm011 device type=DISK


    allocated channel: c8
    channel c8: SID=1563 instance=dbm011 device type=DISK


    Starting backup at 26-MAY-18
    channel c1: starting datafile copy
    input datafile file number=00001 name=+DATA/dbm01/datafile/system.259.976374739
    channel c2: starting datafile copy
    input datafile file number=00002 name=+DATA/dbm01/datafile/sysaux.260.976374743
    channel c3: starting datafile copy
    input datafile file number=00003 name=+DATA/dbm01/datafile/undotbs1.261.976374745
    channel c4: starting datafile copy
    input datafile file number=00004 name=+DATA/dbm01/datafile/undotbs2.263.976374753
    channel c5: starting datafile copy
    input datafile file number=00005 name=+DATA/dbm01/datafile/undotbs3.264.976374755
    channel c6: starting datafile copy
    input datafile file number=00006 name=+DATA/dbm01/datafile/undotbs4.265.976374757
    channel c7: starting datafile copy
    input datafile file number=00007 name=+DATA/dbm01/datafile/users.266.976374757
    channel c8: starting datafile copy
    copying current control file
    output file name=+RECO/dbm01/datafile/users.284.977121353 tag=TAG20180526T063551 RECID=16 STAMP=977121353
    channel c7: datafile copy complete, elapsed time: 00:00:02
    output file name=+RECO/dbm01/controlfile/backup.283.977121353 tag=TAG20180526T063551 RECID=17 STAMP=977121353
    channel c8: datafile copy complete, elapsed time: 00:00:01
    output file name=+RECO/dbm01/datafile/system.291.977121353 tag=TAG20180526T063551 RECID=18 STAMP=977121389
    channel c1: datafile copy complete, elapsed time: 00:00:46
    output file name=+RECO/dbm01/datafile/sysaux.290.977121353 tag=TAG20180526T063551 RECID=23 STAMP=977121392
    channel c2: datafile copy complete, elapsed time: 00:00:46
    output file name=+RECO/dbm01/datafile/undotbs1.289.977121353 tag=TAG20180526T063551 RECID=21 STAMP=977121392
    channel c3: datafile copy complete, elapsed time: 00:00:46
    output file name=+RECO/dbm01/datafile/undotbs2.288.977121353 tag=TAG20180526T063551 RECID=19 STAMP=977121392
    channel c4: datafile copy complete, elapsed time: 00:00:46
    output file name=+RECO/dbm01/datafile/undotbs3.287.977121353 tag=TAG20180526T063551 RECID=20 STAMP=977121392
    channel c5: datafile copy complete, elapsed time: 00:00:46
    output file name=+RECO/dbm01/datafile/undotbs4.286.977121353 tag=TAG20180526T063551 RECID=22 STAMP=977121392
    channel c6: datafile copy complete, elapsed time: 00:00:46
    Finished backup at 26-MAY-18


    Starting Control File and SPFILE Autobackup at 26-MAY-18
    piece handle=+RECO/dbm01/autobackup/2018_05_26/s_977121397.282.977121399 comment=NONE
    Finished Control File and SPFILE Autobackup at 26-MAY-18


    released channel: c1


    released channel: c2


    released channel: c3


    released channel: c4


    released channel: c5


    released channel: c6


    released channel: c7


    released channel: c8




    Step 4: Verify the RMAN Database Copy using RMAN


    RMAN> list copy of database;


    List of Datafile Copies
    =======================


    Key     File S Completion Time Ckp SCN    Ckp Time
    ——- —- – ————— ———- —————
    18      1    A 26-MAY-18       1330853    26-MAY-18
            Name: +RECO/dbm01/datafile/system.291.977121353
            Tag: TAG20180526T063551


    9       1    A 26-MAY-18       1330410    26-MAY-18
            Name: +RECO/dbm01/datafile/system.286.977120961
            Tag: TAG20180526T062919


    3       1    A 26-MAY-18       1330155    26-MAY-18
            Name: +RECO/dbm01/datafile/system.280.977120795
            Tag: TAG20180526T062633


    23      2    A 26-MAY-18       1330856    26-MAY-18
            Name: +RECO/dbm01/datafile/sysaux.290.977121353
            Tag: TAG20180526T063551


    12      2    A 26-MAY-18       1330413    26-MAY-18
            Name: +RECO/dbm01/datafile/sysaux.287.977120961
            Tag: TAG20180526T062919


    2       2    A 26-MAY-18       1330158    26-MAY-18
            Name: +RECO/dbm01/datafile/sysaux.281.977120795
            Tag: TAG20180526T062633


    21      3    A 26-MAY-18       1330859    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs1.289.977121353
            Tag: TAG20180526T063551


    11      3    A 26-MAY-18       1330416    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs1.288.977120961
            Tag: TAG20180526T062919


    4       3    A 26-MAY-18       1330154    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs1.279.977120795
            Tag: TAG20180526T062633


    19      4    A 26-MAY-18       1330862    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs2.288.977121353
            Tag: TAG20180526T063551


    10      4    A 26-MAY-18       1330419    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs2.289.977120961
            Tag: TAG20180526T062919


    1       4    A 26-MAY-18       1330153    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs2.278.977120795
            Tag: TAG20180526T062633


    20      5    A 26-MAY-18       1330865    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs3.287.977121353
            Tag: TAG20180526T063551


    13      5    A 26-MAY-18       1330422    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs3.290.977120961
            Tag: TAG20180526T062919


    7       5    A 26-MAY-18       1330184    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs3.282.977120829
            Tag: TAG20180526T062633


    22      6    A 26-MAY-18       1330868    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs4.286.977121353
            Tag: TAG20180526T063551


    15      6    A 26-MAY-18       1330425    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs4.291.977120961
            Tag: TAG20180526T062919


    6       6    A 26-MAY-18       1330187    26-MAY-18
            Name: +RECO/dbm01/datafile/undotbs4.283.977120829
            Tag: TAG20180526T062633


    16      7    A 26-MAY-18       1330871    26-MAY-18
            Name: +RECO/dbm01/datafile/users.284.977121353
            Tag: TAG20180526T063551


    8       7    A 26-MAY-18       1330428    26-MAY-18
            Name: +RECO/dbm01/datafile/users.292.977120961
            Tag: TAG20180526T062919


    5       7    A 26-MAY-18       1330190    26-MAY-18
            Name: +RECO/dbm01/datafile/users.284.977120829
            Tag: TAG20180526T062633


    RMAN> list copy of controlfile;


    List of Control File Copies
    ===========================


    Key     S Completion Time Ckp SCN    Ckp Time
    ——- – ————— ———- —————
    17      A 26-MAY-18       1330876    26-MAY-18
            Name: +RECO/dbm01/controlfile/backup.283.977121353
            Tag: TAG20180526T063551


    14      A 26-MAY-18       1330434    26-MAY-18
            Name: +RECO/dbm01/controlfile/backup.293.977120965
            Tag: TAG20180526T062919




    Step 5: Verify the RMAN Database Copy backup in ASM


    [oracle@dm01db01 ~]$ asmcmd -p
    ASMCMD [+] > lsdg
    State    Type    Rebal  Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
    MOUNTED  HIGH    N         512   4096  4194304  272154624  271558968          6479872        88359698              0             N  DATA/
    MOUNTED  HIGH    N         512   4096  4194304    2404640    2402468            68704          777921              0             Y  DBFS_DG/
    MOUNTED  NORMAL  N         512   4096  4194304   45389568   45183784           540352        22321716              0             N  RECO/


    ASMCMD [+] > cd +RECO


    ASMCMD [+RECO] > ls -l
    Type  Redund  Striped  Time             Sys  Name
                                            Y    DBM01/
    ASMCMD [+RECO] > cd DBM01
    ASMCMD [+RECO/DBM01] > ls -l
    Type         Redund  Striped  Time             Sys  Name
                                                   Y    ARCHIVELOG/
                                                   Y    AUTOBACKUP/
                                                   Y    CONTROLFILE/
                                                   Y    DATAFILE/
                                                   N    snapcf_dbm01.f => +RECO/DBM01/CONTROLFILE/Backup.285.977120961
    ASMCMD [+RECO/DBM01] > ls -l DATAFILE/
    Type      Redund  Striped  Time             Sys  Name
    DATAFILE  MIRROR  COARSE   MAY 26 06:00:00  Y    SYSAUX.290.977121353
    DATAFILE  MIRROR  COARSE   MAY 26 06:00:00  Y    SYSTEM.291.977121353
    DATAFILE  MIRROR  COARSE   MAY 26 06:00:00  Y    UNDOTBS1.289.977121353
    DATAFILE  MIRROR  COARSE   MAY 26 06:00:00  Y    UNDOTBS2.288.977121353
    DATAFILE  MIRROR  COARSE   MAY 26 06:00:00  Y    UNDOTBS3.287.977121353
    DATAFILE  MIRROR  COARSE   MAY 26 06:00:00  Y    UNDOTBS4.286.977121353
    DATAFILE  MIRROR  COARSE   MAY 26 06:00:00  Y    USERS.284.977121353
    ASMCMD [+RECO/DBM01] > ls -l CONTROLFILE/
    Type         Redund  Striped  Time             Sys  Name
    CONTROLFILE  HIGH    FINE     MAY 26 06:00:00  Y    Backup.283.977121353
    CONTROLFILE  HIGH    FINE     MAY 26 06:00:00  Y    Backup.285.977120961
    CONTROLFILE  HIGH    FINE     MAY 26 06:00:00  Y    Backup.293.977120965


    Step 6: Switch Database to RMAN backup copy. This command will switch the database from +DATA to +RECO ASM Disk Group.


    [oracle@dm01db01 ~]$ srvctl stop database -d dbm01


    [oracle@dm01db01 ~]$ sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Sat May 26 07:22:06 2018


    Copyright (c) 1982, 2013, Oracle.  All rights reserved.


    Connected to an idle instance.


    SQL> startup mount;
    ORACLE instance started.


    Total System Global Area 2.5655E+10 bytes
    Fixed Size                  2265224 bytes
    Variable Size            4160753528 bytes
    Database Buffers         2.1341E+10 bytes
    Redo Buffers              151113728 bytes
    Database mounted.


    [oracle@dm01db01 ~]$ rman target /


    Recovery Manager: Release 11.2.0.4.0 – Production on Sat May 26 07:23:09 2018


    Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.


    connected to target database: DBM01 (DBID=1180720008, not open)


    RMAN> switch database to copy;


    using target database control file instead of recovery catalog
    datafile 1 switched to datafile copy “+RECO/dbm01/datafile/system.291.977121353”
    datafile 2 switched to datafile copy “+RECO/dbm01/datafile/sysaux.290.977121353”
    datafile 3 switched to datafile copy “+RECO/dbm01/datafile/undotbs1.289.977121353”
    datafile 4 switched to datafile copy “+RECO/dbm01/datafile/undotbs2.288.977121353”
    datafile 5 switched to datafile copy “+RECO/dbm01/datafile/undotbs3.287.977121353”
    datafile 6 switched to datafile copy “+RECO/dbm01/datafile/undotbs4.286.977121353”
    datafile 7 switched to datafile copy “+RECO/dbm01/datafile/users.284.977121353”


    RMAN> recover database;


    Starting recover at 26-MAY-18
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: SID=991 instance=dbm011 device type=DISK


    starting media recovery
    media recovery complete, elapsed time: 00:00:02


    Finished recover at 26-MAY-18


    RMAN> alter database open resetlogs;


    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of alter db command at 05/26/2018 07:25:06
    ORA-01139: RESETLOGS option only valid after an incomplete database recovery


    RMAN> alter database open;


    database opened


    [oracle@dm01db01 ~]$ srvctl stop database -d dbm01


    [oracle@dm01db01 ~]$ srvctl start database -d dbm01


    [oracle@dm01db01 ~]$ srvctl status database -d dbm01
    Instance dbm011 is running on node dm01db01
    Instance dbm012 is running on node dm01db02
    Instance dbm013 is running on node dm01db03
    Instance dbm014 is running on node dm01db04


    [oracle@dm01db01 ~]$ sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Sat May 26 07:28:11 2018


    Copyright (c) 1982, 2013, Oracle.  All rights reserved.


    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options


    SQL> select name, open_mode,database_role from gv$database;


    NAME      OPEN_MODE            DATABASE_ROLE
    ——— ——————– —————-
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY




    Step 7: Move Temp and online redo log files


    SQL> alter database tempfile ‘+DATAC1/dbm01/tempfile/temp.262.976375229’ drop;


    Database altered.


    SQL> alter tablespace TEMP add tempfile ‘+RECO’ SIZE 1024M;


    Tablespace altered.


    SQL> alter database add logfile member ‘+RECO’ to group 1;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 2;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 3;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 4;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 5;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 6;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 7;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 8;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 9;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 10;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 11;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 12;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 13;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 14;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 15;


    Database altered.


    SQL> alter database add logfile member ‘+RECO’ to group 16;


    Database altered.


    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_1.257.976374733’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_2.258.976374735’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_7.267.976375073’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_8.268.976375075’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_5.269.976375079’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_6.270.976375083’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_3.271.976375085’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_4.272.976375087’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_9.274.976375205’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_10.275.976375209’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_11.276.976375211’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_12.277.976375215’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_13.278.976375217’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_14.279.976375219’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_15.280.976375223’;
    SQL> alter database drop logfile member ‘+DATA/dbm01/onlinelog/group_16.281.976375225’;




    Step 8: Move control file to +RECO Disk Group


    [oracle@dm01db01 ~]$ srvctl stop database -d dbm01
    [oracle@dm01db01 ~]$ sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Sat May 26 08:53:35 2018


    Copyright (c) 1982, 2013, Oracle.  All rights reserved.


    Connected to an idle instance.


    SQL> startup nomount;
    ORACLE instance started.


    Total System Global Area 2.5655E+10 bytes
    Fixed Size                  2265224 bytes
    Variable Size            4429188984 bytes
    Database Buffers         2.1072E+10 bytes
    Redo Buffers              151113728 bytes
    SQL> exit
    Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, OLAP, Data Mining
    and Real Application Testing options
    [oracle@dm01db01 ~]$ rman target /


    Recovery Manager: Release 11.2.0.4.0 – Production on Sat May 26 08:53:59 2018


    Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.


    connected to target database: DBM01 (not mounted)


    RMAN> restore controlfile to ‘+RECO’ from ‘+DATA/dbm01/controlfile/current.256.976374731’;


    Starting restore at 26-MAY-18
    using target database control file instead of recovery catalog
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: SID=969 instance=dbm011 device type=DISK


    channel ORA_DISK_1: copied control file copy
    Finished restore at 26-MAY-18


    RMAN> exit


    Recovery Manager complete.


    [oracle@dm01db01 ~]$ . oraenv
    ORACLE_SID = [dbm011] ? +ASM1
    The Oracle base remains unchanged with value /u01/app/oracle


    [oracle@dm01db01 ~]$ asmcmd -p


    ASMCMD [+] > cd +RECO


    ASMCMD [+RECO] > ls -l
    Type  Redund  Striped  Time             Sys  Name
                                            Y    DBM01/
    ASMCMD [+RECO] > cd DBM01
    ASMCMD [+RECO/DBM01] > ls -l
    Type         Redund  Striped  Time             Sys  Name
                                                   Y    ARCHIVELOG/
                                                   Y    AUTOBACKUP/
                                                   Y    CHANGETRACKING/
                                                   Y    CONTROLFILE/
                                                   Y    DATAFILE/
                                                   Y    ONLINELOG/
                                                   Y    TEMPFILE/
                                                   N    snapcf_dbm01.f => +RECO/DBM01/CONTROLFILE/Backup.285.977120961
    ASMCMD [+RECO/DBM01] > cd CONTROLFILE/


    ASMCMD [+RECO/DBM01/CONTROLFILE] > ls -l
    Type         Redund  Striped  Time             Sys  Name
    CONTROLFILE  HIGH    FINE     MAY 26 06:00:00  Y    Backup.283.977121353
    CONTROLFILE  HIGH    FINE     MAY 26 08:00:00  Y    Backup.285.977120961
    CONTROLFILE  HIGH    FINE     MAY 26 06:00:00  Y    Backup.293.977120965
    CONTROLFILE  HIGH    FINE     MAY 26 08:00:00  Y    Backup.321.977128799
    CONTROLFILE  HIGH    FINE     MAY 26 08:00:00  Y    current.331.977129649


    ASMCMD [+RECO/DBM01/CONTROLFILE] > pwd
    +RECO/DBM01/CONTROLFILE


    ASMCMD [+RECO/DBM01/CONTROLFILE] > exit


    [oracle@dm01db01 ~]$ . oraenv
    ORACLE_SID = [+ASM1] ? dbm011
    The Oracle base remains unchanged with value /u01/app/oracle


    [oracle@dm01db01 ~]$ sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Sat May 26 08:55:09 2018


    Copyright (c) 1982, 2013, Oracle.  All rights reserved.


    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options


    SQL> alter system set control_files=’+RECO/DBM01/CONTROLFILE/current.331.977129649′ scope=spfile sid=’*’;


    System altered.


    SQL> shutdown immediate;
    ORA-01507: database not mounted




    ORACLE instance shut down.
    SQL> exit


    [oracle@dm01db01 ~]$ srvctl start database -d dbm01


    [oracle@dm01db01 ~]$ srvctl status database -d dbm01
    Instance dbm011 is running on node dm01db01
    Instance dbm012 is running on node dm01db02
    Instance dbm013 is running on node dm01db03
    Instance dbm014 is running on node dm01db04




    Step 9: Move block change tracking file to +RECO Disk Group


    SQL> select filename from v$block_change_tracking;


    FILENAME
    ——————————————————————–
    +DATA/dbm01/changetracking/ctf.282.976375227


    SQL> alter database disable block change tracking;


    Database altered.


    SQL> alter database enable block change tracking using file ‘+RECO’;


    Database altered.


    SQL> select filename from v$block_change_tracking;


    FILENAME
    ——————————————————————–
    +RECO/dbm01/changetracking/ctf.319.977128195




    Step 10: Move Flash Recovery Area to +RECO Disk Group


    SQL> show parameter recover


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    db_recovery_file_dest                string      +DATA


    SQL> alter system set db_recovery_file_dest=’+RECO’;


    System altered.


    SQL> show parameter db_recovery_file_dest


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    db_recovery_file_dest                string      +RECO




    Step 11: Update OMF parameter to point to +RECO


    SQL> show parameter online


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    db_create_online_log_dest_1          string      +DATA


    SQL> alter system set db_create_online_log_dest_1=’+RECO’;


    System altered.


    SQL> show parameter db_create_online_log_dest_1


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    db_create_online_log_dest_1          string      +RECO




    Step 12: Verify the entire database is moved to +RECO ASM Disk Group


    [oracle@dm01db01 ~]$ sqlplus / as sysdba


    SQL*Plus: Release 11.2.0.4.0 Production on Sat May 26 08:57:57 2018


    Copyright (c) 1982, 2013, Oracle.  All rights reserved.


    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options


    SQL> select name, open_mode,database_role from gv$database;


    NAME      OPEN_MODE            DATABASE_ROLE
    ——— ——————– —————-
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY
    DBM01     READ WRITE           PRIMARY


    SQL> set lines 200
    SQL> set pages 200
    SQL> select name from v$tempfile;


    NAME
    ——————————————————-
    +RECO/dbm01/tempfile/temp.297.977125145


    SQL> select name from v$controlfile;


    NAME
    ——————————————————-
    +RECO/dbm01/controlfile/current.331.977129649


    SQL> select name from v$datafile;


    NAME
    ——————————————————–
    +RECO/dbm01/datafile/system.291.977121353
    +RECO/dbm01/datafile/sysaux.290.977121353
    +RECO/dbm01/datafile/undotbs1.289.977121353
    +RECO/dbm01/datafile/undotbs2.288.977121353
    +RECO/dbm01/datafile/undotbs3.287.977121353
    +RECO/dbm01/datafile/undotbs4.286.977121353
    +RECO/dbm01/datafile/users.284.977121353


    7 rows selected.


    SQL> select member from v$logfile;


    MEMBER
    ———————————————————
    +RECO/dbm01/onlinelog/group_1.298.977127719
    +RECO/dbm01/onlinelog/group_2.299.977125295
    +RECO/dbm01/onlinelog/group_3.300.977125299
    +RECO/dbm01/onlinelog/group_4.301.977125309
    +RECO/dbm01/onlinelog/group_5.302.977125313
    +RECO/dbm01/onlinelog/group_6.303.977125317
    +RECO/dbm01/onlinelog/group_7.304.977125321
    +RECO/dbm01/onlinelog/group_8.305.977125327
    +RECO/dbm01/onlinelog/group_9.306.977125329
    +RECO/dbm01/onlinelog/group_10.307.977125333
    +RECO/dbm01/onlinelog/group_11.308.977125335
    +RECO/dbm01/onlinelog/group_12.309.977125339
    +RECO/dbm01/onlinelog/group_13.310.977125343
    +RECO/dbm01/onlinelog/group_14.311.977125345
    +RECO/dbm01/onlinelog/group_15.312.977125349
    +RECO/dbm01/onlinelog/group_16.313.977125351


    16 rows selected.




    Conclusion


    In this article we have learned how to move a Database from +DATA ASM Disk Group to +RECO Disk Group. Using RMAN along with FRA makes it easy to move a database from one location to another. 

  • Exadata – Display and Clear Fault using Fault Manager (faultmgmt)

    Exadata Database Machine consists of a storage grid, compute grid, and network grid. Each grid, or hardware layer, is built with multiple high-performing, industry-standard Oracle servers to provide hardware and system fault tolerance. The hardware components are subjected to failure. Most common failure on Exadata is Hard Disk failure on Storage Cells. With the latest generation of Exadata the hardware failures are very minimal and less troublesome. 


    The Exadata Storage Cells and Compute nodes consists of several hardware components, such as:

    • Hard disk
    • Flash disk 
    • Physical Memory
    • Processor  
    • IB ports
    • Mother Board
    • Batteries
    • Power Supply
    • and So on





    In this article we will demonstrate how to view the hardware fault and clear it using ILOM fault manager (faultmgmt).




    Steps to display and clear hardware fault using faultmgmt:




    Step 1: Login to compute node ILOM where the fault occurred


    [root@dm01db01 ~]# ssh dm01db02-ilom
    Password:


    Oracle(R) Integrated Lights Out Manager


    Version 4.0.0.24 r121523


    Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.


    Warning: HTTPS certificate is set to factory default.


    Hostname: dm01db02-ilom


    Step 2: Check if the fault manager is supported. If you get the output like below then fault manager is supported.

    -> show /SP/faultmgmt/shell


     /SP/faultmgmt/shell
        Targets:


        Properties:


        Commands:
            cd
            show
            start




    Step 3: Start the fault manager shell


    -> start /SP/faultmgmt/shell
    Are you sure you want to start /SP/faultmgmt/shell (y/n)? y


    Step 4: Execute the following command to display the fault. Here we can see that there is no issue with hardware but he ILOM file system is 100% full.


    faultmgmtsp> fmadm faulty
    ——————- ———————————— ————– ——–
    Time                UUID                                 msgid          Severity
    ——————- ———————————— ————– ——–
    2018-06-17/15:55:32 2a854ad2-4a31-e829-e26c-c84ba212d7f2 ILOM-8000-JV   Major


    Problem Status           : open
    Diag Engine              : fdd 1.0
    System
       Manufacturer          : Oracle Corporation
       Name                  : Exadata X5-2
       Part_Number           : Exadata X5-2
       Serial_Number         : AK00XXXXXX


    System Component
       Manufacturer          : Oracle Corporation
       Name                  : ORACLE SERVER X5-2
       Part_Number           : 7090664
       Serial_Number         : 15XXXXXXXX
       Firmware_Manufacturer : Oracle Corporation
       Firmware_Version      : (ILOM)4.0.0.24
       Firmware_Release      : (ILOM)2017.09.23


    —————————————-
    Suspect 1 of 1
       Problem class  : defect.ilom.fs.full
       Certainty      : 100%
       Affects        : /SYS/SP
       Status         : faulted


       FRU
          Status            : faulty
          Location          : /SYS/SP
          Manufacturer      : Oracle Corporation
          Name              : SP
          Part_Number       : PILOT3
          Chassis
             Manufacturer   : Oracle Corporation
             Name           : ORACLE SERVER X5-2
             Part_Number    : 7090664
             Serial_Number  : 1547NM10CX


    Description : An ILOM filesystem has exceeded the filesystem capacity
                  limit.


    Response    : The chassis wide service-required LED will be illuminated.


    Impact      : ILOM commands may fail, especially those which make
                  configuration changes.


    Action      : Please refer to the associated reference document at
                  http://support.oracle.com/msg/ILOM-8000-JV for the latest
                  service procedures and policies regarding this diagnosis.


    Step 5: Execute the below command to clear the fault


    faultmgmtsp> fmadm acquit UUID –> Get the UUID from the from output of the above command.


    faultmgmtsp> fmadm acquit 2a854ad2-4a31-e829-e26c-c84ba212d7f2


    Step 6: Verify that the fault is cleared


    faultmgmtsp> fmadm faulty
    No faults found


    Step 7: Exit from the fault manager


    faultmgmtsp> exit


    Step 8: Reset the ILOM service processor


    -> reset /SP
    Are you sure you want to reset /SP (y/n)? y
    Performing reset on /SP


    Step 9: Exit from the ILOM


    -> exit
    Connection to dm01db02-ilom closed.


    Step 10: Connec to ILOM and verify the ILOM SP is restarted


    [root@dm01db01 ~]# ssh dm01db02-ilom
    Password:


    Oracle(R) Integrated Lights Out Manager


    Version 4.0.0.24 r121523


    Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.


    Warning: HTTPS certificate is set to factory default.


    Hostname: dm01db02-ilom




    -> show -d properties /SP/clock uptime


     /SP/clock
        Properties:
            uptime = 0 days, 00:08:02




    Conclusion


    In this article we have learned how to display and clear a fault using fault manager (faultmgmt). The Fault Management Shell is the preferred method for displaying the details of a diagnosed fault. faultmgmt support for command shell varies depending ILOM release level and server product model. 

  • Extend or Increase root File System Size on Exadata

    When Exadata database machine is installed by Oracle ACS the root file system size is set to 30GB. This space may not be sufficient for storing large files, logfiles, patches and so on and it can be filled very quickly. So you must consider increasing the root file system to avoid space issues. The root file system is built on volume group which makes it easy to resize the logical volume on which the root file system is mounted.


    Root file system is created on two system partitions LVDbSys1 and LVDbSys2 and both system partitions must be size equally at the same time. Only one system partition is active at any time and other is inactive.


    In this article, we will demonstrate how you can extend/increase the root file system size on Exadata Compute node. This activity can be done online without any downtime if the file system feature supports it.


    Steps to extend/increase the root file system on Exadata Compute node


    Step 1: Get the current root file system size and utilization. Here we can see that the root file system was expanded early to 60GB and currently it is 100% used.


    [root@dm01db01 /]# df -h /
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VGExaDb-LVDbSys1
                           59G   57G     0 100% /


    Step 2: Get the logical volumne details


    [root@dm01db01 /]# lvs -o lv_name,lv_path,vg_name,lv_size
      LV                 Path                            VG      LSize
      LVDbOra1           /dev/VGExaDb/LVDbOra1           VGExaDb 200.00g
      LVDbSwap1          /dev/VGExaDb/LVDbSwap1          VGExaDb  24.00g
      LVDbSys1           /dev/VGExaDb/LVDbSys1           VGExaDb  60.00g
      LVDbSys2           /dev/VGExaDb/LVDbSys2           VGExaDb  60.00g
      LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb   1.00g


    Step 3: Check to make sure that root file system can be resized online. Execute the following to determine it. If you get an output the file system can be resized online

    [root@dm01db01 /]# tune2fs -l /dev/mapper/VGExaDb-LVDbSys1 | grep resize_inode
    Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize


    [root@dm01db01 /]# dcli -g ~/dbs_group -l root ‘tune2fs -l /dev/mapper/VGExaDb-LVDbSys1 | grep resize_inode’
    dm01db01: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
    dm01db02: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
    dm01db03: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
    dm01db04: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize


    Step 4: Get the current active partition information. Here the current active partition is LVDbSys1


    [root@dm01db01 ~]# imageinfo


    Kernel version: 4.1.12-94.7.8.el6uek.x86_64 #2 SMP Thu Jan 11 20:41:01 PST 2018 x86_64
    Image kernel version: 4.1.12-94.7.8.el6uek
    Image version: 12.2.1.1.6.180125.1
    Image activated: 2018-04-13 22:11:49 -0500
    Image status: success
    System partition on device: /dev/mapper/VGExaDb-LVDbSys1




    Step 5: Get the free space available in the Volume Group. Currently we have around 1.3TB free space available. So we can easily increase the root file system.


    [root@dm01db01 /]# vgdisplay -s
      “VGExaDb” 1.63 TiB  [345.00 GiB used / 1.30 TiB free]


    Step 6: Using lvextend command increase the both logical volumes. In our case, we are increasing the root file system by 30GB to make 90GB total size.


    [root@dm01db01 /]# lvextend -L +30G /dev/VGExaDb/LVDbSys1
      Size of logical volume VGExaDb/LVDbSys1 changed from 60.00 GiB (15360 extents) to 90.00 GiB (23040 extents).
      Logical volume LVDbSys1 successfully resized.


    [root@dm01db01 /]# lvextend -L +30G /dev/VGExaDb/LVDbSys2
      Size of logical volume VGExaDb/LVDbSys2 changed from 60.00 GiB (15360 extents) to 90.00 GiB (23040 extents).
      Logical volume LVDbSys2 successfully resized.


    Step 7: Using resize2fs command resize the file system as follows


    [root@dm01db01 /]# resize2fs /dev/VGExaDb/LVDbSys1
    resize2fs 1.43-WIP (20-Jun-2013)
    Filesystem at /dev/VGExaDb/LVDbSys1 is mounted on /; on-line resizing required
    old_desc_blocks = 4, new_desc_blocks = 6
    The filesystem on /dev/VGExaDb/LVDbSys1 is now 23592960 blocks long.


    [root@dm01db01 /]# e2fsck -f /dev/VGExaDb/LVDbSys1
    e2fsck 1.43-WIP (20-Jun-2013)
    /dev/VGExaDb/LVDbSys1 is mounted.
    e2fsck: Cannot continue, aborting.




    [root@dm01db01 /]# resize2fs /dev/VGExaDb/LVDbSys2
    resize2fs 1.43-WIP (20-Jun-2013)
    Please run ‘e2fsck -f /dev/VGExaDb/LVDbSys2’ first.


    [root@dm01db01 /]# e2fsck -f /dev/VGExaDb/LVDbSys2
    e2fsck 1.43-WIP (20-Jun-2013)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/VGExaDb/LVDbSys2: 167407/3932160 files (0.1% non-contiguous), 4710754/15728640 blocks


    [root@dm01db01 /]# resize2fs /dev/VGExaDb/LVDbSys2
    resize2fs 1.43-WIP (20-Jun-2013)
    Resizing the filesystem on /dev/VGExaDb/LVDbSys2 to 23592960 (4k) blocks.
    The filesystem on /dev/VGExaDb/LVDbSys2 is now 23592960 blocks long.


    Note: You can’t run the e2fsck for the active paritition LVDbSys1. You run the e2fsck for inactive partition LVDbSys2 first before resizing it.


    Step 8: Verify the new file system size


    [root@dm01db01 /]# df -h /
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VGExaDb-LVDbSys1
                           89G   57G   29G  67% /


    Repeat the steps above on all the compute nodes in the Exadata Rack.




    Conclusion


    In this article we have learned how to extend/increase the root file system on Exadata Compute node online without any outage. Root file system is created on two system partitions LVDbSys1 and LVDbSys2 and both system partitions must be size equally at the same time. Only one system partition is active at any time and other is inactive.