Tag: Exadata

  • Patching Exadata Infiniband Switch

    Overview
    • The Exadata network grid consists of multiple Sun QDR InfiniBand switches.
    • IB Switches are used for the storage network as well as the Oracle RAC interconnect.
    • Exadata compute nodes and storage cells are configured with dual-port InfiniBand ports and connect to each of the two leaf switches.
    • You can access IB Switches using command line and Web ILOM
    • IB Switches run Linux operating system.


    In this article I will demonstrate how to patch or upgrade Oracle Exadata IB Switches.

    About Infiniband Switch Patching

    • Starting with release 11.2.3.3.0, the patchmgr utility is used to upgrade and downgrade the InfiniBand switches.
    • IB Switch patch is delievered with Exadata storage patch.
    • IB Switch patches are released semi annually to annually.
    • IB Switch can be patched in Rolling and Non-Rolling fashion.

    Environment
    • Exadata Full Rack X2-2
    • 8 Compute nodes, 14 Storage cells and 3 IB Switches
    • Current IB Switch Version 2.1.5-1


    Infiniband Switch Upgrade Steps

    • Identify the number of switches in clusters.

    [root@dm01dbadm01 ~]# ibswitches
    Switch  : 0x002128469b8aa0a0 ports 36 “SUN DCS 36P QDR dm01sw-ib1 10.209.41.246” enhanced port 0 lid 5 lmc 0
    Switch  : 0x002128469b97a0a0 ports 36 “SUN DCS 36P QDR dm01sw-ib2 10.209.41.247” enhanced port 0 lid 4 lmc 0
    Switch  : 0x002128469b82a0a0 ports 36 “SUN DCS 36P QDR dm01sw-ib3 10.209.41.248” enhanced port 0 lid 3 lmc 0

    • Identify the current IB switch software version on all the Switches

    [root@dm01dbadm01 patch_12.1.2.2.0.150917]# ssh dm01sw-ib1 version
    SUN DCS 36p version: 2.1.5-1
    Build time: Oct  6 2014 10:35:15
    SP board info:
    Manufacturing Date: 2011.01.12
    Serial Number: “NCD680421”
    Hardware Revision: 0x0006
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010

    • Log in to Exadata Compute node 1 as root user and navigate the Exadata Storage Software staging area

    [root@dm01dbadm01 ESS_121220]# cd /u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917/
    [root@dm01dbadm01 patch_12.1.2.2.0.150917]# pwd
    /u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917

    • Create a file named ibswitches.lst and enter IB switch names one per line as follows:

    [root@dm01dbadm01 patch_12.1.2.2.0.150917]# vi ibswitches.lst
    dm01sw-ib1
    dm01sw-ib2
    dm01sw-ib3

    [root@dm01dbadm01 patch_12.1.2.2.0.150917]# cat ibswitches.lst
    dm01sw-ib1
    dm01sw-ib2
    dm01sw-ib3

    • Execute the following to perform the IB Switch precheck

    [root@dm01dbadm01 patch_12.1.2.2.0.150917]# ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck -force

    2015-11-04 11:31:44 -0600 1 of 1 :SUCCESS: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
     —– InfiniBand switch update process started 2015-11-04 11:31:44 -0600 —–
    [NOTE     ] Log file at /u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917/upgradeIBSwitch.log

    [INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ib1 dm01sw-ib2 dm01sw-ib3 )

    [SUCCESS  ] Verifying Network connectivity to dm01sw-ib1
    [SUCCESS  ] Verifying Network connectivity to dm01sw-ib2
    [SUCCESS  ] Verifying Network connectivity to dm01sw-ib3
    [FAIL     ] Validating verify-topology output
    [INFO     ] Master Subnet Manager is set to “dm01sw-ib3” in all Switches

    [INFO     ] ———- Starting with IBSwitch dm01sw-ib1
    [WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
         To rollback to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
         – Run patchmgr command to initiate rollback.
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib1, found 137M
    [SUCCESS  ] Verifying that / has 80M in dm01sw-ib1, found 197M
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib1
    [SUCCESS  ] Verifying that dm01sw-ib1 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:41:20
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib1
    [SUCCESS  ] Pre-update validation on dm01sw-ib1

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib2
    [WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
         To rollback to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
         – Run patchmgr command to initiate rollback.
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib2, found 249M
    [SUCCESS  ] Verifying that / has 80M in dm01sw-ib2, found 199M
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib2
    [SUCCESS  ] Verifying that dm01sw-ib2 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:41:33
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib2
    [SUCCESS  ] Pre-update validation on dm01sw-ib2

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib3
    [WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
         To rollback to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
         – Run patchmgr command to initiate rollback.
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib3, found 248M
    [SUCCESS  ] Verifying that / has 80M in dm01sw-ib3, found 196M
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib3
    [SUCCESS  ] Verifying that dm01sw-ib3 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:41:42
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib3
    [SUCCESS  ] Pre-update validation on dm01sw-ib3
    [SUCCESS  ] Overall status

     —– InfiniBand switch update process ended 2015-11-04 11:41:42 -0600 —–
    2015-11-04 11:41:42 -0600 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).

    • Upgrade the IB Switches using the following command:

    [root@dm01dbadm01 patch_12.1.2.2.0.150917]# ./patchmgr -ibswitches ibswitches.lst -upgrade -force

    2015-11-04 11:42:51 -0600 1 of 1 :SUCCESS: DO: Initiate upgrade of InfiniBand switches to 2.1.6-2. Expect up to 15 minutes for each switch
     —– InfiniBand switch update process started 2015-11-04 11:42:51 -0600 —–
    [NOTE     ] Log file at /u01/app/oracle/software/ESS_121220/patch_12.1.2.2.0.150917/upgradeIBSwitch.log

    [INFO     ] List of InfiniBand switches for upgrade: ( dm01sw-ib1 dm01sw-ib2 dm01sw-ib3 )

    [SUCCESS  ] Verifying Network connectivity to dm01sw-ib1
    [SUCCESS  ] Verifying Network connectivity to dm01sw-ib2
    [SUCCESS  ] Verifying Network connectivity to dm01sw-ib3
    [FAIL     ] Validating verify-topology output
    [INFO     ] Proceeding with upgrade of InfiniBand switches to version 2.1.6_2
    [INFO     ] Master Subnet Manager is set to “aibdalab1sw-ib3” in all Switches

    [INFO     ] ———- Starting with IBSwitch dm01sw-ib1
    [WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
         To rollback to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
         – Run patchmgr command to initiate rollback.
    [SUCCESS  ] Copy firmware packages to dm01sw-ib1
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib1, found 137M
    [SUCCESS  ] Verifying that / has 80M in dm01sw-ib1, found 197M
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib1
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib1
    [SUCCESS  ] Verifying that dm01sw-ib1 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 17:44:09
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib1
    [SUCCESS  ] Pre-update validation on dm01sw-ib1
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ib1
    [INFO     ] Starting upgrade on dm01sw-ib1 to 2.1.6_2. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [SUCCESS  ] Execute plugin check for Patching on dm01sw-ib1
    [SUCCESS  ] Load firmware 2.1.6_2 onto dm01sw-ib1
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ib1
    [SUCCESS  ] Set SMPriority to 8 on dm01sw-ib1
    [INFO     ] Rebooting dm01sw-ib1. Wait for 240 secs before continuing
    [SUCCESS  ] Reboot dm01sw-ib1
    [SUCCESS  ] SUCCESS
    [INFO     ] Starting post-update validation on dm01sw-ib1
    [SUCCESS  ] Inifiniband switch dm01sw-ib1 is at target patching level
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib1
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib1
    [SUCCESS  ] Verifying that dm01sw-ib1 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:01:21
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ib1
    [SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ib1
    [SUCCESS  ] Post-check validation on IBSwitch dm01sw-ib1
    [SUCCESS  ] Update switch dm01sw-ib1 to 2.1.6_2

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib2
    [WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
         To rollback to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
         – Run patchmgr command to initiate rollback.
    [SUCCESS  ] Copy firmware packages to dm01sw-ib2
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib2, found 138M
    [SUCCESS  ] Verifying that / has 80M in dm01sw-ib2, found 199M
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib2
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib2
    [SUCCESS  ] Verifying that dm01sw-ib2 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:06:23
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib2
    [SUCCESS  ] Pre-update validation on dm01sw-ib2
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ib2
    [INFO     ] Starting upgrade on dm01sw-ib2 to 2.1.6_2. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [SUCCESS  ] Execute plugin check for Patching on dm01sw-ib2
    [SUCCESS  ] Load firmware 2.1.6_2 onto dm01sw-ib2
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ib2
    [SUCCESS  ] Set SMPriority to 2 on dm01sw-ib2
    [INFO     ] Rebooting dm01sw-ib2. Wait for 240 secs before continuing
    [SUCCESS  ] Reboot dm01sw-ib2
    [SUCCESS  ] SUCCESS
    [INFO     ] Starting post-update validation on dm01sw-ib2
    [SUCCESS  ] Inifiniband switch dm01sw-ib2 is at target patching level
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib2
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib2
    [SUCCESS  ] Verifying that dm01sw-ib2 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:23:30
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ib2
    [SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ib2
    [SUCCESS  ] Post-check validation on IBSwitch dm01sw-ib2
    [SUCCESS  ] Update switch dm01sw-ib2 to 2.1.6_2

    [INFO     ] ———- Starting with InfiniBand Switch dm01sw-ib3
    [WARNING  ] Infiniband switch meets minimal version requirements, but rollback is only available to 2.1.5-1 with the current package.
         To rollback to other versions:
         – Manually download the InfiniBand switch firmware package to the patch directory
         – Set export variable “EXADATA_IMAGE_IBSWITCH_ROLLBACK_VERSION” to the appropriate version
         – Run patchmgr command to initiate rollback.
    [SUCCESS  ] Copy firmware packages to dm01sw-ib3
    [SUCCESS  ] Verifying that /tmp has 120M in dm01sw-ib3, found 137M
    [SUCCESS  ] Verifying that / has 80M in dm01sw-ib3, found 196M
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib3
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib3
    [SUCCESS  ] Verifying that dm01sw-ib3 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:28:26
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on dm01sw-ib3
    [SUCCESS  ] Pre-update validation on dm01sw-ib3
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ib3
    [INFO     ] Starting upgrade on dm01sw-ib3 to 2.1.6_2. Please give upto 10 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [SUCCESS  ] Execute plugin check for Patching on dm01sw-ib3
    [SUCCESS  ] Load firmware 2.1.6_2 onto dm01sw-ib3
    [SUCCESS  ] Disable Subnet Manager on dm01sw-ib3
    [SUCCESS  ] Set SMPriority to 2 on dm01sw-ib3
    [INFO     ] Rebooting dm01sw-ib3. Wait for 240 secs before continuing
    [SUCCESS  ] Reboot dm01sw-ib3
    [SUCCESS  ] SUCCESS
    [INFO     ] Starting post-update validation on dm01sw-ib3
    [SUCCESS  ] Inifiniband switch dm01sw-ib3 is at target patching level
    [SUCCESS  ] Service ntpd is running on InfiniBand Switch dm01sw-ib3
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch dm01sw-ib3
    [SUCCESS  ] Verifying that dm01sw-ib3 has at least 1 NTP Server, found 1
    [SUCCESS  ] opensm.conf passed all validations
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2015-11-04 Time:(HH:MM:SS) 18:46:04
    [SUCCESS  ] Firmware verification on InfiniBand switch dm01sw-ib3
    [SUCCESS  ] Execute plugin check for Post Patch on dm01sw-ib3
    [SUCCESS  ] Post-check validation on IBSwitch dm01sw-ib3
    [SUCCESS  ] Update switch dm01sw-ib3 to 2.1.6_2
    [INFO     ] InfiniBand Switches ( dm01sw-ib1 dm01sw-ib2 dm01sw-ib3 ) updated to 2.1.6_2
    [SUCCESS  ] Overall status

     —– InfiniBand switch update process ended 2015-11-04 12:50:22 -0600 —–
    2015-11-04 12:50:22 -0600 1 of 1 :SUCCESS: DONE: Upgrade InfiniBand switch(es) to 2.1.6-2.

    • Verify that all the IB Switches are upgraded to latest version.

    [root@dm01dbadm01 ~]# ssh dm01sw-ib1 version
    SUN DCS 36p version: 2.1.6-2
    Build time: Dec  8 2014 10:36:50
    SP board info:
    Manufacturing Date: 2011.01.12
    Serial Number: “NCD680421”
    Hardware Revision: 0x0006
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010
    [root@dm01dbadm01 ~]#

     

    Conclusion
    In this article we have demonstrated how to patch Exadata IB Switches using patchmgr utility. Patching an Exadata IB switch is very straight forward and can be done in rolling and non-rolling fashion.

  • Steps to Shutdown Exadata Database Machine

    Overview
    When you perform power maintenance, data center maintenance or moving servers from one data center to another you need to shutdown the Clusterware/Database/Servers. If you have Exadata Database Machines sitting in your data center then you need to shutdown it as well. The process of shutting down Exadata Database machine is slightly different when compared to servers or cluster. We should follow a sequence of steps to shutdown Exadata components. 


    In this article, we will demonstrate the steps to shutdown Exadata Database Machine to carry out maintenance task.


    Environment
    Exadata X5-2 Full Rack


    • Log in to the first Exadata compute node as as root user.
    • Change to root home directory and create the following files

    [root@dm01db01 ~]# cd /root


    [root@dm01db01 ~]# vi dbs_group
    dm01db01
    dm01db02
    dm01db03
    dm01db04
    dm01db05
    dm01db06
    dm01db07
    dm01db08


    [root@dm01db01 ~]# vi cell_group
    dm01cel01
    dm01cel02
    dm01cel03
    dm01cel04
    dm01cel05
    dm01cel06
    dm01cel07
    dm01cel08
    dm01cel09
    dm01cel10
    dm01cel11
    dm01cel12
    dm01cel13
    dm01cel14

    • Check if Grid Infrastructure is enabled for autostart:

    [root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl config crs
    dm01db01: CRS-4622: Oracle High Availability Services autostart is enabled.
    dm01db02: CRS-4622: Oracle High Availability Services autostart is enabled.
    dm01db03: CRS-4622: Oracle High Availability Services autostart is enabled.
    dm01db04: CRS-4622: Oracle High Availability Services autostart is enabled.
    dm01db05: CRS-4622: Oracle High Availability Services autostart is enabled.
    dm01db06: CRS-4622: Oracle High Availability Services autostart is enabled.
    dm01db07: CRS-4622: Oracle High Availability Services autostart is enabled.
    dm01db08: CRS-4622: Oracle High Availability Services autostart is enabled.

    • Disable the Grid Infrastructure for autostart on the compute nodes if it is currently enabled for autostart.

    [root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl disable crs
    dm01db01: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db02: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db03: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db04: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db05: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db06: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db07: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db08: CRS-4621: Oracle High Availability Services autostart is disabled.


    [root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl config crs
    dm01db01: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db02: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db03: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db04: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db05: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db06: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db07: CRS-4621: Oracle High Availability Services autostart is disabled.
    dm01db08: CRS-4621: Oracle High Availability Services autostart is disabled.

    • Shutdown the Grid Infrastructure stack on the Exadata compute nodes

    This will take some time to come out
    [root@dm01db01 ~]#dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl stop crs


    [root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl check cluster
    dm01db01: CRS-4639: Could not contact Oracle High Availability Services
    dm01db01: CRS-4000: Command Check failed, or completed with errors.
    dm01db02: CRS-4639: Could not contact Oracle High Availability Services
    dm01db02: CRS-4000: Command Check failed, or completed with errors.
    dm01db03: CRS-4639: Could not contact Oracle High Availability Services
    dm01db03: CRS-4000: Command Check failed, or completed with errors.
    dm01db04: CRS-4639: Could not contact Oracle High Availability Services
    dm01db04: CRS-4000: Command Check failed, or completed with errors.
    dm01db05: CRS-4639: Could not contact Oracle High Availability Services
    dm01db05: CRS-4000: Command Check failed, or completed with errors.
    dm01db06: CRS-4639: Could not contact Oracle High Availability Services
    dm01db06: CRS-4000: Command Check failed, or completed with errors.
    dm01db07: CRS-4639: Could not contact Oracle High Availability Services
    dm01db07: CRS-4000: Command Check failed, or completed with errors.
    dm01db08: CRS-4639: Could not contact Oracle High Availability Services
    dm01db08: CRS-4000: Command Check failed, or completed with errors.

    • Ensure that the Grid Infrastructure stack has shutdown successfully on all the Compute nodes. 

    The following command returns no output if the Grid Infrastructure is shutdown:


    [root@dm01db01 ~]# dcli -g dbs_group -l root “ps -ef | grep diskmo[n]”

    • Disable email/SNMP alerts. The following command will show the current notification method:

    [root@dm01db01 ~]# dcli -g cell_group -l root “cellcli -e list cell attributes name,notificationMethod”
    dm01cel01: dm01cel01     none
    dm01cel02: dm01cel02     none
    dm01cel03: dm01cel03     none
    dm01cel04: dm01cel04     none
    dm01cel05: dm01cel05     none
    dm01cel06: dm01cel06     none
    dm01cel07: dm01cel07     none
    dm01cel08: dm01cel08     none
    dm01cel09: dm01cel09     none
    dm01cel10: dm01cel10     none
    dm01cel11: dm01cel11     none
    dm01cel12: dm01cel12     none
    dm01cel13: dm01cel13     none
    dm01cel14: dm01cel14     none

    • Now set the notification method to null to disable alerts:

    [root@dm01db01 ~]# dcli -g cell_group -l root “cellcli -e alter cell notificationMethod=none”
    dm01cel01: Cell dm01cel01 successfully altered
    dm01cel02: Cell dm01cel02 successfully altered
    dm01cel03: Cell dm01cel03 successfully altered
    dm01cel04: Cell dm01cel04 successfully altered
    dm01cel05: Cell dm01cel05 successfully altered
    dm01cel06: Cell dm01cel06 successfully altered
    dm01cel07: Cell dm01cel07 successfully altered
    dm01cel08: Cell dm01cel08 successfully altered
    dm01cel09: Cell dm01cel09 successfully altered
    dm01cel10: Cell dm01cel10 successfully altered
    dm01cel11: Cell dm01cel11 successfully altered
    dm01cel12: Cell dm01cel12 successfully altered
    dm01cel13: Cell dm01cel13 successfully altered
    dm01cel14: Cell dm01cel14 successfully altered

    • Shutdown cell services on all the cells:

    [root@dm01db01 ~]# dcli -g cell_group -l root “cellcli -e alter cell shutdown services all”
    dm01cel01:
    dm01cel01: Stopping the RS, CELLSRV, and MS services…
    dm01cel01: The SHUTDOWN of services was successful.
    dm01cel02:
    dm01cel02: Stopping the RS, CELLSRV, and MS services…
    dm01cel02: The SHUTDOWN of services was successful.
    dm01cel03:
    dm01cel03: Stopping the RS, CELLSRV, and MS services…
    dm01cel03: The SHUTDOWN of services was successful.
    dm01cel04:
    dm01cel04: Stopping the RS, CELLSRV, and MS services…
    dm01cel04: The SHUTDOWN of services was successful.
    dm01cel05:
    dm01cel05: Stopping the RS, CELLSRV, and MS services…
    dm01cel05: The SHUTDOWN of services was successful.
    dm01cel06:
    dm01cel06: Stopping the RS, CELLSRV, and MS services…
    dm01cel06: The SHUTDOWN of services was successful.
    dm01cel07:
    dm01cel07: Stopping the RS, CELLSRV, and MS services…
    dm01cel07: The SHUTDOWN of services was successful.
    dm01cel08:
    dm01cel08: Stopping the RS, CELLSRV, and MS services…
    dm01cel08: The SHUTDOWN of services was successful.
    dm01cel09:
    dm01cel09: Stopping the RS, CELLSRV, and MS services…
    dm01cel09: The SHUTDOWN of services was successful.
    dm01cel10:
    dm01cel10: Stopping the RS, CELLSRV, and MS services…
    dm01cel10: The SHUTDOWN of services was successful.
    dm01cel11:
    dm01cel11: Stopping the RS, CELLSRV, and MS services…
    dm01cel11: The SHUTDOWN of services was successful.
    dm01cel12:
    dm01cel12: Stopping the RS, CELLSRV, and MS services…
    dm01cel12: The SHUTDOWN of services was successful.
    dm01cel13:
    dm01cel13: Stopping the RS, CELLSRV, and MS services…
    dm01cel13: The SHUTDOWN of services was successful.
    dm01cel14:
    dm01cel14: Stopping the RS, CELLSRV, and MS services…
    dm01cel14: The SHUTDOWN of services was successful.
    [root@dm01db01 ~]#

    • Power off all storage cells:

    [root@dm01db01 ~]# dcli -g cell_group -l root poweroff
    or
    [root@dm01db01 ~]# dcli -g cell_group -l root ‘shutdown -h now’

    • Copy the /root/dbs_group to /root/dbs_group1:

    [root@dm01db01 ~]# cp /root/dbs_group /root/dbs_group1

    • Remove the “first compute node name” from the /root/dbs_group1 file (using any editor, e.g. vi).

    [root@dm01db01 ~]# vi /root/dbs_group1


    [root@dm01db01 ~]# cat /root/dbs_group1
    dm01db02
    dm01db03
    dm01db04
    dm01db05
    dm01db06
    dm01db07
    dm01db08

    • Power off all the compute nodes (except compute node # 1):

    [root@dm01db01 ~]# dcli -g dbs_group1 -l root poweroff
    or
    [root@dm01db01 ~]# dcli -g dbs_group1 -l root ‘shutdown -h now’

    • Power off the compute node #1:

    [root@dm01db01 ~]# poweroff


    Broadcast message from root@dm01db01.netsoftmate.com
            (/dev/pts/0) at 7:36 …


    The system is going down for power off NOW!

    • Exadata components are shutdown for maintenance.



    Conclusion
    In this article we have learnt how to Shutdown Exadata Database machine Rack for maintenance.

  • Install EM Agent On Exadata Using Oracle Enterprise Manager 13c

    Overview

    Oracle Exadata Database Machine consists of several components
    such as Compute nodes, Storage Cells, ILOM, Infiniband Switches, Cisco Switch
    and PDUs. So we need a tool that can manage all these components from one
    single console. Oracle Enterprise manager Cloud Control is the recommended best
    practices for monitoring and managing Exadata Database machine. Once the
    Exadata database Machine is installed the next step is enable monitoring for it.

    The following Exadata components can be monitored and managed by
    OEM:
    • Compute Nodes
    • Storage Cells
    • Infiniband Switches
    • Cisco Switches
    • Power Distribution Units
    • KVM

    The first step in monitoring and managing Exadata using OEM is to
    install the EM Agent. You can install EM Agent in several ways, such as:
    • Using EM Kit Method
    • Using the Agent push method
    • Using an RPM file
    • Using the AgentPull script
    • Using the AgentDeploy script

    In this article we will demonstrate how to Install EM Agent on
    Exadata using Agent Push method using OEM 13c. Agent Software is installed only
    on Compute nodes.

    Environment Details

    Here we will be installing EM Agent on a Exadata V2 Full Rack
    consists of:
    • 8 Compute Nodes
    • 14 Storage Cells
    • 3 Infiniband Switches
    • 1 Cisco Switches
    • 2 Power Distribution Units
    • 1 KVM

    Procedure to Install EM Agent on Exadata Compute nodes

    Enter the OEM 13c URL into the web browser and
    hit enter

    Enter the SYSMAN Credentials or any other user
    that have necessary permissions to Install Agent software

    We are on the OEM 13c Home page now

    From the Home page, click on Setup à
    Add Target à
    Add Targets manually

    Click on Install Agent on Host

    On this page, click on the +Add button to add the
    Host targets

    Enter all fully qualified Hostnames (8 compute
    nodes) and for Platform “Same for All Hosts” and click next

    Enter the Installation Base Directory.
    The instance base directory will be
    populated automatically for you.
    Click on the + symbol on the Named
    credentials and enter oracle user and its password.

    Click on the + symbol on the Root credentials line and
    enter root user and its password.



    Click next

    Click Deploy Agent

    The Agent Installation process started

    Initialization in progress

    The Remote prerequisite check shows warning.
    This can be ignored as oracle user doesn’t have sudo permissions to run the
    root.sh script. The can be ran at the end.

    Click Continue à Continue, All Hosts

    Installation process completed. Click Done.

    Connect to Exadata compute node 1 and run the root.sh
    script as follows:

    [root@dm01db01 ~]# dcli -g dbs_group -l root
    ‘/u01/app/oracle/product/Agent13c/agent_13.2.0.0.0/root.sh’
    dm01db01: Finished product-specific root actions.
    dm01db01: /etc exist
    dm01db02: Finished product-specific root actions.
    dm01db02: /etc exist
    dm01db03: Finished product-specific root actions.
    dm01db03: /etc exist
    dm01db04: Finished product-specific root actions.
    dm01db04: /etc exist
    dm01db05: Finished product-specific root actions.
    dm01db05: /etc exist
    dm01db06: Finished product-specific root actions.
    dm01db06: /etc exist
    dm01db07: Finished product-specific root actions.
    dm01db07: /etc exist
    dm01db08: Finished product-specific root actions.
    dm01db08: /etc exist

    Verify Agent is running on Exadata compute nodes.

    dm01db01-orcl1
    {/home/oracle}:/u01/app/oracle/product/Agent13c/agent_inst/bin/emctl status
    agent
    Oracle Enterprise Manager Cloud Control 13c Release 2
    Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
    —————————————————————
    Agent Version         
    : 13.2.0.0.0
    OMS Version           
    : 13.2.0.0.0
    Protocol Version      
    : 12.1.0.1.0
    Agent Home            
    : /u01/app/oracle/product/Agent13c/agent_inst
    Agent Log Directory   
    : /u01/app/oracle/product/Agent13c/agent_inst/sysman/log
    Agent Binaries        
    : /u01/app/oracle/product/Agent13c/agent_13.2.0.0.0
    Core JAR Location     
    : /u01/app/oracle/product/Agent13c/agent_13.2.0.0.0/jlib
    Agent Process ID      
    : 4544
    Parent Process ID     
    : 4424
    Agent URL             
    : https://dm01db01.int.thomsonreuters.com:3872/emd/main/
    Local Agent URL in NAT : https://dm01db01.int.thomsonreuters.com:3872/emd/main/
    Repository URL        
    : https://oem13c.domain.com:1159/empbs/upload
    Started at             : 2016-12-29 04:40:07
    Started by user       
    : oracle
    Operating System      
    : Linux version 2.6.39-400.248.3.el6uek.x86_64 (amd64)
    Number of Targets     
    : 3
    Last Reload           
    : (none)
    Last successful upload                       : 2016-12-29 04:44:54
    Last attempted upload                        : 2016-12-29 04:44:54
    Total Megabytes of XML files uploaded so far : 0.33
    Number of XML files pending upload           : 0
    Size of XML files pending upload(MB)         : 0
    Available disk space on upload filesystem    : 17.84%
    Collection Status                            : Collections
    enabled
    Heartbeat Status                             : Ok
    Last attempted heartbeat to OMS              : 2016-12-29 04:47:11
    Last successful heartbeat to OMS             : 2016-12-29 04:47:11
    Next scheduled heartbeat to OMS              : 2016-12-29 04:48:12
    —————————————————————
    Agent is Running and Ready

    Verify that Exadata computes are visible in OEM 13c


    Next we will discover Exadata Database Machine in OEM 13c
  • Deinstall Oracle Homes On Exadata Database Machine

    Overview
    After
    Oracle Platinum upgrades Oracle software for example from 11.2.0.3 to 11.2.0.4,
    the OLD Oracle Homes are not removed immediately. There are 2 different types
    of Oracle Home, Grid Infrastructure (ASM + Clusterware) home and RDBMS
    (Database software) home.
    In this article I will
    demonstrate how to De-install Oracle 11gR2 and 12cR1 Homes on Exadata database
    machine.
    Why De-install Oracle Software?
    One should consider de-installing
    Oracle software for the following reasons:
    • To save disk
      space
    • To ease
      management complexities
    • To avoid
      accidentally using unused home

    About De-install Utility
    The de-install utility works
    different for Oracle Database 11g and 12c.
    Starting with Oracle database
    11.2.0.1, Oracle recommends to use de-install utility to de-install Oracle 11.2
    software.

    You can de-install Oracle database 11gR2 software in 2
    ways:
    • External De-install
      Utility – highly recommended method
    • runInstaller

    In Oracle Database 12c, deinstall
    utility is now integrated with the Database Installation Media. You can run the
    deinstall utility using the runInstaller command on Linux and UNIX. In 12c, the
    de-install external standalone utility is replaced with a deinstall option
    using Oracle Universal Installer (OUI). The standalone or external deinstall
    utility is no longer provided as a separate installation media.
    You can de-install Oracle database 12cR1 software in 3
    ways:
    • From
      Installation Media using runInstaller (UNIX)
    • After
      Installation from ORACLE_HOME/deinstall directory
    • Using
      Parameter / Response file

    De-install utility can be used to de-install
    • Grid
      Infrastructure home (GRID_HOME)
    • Oracle Real
      Application (RAC) Database Home
    • Non-RAC
      Database Home (Single instance)
    • Grid
      Infrastructure for standalone server Homes.

    Procedure Download De-install utility for 11gR2
    • De-install
      utility can be downloaded from:

    • Accept
      license Agreement

    • Select
      latest Oracle database version based on your Operating system. 

    Here I am
    selecting Oracle Database 11.2.0.1 and Linux x86-64
    • Click
      on the “See All” hyper link to see
      all the software underneath it.
    • Accept
      License Agreement and look for the latest Oracle database version. For Example
      11.2.0.4

    Note: It is recommended to use the latest version
    of the 11.2 de-install utility. If you want to de-install 11.2.0.1, 11.2.0.2 or
    11.2.0.3 then you should use the latest version of the 11.2 de-install utility,
    which would be version 11.2.0.4.
    • Click on the
      download link to download the De-install utility for 11.2.0.4
    •  Enter the
      username and password, if prompted.
    • Download box
      will appear on your screen click OK to save the file.


    Copy De-install Utility to Exadata Compute
    Once De-install utility is
    downloaded, copy the zip to Exadata Compute node 1 and unzip it into a staging
    area. You can use WinScp or similar tool to copy the de-install utility to
    Exadata Compute node.
    Here I copied the file under “/u01/app/oracle/software” directory.

    Unzip De-install utility
    Use unzip OS command to unzip the
    de-install utility. This will create a sub-directory called “deinstall” which
    contains deinstall script.

    De-install utility Options
    The following
    are the options available with Oracle De-install utility.

    De-install DATABASE and GRID Homes
    Before running de-install:
    •  Unzip
      de-install utility into a directory outside Oracle Home.
    • No databases
      configured to use the Oracle home.

    $ srvctl config database -d <db_name>
    •  Home is not
      a configured Grid Home.

    $ ps -ef|grep d.bin
    • ASM is not
      detected in the Oracle Home.

    $ env | grep
    HOME

    Steps to De-install Oracle 11gR2 database
    software
    • After the upgrade is complete and
      the database and application have been validated and in use for some time, the
      11.2.0.2 or 11.2.0.3 database and grid homes can be removed using the deinstall
      tool. 
    • Run deinstall commands on the
      first database server. 
    • The deinstall tool will perform
      the deinstallation on all database servers. 

    Deinstall Database Home

    Here we are deinstalling Oracle
    Home:  /u01/app/oracle/product/11.2.0.2/dbhome_1/
    Replace the location with Oracle
    Home location you are deinstalling.
    Navigate to deinstall directory
    and perform a deinstall check only. The –checkonly option we will only perform
    a check without deinstalling the software.
    oracle@dm01db01-/u01/app/oracle/software:$
    cd /u01/app/oracle/software/deinstall

    oracle@dm01db01-/u01/app/oracle/software/deinstall:$
    ./deinstall -checkonly -home /u01/app/oracle/product/11.2.0.2/dbhome_1/
    Location of logs
    /u01/app/oraInventory/logs/
    ############ ORACLE DEINSTALL
    & DECONFIG TOOL START ############
    ######################### CHECK
    OPERATION START #########################
    ## [START] Install check
    configuration ##
    Checking for existence of the
    Oracle home location /u01/app/oracle/product/11.2.0.2/dbhome_1
    Oracle Home type selected for
    deinstall is: Oracle Real Application Cluster Database
    Oracle Base selected for
    deinstall is: /u01/app/oracle
    Checking for existence of central
    inventory location /u01/app/oraInventory
    Checking for existence of the
    Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
    The following nodes are part of
    this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Checking for sufficient temp
    space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    ## [END] Install check
    configuration ##
    Network Configuration check
    config START
    Network de-configuration trace
    file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-33-18-AM.log
    Network Configuration check
    config END
    Database Check Configuration
    START
    Database de-configuration trace
    file location:
    /u01/app/oraInventory/logs/databasedc_check2015-08-13_03-33-23-AM.log
    Use comma as separator when
    specifying list of values as input
    Specify the list of database
    names that are configured in this Oracle home []:
    Database Check Configuration END
    Enterprise Manager Configuration
    Assistant START
    EMCA de-configuration trace file
    location: /u01/app/oraInventory/logs/emcadc_check2015-08-13_03-33-30-AM.log
    Enterprise Manager Configuration
    Assistant END
    Oracle Configuration Manager
    check START
    OCM check log file location :
    /u01/app/oraInventory/logs//ocm_check_2015-08-13_03-33-30-AM.log
    Oracle Configuration Manager
    check END
    ######################### CHECK
    OPERATION END #########################
    ####################### CHECK
    OPERATION SUMMARY #######################
    Oracle Grid Infrastructure Home
    is: /u01/app/11.2.0.4/grid
    The cluster node(s) on which the
    Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Oracle Home selected for
    deinstall is: /u01/app/oracle/product/11.2.0.2/dbhome_1
    Inventory Location where the
    Oracle home registered is: /u01/app/oraInventory
    No Enterprise Manager
    configuration to be updated for any database(s)
    No Enterprise Manager ASM targets
    to update
    No Enterprise Manager listener
    targets to migrate
    Checking the config status for
    CCR
    dm01db01 : Oracle Home exists and
    CCR is configured
    dm01db02 : Oracle Home exists and
    CCR is configured
    dm01db03 : Oracle Home exists and
    CCR is configured
    dm01db04 : Oracle Home exists and
    CCR is configured
    dm01db05 : Oracle Home exists and
    CCR is configured
    dm01db06 : Oracle Home exists and
    CCR is configured
    dm01db07 : Oracle Home exists and
    CCR is configured
    dm01db08 : Oracle Home exists and
    CCR is configured
    CCR check is finished
    A log of this session will be
    written to:
    ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-33-08-AM.out’
    Any error messages from this
    session will be written to:
    ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-33-08-AM.err’
    ############# ORACLE DEINSTALL
    & DECONFIG TOOL END #############
    Now perform the deinstall using
    the following command.
    orcl1@dm01db01-/u01/app/oracle/software/deinstall:
    ./deinstall -home /u01/app/oracle/product/11.2.0.2/dbhome_1/
    Location of logs
    /u01/app/oraInventory/logs/
    ############ ORACLE DEINSTALL
    & DECONFIG TOOL START ############
    ######################### CHECK
    OPERATION START #########################
    ## [START] Install check
    configuration ##
    Checking for existence of the
    Oracle home location /u01/app/oracle/product/11.2.0.2/dbhome_1
    Oracle Home type selected for
    deinstall is: Oracle Real Application Cluster Database
    Oracle Base selected for
    deinstall is: /u01/app/oracle
    Checking for existence of central
    inventory location /u01/app/oraInventory
    Checking for existence of the
    Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
    The following nodes are part of
    this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Checking for sufficient temp
    space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    ## [END] Install check
    configuration ##
    Network Configuration check
    config START
    Network de-configuration trace
    file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-34-50-AM.log
    Network Configuration check
    config END
    Database Check Configuration
    START
    Database de-configuration trace
    file location:
    /u01/app/oraInventory/logs/databasedc_check2015-08-13_03-34-56-AM.log
    Use comma as separator when
    specifying list of values as input
    Specify the list of database
    names that are configured in this Oracle home []:
    Database Check Configuration END
    Enterprise Manager Configuration
    Assistant START
    EMCA de-configuration trace file
    location: /u01/app/oraInventory/logs/emcadc_check2015-08-13_03-35-05-AM.log
    Enterprise Manager Configuration
    Assistant END
    Oracle Configuration Manager
    check START
    OCM check log file location :
    /u01/app/oraInventory/logs//ocm_check_2015-08-13_03-35-06-AM.log
    Oracle Configuration Manager
    check END
    ######################### CHECK
    OPERATION END #########################
    ####################### CHECK
    OPERATION SUMMARY #######################
    Oracle Grid Infrastructure Home
    is: /u01/app/11.2.0.4/grid
    The cluster node(s) on which the
    Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Oracle Home selected for
    deinstall is: /u01/app/oracle/product/11.2.0.2/dbhome_1
    Inventory Location where the
    Oracle home registered is: /u01/app/oraInventory
    No Enterprise Manager
    configuration to be updated for any database(s)
    No Enterprise Manager ASM targets
    to update
    No Enterprise Manager listener
    targets to migrate
    Checking the config status for
    CCR
    dm01db01 : Oracle Home exists and
    CCR is configured
    dm01db02 : Oracle Home exists and
    CCR is configured
    dm01db03 : Oracle Home exists and
    CCR is configured
    dm01db04 : Oracle Home exists and
    CCR is configured
    dm01db05 : Oracle Home exists and
    CCR is configured
    dm01db06 : Oracle Home exists and
    CCR is configured
    dm01db07 : Oracle Home exists and
    CCR is configured
    dm01db08 : Oracle Home exists and
    CCR is configured
    CCR check is finished
    Do you want to continue (y – yes,
    n – no)? [n]: y
    A log of this session will be
    written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-34-40-AM.out’
    Any error messages from this
    session will be written to:
    ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-34-40-AM.err’
    ######################## CLEAN
    OPERATION START ########################
    Enterprise Manager Configuration
    Assistant START
    EMCA de-configuration trace file
    location: /u01/app/oraInventory/logs/emcadc_clean2015-08-13_03-35-05-AM.log
    Updating Enterprise Manager ASM
    targets (if any)
    Updating Enterprise Manager listener
    targets (if any)
    Enterprise Manager Configuration
    Assistant END
    Database de-configuration trace
    file location:
    /u01/app/oraInventory/logs/databasedc_clean2015-08-13_03-35-33-AM.log
    Network Configuration clean
    config START
    Network de-configuration trace
    file location: /u01/app/oraInventory/logs/netdc_clean2015-08-13_03-35-33-AM.log
    De-configuring Listener
    configuration file on all nodes…
    Listener configuration file
    de-configured successfully.
    De-configuring Naming Methods
    configuration file on all nodes…
    Naming Methods configuration file
    de-configured successfully.
    De-configuring Local Net Service
    Names configuration file on all nodes…
    Local Net Service Names
    configuration file de-configured successfully.
    De-configuring Directory Usage
    configuration file on all nodes…
    Directory Usage configuration
    file de-configured successfully.
    De-configuring backup files on
    all nodes…
    Backup files de-configured
    successfully.
    The network configuration has
    been cleaned up successfully.
    Network Configuration clean
    config END
    Oracle Configuration Manager
    clean START
    OCM clean log file location :
    /u01/app/oraInventory/logs//ocm_clean_2015-08-13_03-35-06-AM.log
    Oracle Configuration Manager
    clean END
    Setting the force flag to false
    Setting the force flag to cleanup
    the Oracle Base
    Oracle Universal Installer clean
    START
    Detach Oracle home
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ from the central inventory on the
    local node : Done
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-auto-inplace-rolling-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-inplace-rolling-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-out-of-place-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-out-of-place-switchback.xml’.
    The file is in use.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations’.
    The directory is not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable’.
    The directory is not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/.tmp’.
    The directory is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/log.txt’.
    The file is in use.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19’.
    The directory is not empty.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-auto-inplace-rolling-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-inplace-rolling-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-out-of-place-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-out-of-place-switchback.xml’.
    The file is in use.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations’.
    The directory is not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable’.
    The directory is not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/.tmp’.
    The directory is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/log.txt’.
    The file is in use.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50’.
    The directory is not empty.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-auto-inplace-rolling-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-inplace-rolling-automation.xml’.
    The file is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-out-of-place-automation.xml’.
    The file is in use.
    Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-out-of-place-switchback.xml’.
    The file is in use.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations’.
    The directory is not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable’.
    The directory is not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/.tmp’.
    The directory is in use.
    Failed to delete the file
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/log.txt’.
    The file is in use.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29’.
    The directory is not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan’. The directory is
    not empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs’. The directory is not
    empty.
    Failed to delete the directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’. The directory is not empty.
    Delete directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’
    on the local node : Failed <<<<
    The Oracle Base directory
    ‘/u01/app/oracle’ will not be removed on local node. The directory is in use by
    Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.
    Detach Oracle home ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’
    from the central inventory on the remote nodes ‘dm01db04,dm01db07,dm01db02,dm01db06,dm01db03,dm01db08,dm01db05’
    : Done
    Delete directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    : Done
    The Oracle Base directory
    ‘/u01/app/oracle’ will not be removed on node ‘dm01db04’. The directory is in
    use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.
    The Oracle Base directory ‘/u01/app/oracle’
    will not be removed on node ‘dm01db07’. The directory is in use by Oracle Home
    ‘/u01/app/oracle/product/10.2.0/agent10g’.
    The Oracle Base directory
    ‘/u01/app/oracle’ will not be removed on node ‘dm01db02’. The directory is in
    use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.
    The Oracle Base directory
    ‘/u01/app/oracle’ will not be removed on node ‘dm01db06’. The directory is in
    use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.
    The Oracle Base directory
    ‘/u01/app/oracle’ will not be removed on node ‘dm01db03’. The directory is in
    use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.
    The Oracle Base directory
    ‘/u01/app/oracle’ will not be removed on node ‘dm01db08’. The directory is in
    use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.
    The Oracle Base directory
    ‘/u01/app/oracle’ will not be removed on node ‘dm01db05’. The directory is in
    use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.
    Oracle Universal Installer
    cleanup was successful.
    Oracle Universal Installer clean
    END
    ## [START] Oracle install clean
    ##
    Clean install operation removing
    temporary directory ‘/tmp/deinstall2015-08-13_03-34-40AM’ on node ‘dm01db01’
    Clean install operation removing
    temporary directory ‘/tmp/deinstall2015-08-13_03-34-40AM’ on node ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    ## [END] Oracle install clean ##
    ######################### CLEAN
    OPERATION END #########################
    ####################### CLEAN
    OPERATION SUMMARY #######################
    Cleaning the config for CCR
    Cleaning the CCR configuration by
    executing its binaries
    CCR clean is finished
    Successfully detached Oracle home
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ from the central inventory on the
    local node.
    Failed to delete directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ on the local node.
    Successfully detached Oracle home
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ from the central inventory on the
    remote nodes ‘dm01db04,dm01db07,dm01db02,dm01db06,dm01db03,dm01db08,dm01db05’.
    Successfully deleted directory
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’.
    Oracle Universal Installer
    cleanup was successful.
    Oracle deinstall tool successfully
    cleaned up temporary directories.
    #######################################################################
    ############# ORACLE DEINSTALL
    & DECONFIG TOOL END #############
    Remove the directories if not
    removed by the deinstall utility
    orcl1@dm01db01-/u01/app/oracle/software/deinstall:
    orcl1@dm01db01-/u01/app/oracle/product/11.2.0.2:
    su – root
    Password:
    [root@dm01db01 ~]# cd
    /u01/app/oracle/product/11.2.0.2
    [root@dm01db01 11.2.0.2]# ls -ltr
    total 4
    drwxr-xr-x 3 oracle oinstall 4096
    Aug 13 03:36 dbhome_1
    [root@dm01db01 11.2.0.2]# du -sh dbhome_1/
    276K    dbhome_1/
    [root@dm01db01 11.2.0.2]# rm -rf dbhome_1/
    [root@dm01db01 11.2.0.2]# cd ..
    [root@dm01db01 product]# rmdir
    11.2.0.2/

    Deinstall Grid Home

    Here we deinstalling GRID
    Home:  /u01/app/11.2.0.2/grid_1/
    Replace the location with GRID
    Home location you are deinstalling.
    [root@dm01db01 ~]# cd
    /u01/app/11.2.0.2/grid_1

    [root@dm01db01 grid_1]# dcli -l
    root -g ~/dbs_group chmod -R 755 /u01/app/11.2.0.2/grid_1

    [root@dm01db01 grid_1]# dcli -l
    root -g ~/dbs_group chown -R oracle /u01/app/11.2.0.2/grid_1

    [root@dm01db01 grid_1]# cd ..

    [root@dm01db01 11.2.0.2]# ls -l
    total 4763768
    drwxr-xr-x 72 oracle
    oinstall       4096 Aug 22  2012 grid_1
    [root@dm01db01 11.2.0.2]# dcli -l
    root -g ~/dbs_group chown oracle /u01/app/11.2.0.2/grid_1
    [root@dm01db01 11.2.0.2]# ls -l
    total 4763768
    drwxr-xr-x 72 oracle
    oinstall       4096 Aug 22  2012 grid_1
    Navigate to deinstall directory
    and perform a deinstall check only. The –checkonly option we will only perform
    a check without deinstalling the software.
    oracle@dm01db01-/home/oracle:$  cd /u01/app/oracle/software/deinstall

    oracle@dm01db01-/u01/app/oracle/software/deinstall:$
     ./deinstall -checkonly -home
    /u01/app/11.2.0.2/grid_1/
    Location of logs /u01/app/oraInventory/logs/
    ############ ORACLE DEINSTALL
    & DECONFIG TOOL START ############
    ######################### CHECK
    OPERATION START #########################
    ## [START] Install check
    configuration ##
    The deinstall tool cannot
    determine the home type needed to deconfigure the selected home.  Please select the type of Oracle home you are
    trying to deinstall.
    Single Instance database – Enter
    1
    Real Application Cluster database
    – Enter 2
    Grid Infrastructure for a cluster
    – Enter 3
    Grid Infrastructure for a stand-alone
    server – Enter 4
    Client Oracle Home – Enter 5
    Transparent Gateways Oracle Home
    – Enter 6
    3
    The product version number of the
    specified home cannot be determined. Is the product version at least 11.2.0.1.0
    (y – yes, n – no)? [n]
    y
    Specify a comma-separated list of
    nodes on which to perform the deinstallation task:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Checking for existence of the
    Oracle home location /u01/app/11.2.0.2/grid_1
    Oracle Home type selected for
    deinstall is: Oracle Grid Infrastructure for a Cluster
    Oracle Base selected for
    deinstall is:
    Checking for existence of central
    inventory location /u01/app/oraInventory
    Checking for existence of the
    Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
    The following nodes are part of
    this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Checking for sufficient temp
    space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    ## [END] Install check
    configuration ##
    Traces log file:
    /u01/app/oraInventory/logs//crsdc.log
    Network Configuration check
    config START
    Network de-configuration trace
    file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-54-29-AM.log
    Network Configuration check
    config END
    Asm Check Configuration START
    ASM de-configuration trace file
    location: /u01/app/oraInventory/logs/asmcadc_check2015-08-13_03-54-30-AM.log
    ASM was not detected in the
    Oracle Home
    ######################### CHECK
    OPERATION END #########################
    ####################### CHECK
    OPERATION SUMMARY #######################
    Oracle Grid Infrastructure Home
    is: /u01/app/11.2.0.4/grid
    The cluster node(s) on which the
    Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Oracle Home selected for
    deinstall is: /u01/app/11.2.0.2/grid_1
    Inventory Location where the
    Oracle home registered is: /u01/app/oraInventory
    The home being deconfigured is
    NOT a configured Grid Infrastructure home (/u01/app/11.2.0.4/grid)
    ASM was not detected in the
    Oracle Home
    A log of this session will be
    written to:
    ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-52-24-AM.out’
    Any error messages from this
    session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-52-24-AM.err’
    ############# ORACLE DEINSTALL
    & DECONFIG TOOL END #############
    Now perform the deinstall using
    the following command.
    orcl1@dm01db01-/u01/app/oracle/software/deinstall:
    ./deinstall -home /u01/app/11.2.0.2/grid_1/
    Location of logs
    /u01/app/oraInventory/logs/
    ############ ORACLE DEINSTALL
    & DECONFIG TOOL START ############
    ######################### CHECK
    OPERATION START #########################
    ## [START] Install check
    configuration ##
    The deinstall tool cannot
    determine the home type needed to deconfigure the selected home.  Please select the type of Oracle home you are
    trying to deinstall.
    Single Instance database – Enter
    1
    Real Application Cluster database
    – Enter 2
    Grid Infrastructure for a cluster
    – Enter 3
    Grid Infrastructure for a
    stand-alone server – Enter 4
    Client Oracle Home – Enter 5
    Transparent Gateways Oracle Home
    – Enter 6
    3
    The product version number of the
    specified home cannot be determined. Is the product version at least 11.2.0.1.0
    (y – yes, n – no)? [n]
    y
    Specify a comma-separated list of
    nodes on which to perform the deinstallation task:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Checking for existence of the
    Oracle home location /u01/app/11.2.0.2/grid_1
    Oracle Home type selected for
    deinstall is: Oracle Grid Infrastructure for a Cluster
    Oracle Base selected for
    deinstall is:
    Checking for existence of central
    inventory location /u01/app/oraInventory
    Checking for existence of the
    Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
    The following nodes are part of
    this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Checking for sufficient temp
    space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    ## [END] Install check
    configuration ##
    Traces log file:
    /u01/app/oraInventory/logs//crsdc.log
    Network Configuration check
    config START
    Network de-configuration trace
    file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-56-28-AM.log
    Network Configuration check
    config END
    Asm Check Configuration START
    ASM de-configuration trace file
    location: /u01/app/oraInventory/logs/asmcadc_check2015-08-13_03-56-29-AM.log
    ASM was not detected in the
    Oracle Home
    ######################### CHECK
    OPERATION END #########################
    ####################### CHECK
    OPERATION SUMMARY #######################
    Oracle Grid Infrastructure Home
    is: /u01/app/11.2.0.4/grid
    The cluster node(s) on which the
    Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Oracle Home selected for
    deinstall is: /u01/app/11.2.0.2/grid_1
    Inventory Location where the
    Oracle home registered is: /u01/app/oraInventory
    The home being deconfigured is
    NOT a configured Grid Infrastructure home (/u01/app/11.2.0.4/grid)
    ASM was not detected in the
    Oracle Home
    Do you want to continue (y – yes,
    n – no)? [n]: y
    A log of this session will be
    written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-56-01-AM.out’
    Any error messages from this
    session will be written to:
    ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-56-01-AM.err’
    ######################## CLEAN
    OPERATION START ########################
    ASM de-configuration trace file
    location: /u01/app/oraInventory/logs/asmcadc_clean2015-08-13_03-56-47-AM.log
    ASM Clean Configuration END
    Network Configuration clean
    config START
    Network de-configuration trace
    file location: /u01/app/oraInventory/logs/netdc_clean2015-08-13_03-56-47-AM.log
    De-configuring Naming Methods
    configuration file on all nodes…
    Naming Methods configuration file
    de-configured successfully.
    De-configuring Local Net Service
    Names configuration file on all nodes…
    Local Net Service Names configuration
    file de-configured successfully.
    De-configuring Directory Usage
    configuration file on all nodes…
    Directory Usage configuration
    file de-configured successfully.
    De-configuring backup files on
    all nodes…
    Backup files de-configured
    successfully.
    The network configuration has
    been cleaned up successfully.
    Network Configuration clean
    config END
    Setting the force flag to false
    Setting the force flag to cleanup
    the Oracle Base
    Oracle Universal Installer clean
    START
    Failed to delete the directory
    ‘/u01/app/11.2.0.2/grid_1’. The directory is in use.
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the local node : Failed <<<<
    Failed to delete the directory
    ‘/u01/app/11.2.0.2/grid_1’. The directory is in use.
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the local node : Failed <<<<
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    : Failed <<<<
    Could not remove listed
    directories based on ‘/tmp/OraInstall2015-08-13_03-56-54-AM/installRemoveDirFile.lst’
    from nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08′.
    [/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission
    denied]
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db02’ : Failed
    <<<<
    The directory ‘/u01/app/11.2.0.2/grid_1’
    could not be deleted on the nodes ‘dm01db02’.
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db03’ : Failed
    <<<<
    The directory ‘/u01/app/11.2.0.2/grid_1’
    could not be deleted on the nodes ‘dm01db03’.
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db04’ : Failed
    <<<<
    The directory ‘/u01/app/11.2.0.2/grid_1’
    could not be deleted on the nodes ‘dm01db04’.
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db05’ : Failed
    <<<<
    The directory ‘/u01/app/11.2.0.2/grid_1’
    could not be deleted on the nodes ‘dm01db05’.
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db06’ : Failed
    <<<<
    The directory ‘/u01/app/11.2.0.2/grid_1’
    could not be deleted on the nodes ‘dm01db06’.
    Delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db07’ : Failed
    <<<<
    The directory ‘/u01/app/11.2.0.2/grid_1’
    could not be deleted on the nodes ‘dm01db07’.
    Delete directory ‘/u01/app/11.2.0.2/grid_1’
    on the remote nodes ‘dm01db08’ : Failed <<<<
    The directory ‘/u01/app/11.2.0.2/grid_1’
    could not be deleted on the nodes ‘dm01db08’.
    Oracle Universal Installer
    cleanup completed with errors.
    Oracle Universal Installer clean
    END
    ## [START] Oracle install clean
    ##
    Clean install operation removing
    temporary directory ‘/tmp/deinstall2015-08-13_03-56-01AM’ on node ‘dm01db01’
    Clean install operation removing
    temporary directory ‘/tmp/deinstall2015-08-13_03-56-01AM’ on node ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’
    ## [END] Oracle install clean ##
    ######################### CLEAN
    OPERATION END #########################
    ####################### CLEAN
    OPERATION SUMMARY #######################
    The home being deconfigured is
    NOT a configured Grid Infrastructure home (/u01/app/11.2.0.4/grid)
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the local node.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the local node.
    Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’
    on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db02’.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db03’.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db04’.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db05’.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db06’.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db07’.
    Failed to delete directory
    ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db08’.
    Oracle Universal Installer
    cleanup completed with errors.
    For complete clean up of Oracle
    Clusterware software from the system, deinstall the following old clusterware
    home(s). Refer to Clusterware Install guide of respective old release for
    details.
        /u01/app/11.2.0.3/grid on nodes : dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
    Oracle deinstall tool
    successfully cleaned up temporary directories.
    #######################################################################
    ############# ORACLE DEINSTALL
    & DECONFIG TOOL END #############

    Post De-install Steps
    • Verify the OraInventory is update
      and OLD Oracle Homes are removed.

    orcl1@dm01db01-/u01/app/11.2.0.2/grid_1:
    dcli -g ~/dbs_group -l oracle -s “-q” “grep
    ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ /u01/app/oraInventory/ContentsXML/inventory.xml”

    orcl1@dm01db01-/u01/app/11.2.0.2/grid_1:
    dcli -g ~/dbs_group -l oracle -s “-q” “grep ‘/u01/app/11.2.0.2/grid_1’
    /u01/app/oraInventory/ContentsXML/inventory.xml”

    • Verify the directories are deleted;
      if not manually delete them carefully.

     orcl1@dm01db01-/u01/app/oracle/software/deinstall:
    dcli -g ~/dbs_group -l oracle -s “-q” ‘ls -l
    /u01/app/oracle/product/11.2.0.2’
    dm01db01: ls:
    /u01/app/oracle/product/11.2.0.2: No such file or directory
    dm01db02: total 0
    dm01db03: total 0
    dm01db04: total 0
    dm01db05: total 0
    dm01db06: total 0
    dm01db07: total 0
    dm01db08: total 0
      
    orcl1@dm01db01-/u01/app/oracle/software/deinstall:
    dcli -g ~/dbs_group -l oracle -s “-q” ‘ls -l /u01/app/11.2.0.2’
    dm01db01: total 4763768
    dm01db01: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:57 grid_1
    dm01db02: total 4
    dm01db02: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:58 grid_1
    dm01db03: total 4
    dm01db03: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:58 grid_1
    dm01db04: total 4
    dm01db04: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:58 grid_1
    dm01db05: total 4
    dm01db05: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:58 grid_1
    dm01db06: total 4
    dm01db06: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:58 grid_1
    dm01db07: total 4
    dm01db07: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:58 grid_1
    dm01db08: total 4
    dm01db08: drwxr-xr-x 2 oracle
    oinstall 4096 Aug 13 03:58 grid_1
    orcl1@dm01db01-/u01/app/11.2.0.2/grid_1:
    su – root
    [root@dm01db01 ~]#  dcli -g ~/dbs_group -l root -s “-q”
    ‘cd /u01/app/; rmdir 11.2.0.2’
    [root@dm01db01 ~]# dcli -g
    ~/dbs_group -l root -s “-q” ‘ls -l /u01/app/11.2.0.2’
    dm01db01: total 4763764
    dm01db02: total 0
    dm01db03: total 0
    dm01db04: total 0
    dm01db05: total 0
    dm01db06: total 0
    dm01db07: total 0
    dm01db08: total 0
    [root@dm01db01 ~]# dcli -g
    ~/dbs_group -l root -s “-q” ‘cd /u01/app/; rmdir 11.2.0.2’
    [root@dm01db01 ~]# dcli -g
    ~/dbs_group -l root -s “-q” ‘cd /u01/app/; ls -l 11.2.0.2’
    dm01db01: ls: 11.2.0.2: No such
    file or directory
    dm01db02: ls: 11.2.0.2: No such
    file or directory
    dm01db03: ls: 11.2.0.2: No such
    file or directory
    dm01db04: ls: 11.2.0.2: No such
    file or directory
    dm01db05: ls: 11.2.0.2: No such
    file or directory
    dm01db06: ls: 11.2.0.2: No such
    file or directory
    dm01db07: ls: 11.2.0.2: No such
    file or directory
    dm01db08: ls: 11.2.0.2: No such
    file or directory
    •  Verify the free space available
      after de-install completed.

    [root@dm01db01 ~]# dcli -g
    ~/dbs_group -l root ‘df -h /u01’
    dm01db01: Filesystem            Size  Used Avail Use% Mounted on
    dm01db01: /dev/sda3             193G   81G 
    103G  45% /u01
    dm01db02: Filesystem            Size  Used Avail Use% Mounted on
    dm01db02: /dev/sda3             193G   48G 
    136G  26% /u01
    dm01db03: Filesystem            Size  Used Avail Use% Mounted on
    dm01db03: /dev/sda3             193G   48G 
    136G  27% /u01
    dm01db04: Filesystem            Size  Used Avail Use% Mounted on
    dm01db04: /dev/sda3             193G   58G 
    126G  32% /u01
    dm01db05: Filesystem            Size  Used Avail Use% Mounted on
    dm01db05: /dev/sda3             193G   47G 
    137G  26% /u01
    dm01db06: Filesystem            Size  Used Avail Use% Mounted on
    dm01db06: /dev/sda3             193G   63G 
    121G  35% /u01
    dm01db07: Filesystem            Size  Used Avail Use% Mounted on
    dm01db07: /dev/sda3             193G   42G 
    142G  23% /u01
    dm01db08: Filesystem            Size  Used Avail Use% Mounted on
    dm01db08: /dev/sda3             193G   54G 
    130G  30% /u01

    IMPORTANT NOTES
    • For 12.1.0.2.0, please be aware
      of the known issue with oraInst.loc discussed in “Deinstall Unable to
      Determine Product Version Number (Doc ID 1966643.1)”
    • Oracle 12c Release 1 (12.1)
      deinstall tool from Installation Media should not be used to deinstall earlier
      releases of Oracle Home as it is not supported.
    • The deinstall tool from Oracle
      Home should be used only if the Oracle Database 12.1 Software is being
      successfully Installed.
    • For Incomplete or Failed
      Software Installation, you must only use the deinstall tool from Installation
      Media.

    Conclusion
    In this article we have seen how to de-install Oracle Grid and RDBMS software. The external deinstall utility for 11gR2 and built in De-install utility makes the process easier to remove the unused or old Oracle Homes.


  • History Of Exadata

    Exadata was first launched in the year 2008 and it was names
    as Exadata V1. Since then Exadata has undergone many significant changes.
    In this article I will give a quick overview of Exadata
    history and changes made in each of the model.
            
    V1
         First
    Exadata model released in 2008
         It
    was a combination of HP hardware + Oracle Software
         Exclusively
    for Data Warehouse workload
         There
    were too many hardware failures noticed in this model
         Working
    with Hardware issues was not easy as it involved multiple vendors
         There
    was no Flash in this version
         Major
    problem with V1 was the cabinet gets overheated. You can fry Eggs on the
    cabinetJ

            
    V2
         V2
    was released in 2009
         Combination
    of Sun hardware and Oracle Software
         Flash
    Cache was introduced for the first time in V2
         Caches
    data intelligently on Flash Cache
         Faster
    components and used solid state disks
         More
    than just Data Warehouse

                
    X2
         X2
    is the third version of Exadata and it was released in 2010
         This
    time the naming changed to X series
         Introduced
    two variants X2-2 and X2-8
         Introduced
    Exadata Storage Expansion for addition storage capacity
         Faster
    components and increased capacity (CPU, memory and disk)

            
    X3
         X3
    was released in 2012
         Named
    as “In-Memory Database machines”
         Introduced
    Exadata Eight Rack configuration
         Increased
    Flash cache
         Introduced
    write back flash cache
         Faster
    components and increased capacity
         Increased
    High Capacity disk Size from 2TB to 3TB

         Introduced
    new Software feature Write-Back Flash Cache


            
    X4
         It
    was released in 2013
         Introduced
    Xeon processors
         Physical
    memory can be expanded up to 512GB on Compute nodes
         Doubled
    Flash Cache
         Increased
    High Performance disk size from 600GB to 1.2TB
         Increased
    High Capacity disk size from 3TB to 4TB
         Introduced
    Active/Active Infiniband network connection
      
            
    X5
         It
    was released in 2015
         Uses
    2-socket CPU
         Physical
    memory can be expanded up to 768GB on Compute nodes
         Introduced
    OVM
         Introduced
    Extreme Flash Cache servers
         Introduced
    Elastic Exadata configuration
         High
    performance disk are removed in favor of Extreme Flash Cache servers
         Doubled
    flash cache and used NVMe

            
    X6
         X6
    was released in 2016
         Increase
    Flash cache
         Faster
    components and increased capacity
         Also
    available in Public Cloud
         Ideal
    for DBaas

    The diagram below gives you an overview of the Exadata hardware evolution.


    Conclusion

    In this article we have learned the history of Exadata. We
    have seen how Exadata has undergone significant changes since it’s launched in
    the year 2008. 
  • Exadata Database Machine and Its Components

    What is Exadata Database Machine?
    Exadata Database Machine or
    simply known as Exadata is:
    •           An
      Engineered System
    •           A
      pre-configured combination of balanced Hardware and unique software
    •           A unique
      platform for running Oracle Databases
    •           Consists of
      Compute Grid, Storage Grid and Network Grid
    •           A fully
      integrated platform for Oracle Database
    •           Ideal for
      Database Consolidation platform
    •           It provides
      High Availability and High Performance for all types of Workloads

    The Oracle Exadata Database
    Machine is an Engineered System designed to deliver extreme performance and
    high availability for all type of Oracle database workloads (OLTP, OLAP &
    Mixed Workload).
    The Oracle Exadata vision is to
    deliver dramatically better performance for all database workloads.
    Why Exadata?
    •           It provides
      Extreme Performance and High Availability
    •           Exadata is a
      flash optimized Oracle database platform by providing intelligent caching
      algorithms to cache *appropriate* data in flash Cache
    •           For large
      Data Warehouse databases, it addresses the inability to move large volumes of
      database from the storage to database servers
    •           For OLTP, it
      supports larger user population and transaction volumes and provides faster
      response time
    •           It supports
      mixed workload in a Consolidation environment and prioritizes workload

    Exadata Features
    Exadata provides unique
    performance features such as:
    •           Smart Scan
    •           Smart Flash
      Cache
    •           Smart Flash
      Logging
    •           Storage
      Indexes
    •           I/O Resource
      Management (IORM)
    •           Hybrid Columnar
      Compression (EHCC)

    Exadata Hardware and Software Components
    Exadata Database Machine consists
    of a storage grid, compute grid, and network grid. Each grid, or hardware
    layer, is built with multiple high-performing, industry-standard Oracle servers
    to provide hardware and system fault tolerance.
    Exadata Storage Server Hardware
    & Software
            
    High
    Performance storage for Oracle Database
            
    64 bit
    Intel-based sun Fire Servers
            
    Contains
    either 12 hard disks or 8 flash disks
            
    Preinstalled
    Software:
         Exadata Storage Server Software
         Oracle Linux x86_64
         Drivers and utilities
            
    Only
    available in conjunction with Database Machines
    Exadata Compute Node Hardware
    & Software
            
    X5-2
    database servers are based on the Intel Sun Fire.
            
    Each server
    has 2×18-core Intel Xeon Processors and 256GB of memory.
            
    Can run
    Oracle Linux x86_64 or Oracle Solaris.
            
    Oracle Linux
    is supported on 2-socket and 8 socket database servers.
            
    Oracle
    Solaris is supported only on 2-socket database servers
            
    12.1.1.1.1
    is the terminal version that supports database servers running Oracle
    Solaris. 
            
    Run Grid
    Infrastructure and RDBMS software
           Infiniband Switch
            
    One of the
    more important hardware components of Exadata is the InfiniBand network.
            
    It is used
    for transferring data between the database tier and the storage tier.
            
    It is also
    used for interconnect traffic between the database servers, if they are
    configured in a RAC cluster.
            
    In addition,
    the InfiniBand network may be used to connect to external systems for such uses
    as backups.
    Exadata Configurations
            
    Eight Rack
         2 Compute node, 3 Storage Cells and 2
    Infiniband Switch
            
    Quarter Rack
         2 Compute node, 3 Storage Cells and 2
    Infiniband Switch
            
    Half Rack
         4 Compute node, 7 Storage Cells and 2
    Infiniband Switch
            
    Full Rack
         8 Compute node, 14 Storage Cells and 2
    Infiniband Switch
            
    Elastic
    Configuration
         2 – 9 Compute nodes and 3 – 18 Storage cells
    per Rack
    The difference between Eight and
    Quarter is that, the Eight Rack comes with half of the capacity compared to
    Quarter Rack. In Eight Rack half of the processor and Disks are disabled.
    Exadata Benefits
            
    High
    performance
            
    High
    availability
            
    Scalability
            
    Manage any
    kind of workload
            
    Easy to
    deploy
            
    Ideal for
    Consolidation
            
    Virtualization
            
    Available
    on-premises and cloud
    Exadata Rack
    This image gives an overview of
    Exadata X5-2 database machine.
    Individual Exadata Components



    Exadata Storage Cell

    Exadata Compute node

    Exadata Infiniband Switch


    Conclusion
    In this article we have learned
    about Exadata Database Machine and its components. Exadata truly provides
    extreme performance and highly availability using its faster and balanced
    hardware and unique Exadata storage software.