Overview

Oracle Exadata Database Machine consists of several components such as Compute nodes, Storage Cells, ILOM, Infiniband Switches, Cisco Switch and PDUs. So we need a tool that can manage all these components from one single console. Oracle Enterprise manager Cloud Control is the recommended best practices for monitoring and managing Exadata Database machine. Once the Exadata database Machine is installed the next step is enable monitoring for it.

The following Exadata components can be monitored and managed by OEM:
  • Compute Nodes
  • Storage Cells
  • Infiniband Switches
  • Cisco Switches
  • Power Distribution Units
  • KVM

The first step in monitoring and managing Exadata using OEM is to install the EM Agent. You can install EM Agent in several ways, such as:
  • Using EM Kit Method
  • Using the Agent push method
  • Using an RPM file
  • Using the AgentPull script
  • Using the AgentDeploy script

In this article we will demonstrate how to Install EM Agent on Exadata using Agent Push method using OEM 13c. Agent Software is installed only on Compute nodes.

Environment Details

Here we will be installing EM Agent on a Exadata V2 Full Rack consists of:
  • 8 Compute Nodes
  • 14 Storage Cells
  • 3 Infiniband Switches
  • 1 Cisco Switches
  • 2 Power Distribution Units
  • 1 KVM

Procedure to Install EM Agent on Exadata Compute nodes

Enter the OEM 13c URL into the web browser and hit enter

Enter the SYSMAN Credentials or any other user that have necessary permissions to Install Agent software

We are on the OEM 13c Home page now

From the Home page, click on Setup à Add Target à Add Targets manually

Click on Install Agent on Host

On this page, click on the +Add button to add the Host targets

Enter all fully qualified Hostnames (8 compute nodes) and for Platform “Same for All Hosts” and click next

Enter the Installation Base Directory.
The instance base directory will be populated automatically for you.
Click on the + symbol on the Named credentials and enter oracle user and its password.
Click on the + symbol on the Root credentials line and enter root user and its password.




Click next

Click Deploy Agent

The Agent Installation process started

Initialization in progress

The Remote prerequisite check shows warning. This can be ignored as oracle user doesn’t have sudo permissions to run the root.sh script. The can be ran at the end.

Click Continue à Continue, All Hosts

Installation process completed. Click Done.

Connect to Exadata compute node 1 and run the root.sh script as follows:

[root@dm01db01 ~]# dcli -g dbs_group -l root ‘/u01/app/oracle/product/Agent13c/agent_13.2.0.0.0/root.sh’
dm01db01: Finished product-specific root actions.
dm01db01: /etc exist
dm01db02: Finished product-specific root actions.
dm01db02: /etc exist
dm01db03: Finished product-specific root actions.
dm01db03: /etc exist
dm01db04: Finished product-specific root actions.
dm01db04: /etc exist
dm01db05: Finished product-specific root actions.
dm01db05: /etc exist
dm01db06: Finished product-specific root actions.
dm01db06: /etc exist
dm01db07: Finished product-specific root actions.
dm01db07: /etc exist
dm01db08: Finished product-specific root actions.
dm01db08: /etc exist

Verify Agent is running on Exadata compute nodes.

dm01db01-orcl1 {/home/oracle}:/u01/app/oracle/product/Agent13c/agent_inst/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/product/Agent13c/agent_inst
Agent Log Directory    : /u01/app/oracle/product/Agent13c/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/product/Agent13c/agent_13.2.0.0.0
Core JAR Location      : /u01/app/oracle/product/Agent13c/agent_13.2.0.0.0/jlib
Agent Process ID       : 4544
Parent Process ID      : 4424
Agent URL              : https://dm01db01.int.thomsonreuters.com:3872/emd/main/
Local Agent URL in NAT : https://dm01db01.int.thomsonreuters.com:3872/emd/main/
Repository URL         : https://oem13c.domain.com:1159/empbs/upload
Started at             : 2016-12-29 04:40:07
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.248.3.el6uek.x86_64 (amd64)
Number of Targets      : 3
Last Reload            : (none)
Last successful upload                       : 2016-12-29 04:44:54
Last attempted upload                        : 2016-12-29 04:44:54
Total Megabytes of XML files uploaded so far : 0.33
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 17.84%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-12-29 04:47:11
Last successful heartbeat to OMS             : 2016-12-29 04:47:11
Next scheduled heartbeat to OMS              : 2016-12-29 04:48:12

—————————————————————
Agent is Running and Ready

Verify that Exadata computes are visible in OEM 13c


Next we will discover Exadata Database Machine in OEM 13c


4


Overview:

Oracle Enterprise Manager is Oracle’s integrated enterprise information technology management product line, which provides the industry’s only complete, integrated, and business-driven enterprise cloud management solution. Oracle Enterprise Manager Cloud Control 13c is Oracle’s single, integrated solution for managing all aspects of the Oracle Cloud and the applications running on it.



Media:

Download the software from following link.




Required Packages for Oracle Cloud Control Installation (Linux 6)

make
binutils
gcc
libaio
glibc-common
libstdc++
libXtst
sysstat
glibc
glibc-devel
glibc-devel.i686

Assumptions:

Here we are assuming that:

Oracle RDMS software Installed and
Oracle Database has been created for Repository.
 


Steps to Install OEM Cloud Control 13c.

      1. Download media and unzip
     2. Execute runInstaller.
     3. Uncheck the box to prevent from receive security updates from Oracle Support and click next.
    4. Click yes on warning pop-up window.
    5. Provide Inventory Location.   

6. Prerequisites window will appear where all pre-check must succeeded.  

   7. In Installation types, Choose Advanced method.   



8. Provide Middleware Home Location, Agent Base Directory and Hostname.
9.Provide weblogic details then click next.
10. Provide Database Connection Details.
11.Provide Sysman credentials and click next.
     12.  Check on Select configure Oracle Software Library, Uncheck configure shared location and tick on Enable Oracle BI Publisher.
    13. Provide port details and click next.
    14. Review the information, which we have provided then click on Install.
    15. Execute configuration script as Root user.
  


[MN-Dboem]root@*****:/opt/oem/MW_13c# sh allroot.sh



Starting to execute allroot.sh ………

Starting to execute /opt/oem/MW_13c/root.sh ……

/etc exist

/opt/oem/MW_13c

Finished product-specific root actions.

/etc exist

Finished execution of  /opt/oem/MW_13c/root.sh ……

Starting to execute /opt/oem/Agent_13c/agent_13.1.0.0.0/root.sh ……

Finished product-specific root actions.

/etc exist

Finished execution of  /opt/oem/Agent_13c/agent_13.1.0.0.0/root.sh ……

[MN-Dboem]root@*****:/opt/oem/MW_13c#
  

16. Review the Details then click on Close

 
Conclusion:


In the preceding Blog we have learned Oracle Enterprise Manager Cloud Control 13c Installation. We have select advanced method Installation which is meant for small, medium and large deployments and also allows you customize the ports according to your environment.


By
Name : Mirza Hidayathullah Baig
Designation : Senior Database Engineer
Organization: NetSoftMate IT Solutions














1

Overview
After Oracle Platinum upgrades Oracle software for example from 11.2.0.3 to 11.2.0.4, the OLD Oracle Homes are not removed immediately. There are 2 different types of Oracle Home, Grid Infrastructure (ASM + Clusterware) home and RDBMS (Database software) home.

In this article I will demonstrate how to De-install Oracle 11gR2 and 12cR1 Homes on Exadata database machine.

Why De-install Oracle Software?
One should consider de-installing Oracle software for the following reasons:
  • To save disk space
  • To ease management complexities
  • To avoid accidentally using unused home

About De-install Utility
The de-install utility works different for Oracle Database 11g and 12c.
Starting with Oracle database 11.2.0.1, Oracle recommends to use de-install utility to de-install Oracle 11.2 software.

You can de-install Oracle database 11gR2 software in 2 ways:
  • External De-install Utility – highly recommended method
  • runInstaller

In Oracle Database 12c, deinstall utility is now integrated with the Database Installation Media. You can run the deinstall utility using the runInstaller command on Linux and UNIX. In 12c, the de-install external standalone utility is replaced with a deinstall option using Oracle Universal Installer (OUI). The standalone or external deinstall utility is no longer provided as a separate installation media.

You can de-install Oracle database 12cR1 software in 3 ways:
  • From Installation Media using runInstaller (UNIX)
  • After Installation from ORACLE_HOME/deinstall directory
  • Using Parameter / Response file

De-install utility can be used to de-install
  • Grid Infrastructure home (GRID_HOME)
  • Oracle Real Application (RAC) Database Home
  • Non-RAC Database Home (Single instance)
  • Grid Infrastructure for standalone server Homes.

Procedure Download De-install utility for 11gR2
  • De-install utility can be downloaded from:

  • Accept license Agreement

  • Select latest Oracle database version based on your Operating system. 

Here I am selecting Oracle Database 11.2.0.1 and Linux x86-64
  • Click on the “See All” hyper link to see all the software underneath it.
  • Accept License Agreement and look for the latest Oracle database version. For Example 11.2.0.4

Note: It is recommended to use the latest version of the 11.2 de-install utility. If you want to de-install 11.2.0.1, 11.2.0.2 or 11.2.0.3 then you should use the latest version of the 11.2 de-install utility, which would be version 11.2.0.4.
  • Click on the download link to download the De-install utility for 11.2.0.4
  •  Enter the username and password, if prompted.
  • Download box will appear on your screen click OK to save the file.


Copy De-install Utility to Exadata Compute
Once De-install utility is downloaded, copy the zip to Exadata Compute node 1 and unzip it into a staging area. You can use WinScp or similar tool to copy the de-install utility to Exadata Compute node.
Here I copied the file under “/u01/app/oracle/software” directory.

Unzip De-install utility
Use unzip OS command to unzip the de-install utility. This will create a sub-directory called “deinstall” which contains deinstall script.

De-install utility Options
The following are the options available with Oracle De-install utility.

De-install DATABASE and GRID Homes
Before running de-install:
  •  Unzip de-install utility into a directory outside Oracle Home.
  • No databases configured to use the Oracle home.

$ srvctl config database -d <db_name>
  •  Home is not a configured Grid Home.

$ ps -ef|grep d.bin
  • ASM is not detected in the Oracle Home.

$ env | grep HOME

Steps to De-install Oracle 11gR2 database software
  • After the upgrade is complete and the database and application have been validated and in use for some time, the 11.2.0.2 or 11.2.0.3 database and grid homes can be removed using the deinstall tool. 
  • Run deinstall commands on the first database server. 
  • The deinstall tool will perform the deinstallation on all database servers. 

Deinstall Database Home

Here we are deinstalling Oracle Home:  /u01/app/oracle/product/11.2.0.2/dbhome_1/
Replace the location with Oracle Home location you are deinstalling.

Navigate to deinstall directory and perform a deinstall check only. The –checkonly option we will only perform a check without deinstalling the software.

oracle@dm01db01-/u01/app/oracle/software:$ cd /u01/app/oracle/software/deinstall

oracle@dm01db01-/u01/app/oracle/software/deinstall:$ ./deinstall -checkonly -home /u01/app/oracle/product/11.2.0.2/dbhome_1/
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################
## [START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/oracle/product/11.2.0.2/dbhome_1
Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
The following nodes are part of this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Checking for sufficient temp space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’

## [END] Install check configuration ##


Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-33-18-AM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2015-08-13_03-33-23-AM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home []:
Database Check Configuration END

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_check2015-08-13_03-33-30-AM.log

Enterprise Manager Configuration Assistant END
Oracle Configuration Manager check START
OCM check log file location : /u01/app/oraInventory/logs//ocm_check_2015-08-13_03-33-30-AM.log
Oracle Configuration Manager check END

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0.4/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Oracle Home selected for deinstall is: /u01/app/oracle/product/11.2.0.2/dbhome_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
No Enterprise Manager configuration to be updated for any database(s)
No Enterprise Manager ASM targets to update
No Enterprise Manager listener targets to migrate
Checking the config status for CCR
dm01db01 : Oracle Home exists and CCR is configured
dm01db02 : Oracle Home exists and CCR is configured
dm01db03 : Oracle Home exists and CCR is configured
dm01db04 : Oracle Home exists and CCR is configured
dm01db05 : Oracle Home exists and CCR is configured
dm01db06 : Oracle Home exists and CCR is configured
dm01db07 : Oracle Home exists and CCR is configured
dm01db08 : Oracle Home exists and CCR is configured
CCR check is finished
A log of this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-33-08-AM.out’
Any error messages from this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-33-08-AM.err’

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

Now perform the deinstall using the following command.

orcl1@dm01db01-/u01/app/oracle/software/deinstall: ./deinstall -home /u01/app/oracle/product/11.2.0.2/dbhome_1/
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /u01/app/oracle/product/11.2.0.2/dbhome_1
Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
The following nodes are part of this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Checking for sufficient temp space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’

## [END] Install check configuration ##


Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-34-50-AM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2015-08-13_03-34-56-AM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home []:
Database Check Configuration END

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_check2015-08-13_03-35-05-AM.log

Enterprise Manager Configuration Assistant END
Oracle Configuration Manager check START
OCM check log file location : /u01/app/oraInventory/logs//ocm_check_2015-08-13_03-35-06-AM.log
Oracle Configuration Manager check END

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0.4/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Oracle Home selected for deinstall is: /u01/app/oracle/product/11.2.0.2/dbhome_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
No Enterprise Manager configuration to be updated for any database(s)
No Enterprise Manager ASM targets to update
No Enterprise Manager listener targets to migrate
Checking the config status for CCR
dm01db01 : Oracle Home exists and CCR is configured
dm01db02 : Oracle Home exists and CCR is configured
dm01db03 : Oracle Home exists and CCR is configured
dm01db04 : Oracle Home exists and CCR is configured
dm01db05 : Oracle Home exists and CCR is configured
dm01db06 : Oracle Home exists and CCR is configured
dm01db07 : Oracle Home exists and CCR is configured
dm01db08 : Oracle Home exists and CCR is configured
CCR check is finished
Do you want to continue (y – yes, n – no)? [n]: y
A log of this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-34-40-AM.out’
Any error messages from this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-34-40-AM.err’

######################## CLEAN OPERATION START ########################

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_clean2015-08-13_03-35-05-AM.log

Updating Enterprise Manager ASM targets (if any)
Updating Enterprise Manager listener targets (if any)
Enterprise Manager Configuration Assistant END
Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2015-08-13_03-35-33-AM.log

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2015-08-13_03-35-33-AM.log

De-configuring Listener configuration file on all nodes…
Listener configuration file de-configured successfully.

De-configuring Naming Methods configuration file on all nodes…
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes…
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes…
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes…
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START
OCM clean log file location : /u01/app/oraInventory/logs//ocm_clean_2015-08-13_03-35-06-AM.log
Oracle Configuration Manager clean END
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ from the central inventory on the local node : Done

Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-auto-inplace-rolling-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-inplace-rolling-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-out-of-place-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations/bp8-out-of-place-switchback.xml’. The file is in use.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable/automations’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/machine-readable’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/.tmp’. The directory is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19/log.txt’. The file is in use.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-46-19’. The directory is not empty.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-auto-inplace-rolling-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-inplace-rolling-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-out-of-place-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations/bp8-out-of-place-switchback.xml’. The file is in use.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable/automations’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/machine-readable’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/.tmp’. The directory is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50/log.txt’. The file is in use.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-12-14-56-50’. The directory is not empty.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-auto-inplace-rolling-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-inplace-rolling-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-out-of-place-automation.xml’. The file is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations/bp8-out-of-place-switchback.xml’. The file is in use.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable/automations’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/machine-readable’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/.tmp’. The directory is in use.
Failed to delete the file ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29/log.txt’. The file is in use.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan/2011-08-15-07-46-29’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs/oplan’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1/cfgtoollogs’. The directory is not empty.
Failed to delete the directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’. The directory is not empty.
Delete directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ on the local node : Failed <<<<

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on local node. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

Detach Oracle home ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ from the central inventory on the remote nodes ‘dm01db04,dm01db07,dm01db02,dm01db06,dm01db03,dm01db08,dm01db05’ : Done

Delete directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’ : Done

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on node ‘dm01db04’. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on node ‘dm01db07’. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on node ‘dm01db02’. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on node ‘dm01db06’. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on node ‘dm01db03’. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on node ‘dm01db08’. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

The Oracle Base directory ‘/u01/app/oracle’ will not be removed on node ‘dm01db05’. The directory is in use by Oracle Home ‘/u01/app/oracle/product/10.2.0/agent10g’.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory ‘/tmp/deinstall2015-08-13_03-34-40AM’ on node ‘dm01db01’
Clean install operation removing temporary directory ‘/tmp/deinstall2015-08-13_03-34-40AM’ on node ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’

## [END] Oracle install clean ##


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
Cleaning the config for CCR
Cleaning the CCR configuration by executing its binaries
CCR clean is finished
Successfully detached Oracle home ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ from the central inventory on the local node.
Failed to delete directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ on the local node.
Successfully detached Oracle home ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ from the central inventory on the remote nodes ‘dm01db04,dm01db07,dm01db02,dm01db06,dm01db03,dm01db08,dm01db05’.
Successfully deleted directory ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’.
Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

Remove the directories if not removed by the deinstall utility

orcl1@dm01db01-/u01/app/oracle/software/deinstall:

orcl1@dm01db01-/u01/app/oracle/product/11.2.0.2: su – root
Password:
[root@dm01db01 ~]# cd /u01/app/oracle/product/11.2.0.2
[root@dm01db01 11.2.0.2]# ls -ltr
total 4
drwxr-xr-x 3 oracle oinstall 4096 Aug 13 03:36 dbhome_1
[root@dm01db01 11.2.0.2]# du -sh dbhome_1/
276K    dbhome_1/
[root@dm01db01 11.2.0.2]# rm -rf dbhome_1/

[root@dm01db01 11.2.0.2]# cd ..

[root@dm01db01 product]# rmdir 11.2.0.2/

Deinstall Grid Home

Here we deinstalling GRID Home:  /u01/app/11.2.0.2/grid_1/
Replace the location with GRID Home location you are deinstalling.

[root@dm01db01 ~]# cd /u01/app/11.2.0.2/grid_1

[root@dm01db01 grid_1]# dcli -l root -g ~/dbs_group chmod -R 755 /u01/app/11.2.0.2/grid_1

[root@dm01db01 grid_1]# dcli -l root -g ~/dbs_group chown -R oracle /u01/app/11.2.0.2/grid_1

[root@dm01db01 grid_1]# cd ..

[root@dm01db01 11.2.0.2]# ls -l
total 4763768
drwxr-xr-x 72 oracle oinstall       4096 Aug 22  2012 grid_1

[root@dm01db01 11.2.0.2]# dcli -l root -g ~/dbs_group chown oracle /u01/app/11.2.0.2/grid_1

[root@dm01db01 11.2.0.2]# ls -l
total 4763768
drwxr-xr-x 72 oracle oinstall       4096 Aug 22  2012 grid_1

Navigate to deinstall directory and perform a deinstall check only. The –checkonly option we will only perform a check without deinstalling the software.

oracle@dm01db01-/home/oracle:$  cd /u01/app/oracle/software/deinstall

oracle@dm01db01-/u01/app/oracle/software/deinstall:$  ./deinstall -checkonly -home /u01/app/11.2.0.2/grid_1/
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##
The deinstall tool cannot determine the home type needed to deconfigure the selected home.  Please select the type of Oracle home you are trying to deinstall.
Single Instance database – Enter 1
Real Application Cluster database – Enter 2
Grid Infrastructure for a cluster – Enter 3
Grid Infrastructure for a stand-alone server – Enter 4
Client Oracle Home – Enter 5
Transparent Gateways Oracle Home – Enter 6
3
The product version number of the specified home cannot be determined. Is the product version at least 11.2.0.1.0 (y – yes, n – no)? [n]
y

Specify a comma-separated list of nodes on which to perform the deinstallation task:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08

Checking for existence of the Oracle home location /u01/app/11.2.0.2/grid_1
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is:
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
The following nodes are part of this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Checking for sufficient temp space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’

## [END] Install check configuration ##

Traces log file: /u01/app/oraInventory/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-54-29-AM.log

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /u01/app/oraInventory/logs/asmcadc_check2015-08-13_03-54-30-AM.log

ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0.4/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Oracle Home selected for deinstall is: /u01/app/11.2.0.2/grid_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
The home being deconfigured is NOT a configured Grid Infrastructure home (/u01/app/11.2.0.4/grid)
ASM was not detected in the Oracle Home
A log of this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-52-24-AM.out’
Any error messages from this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-52-24-AM.err’

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

Now perform the deinstall using the following command.

orcl1@dm01db01-/u01/app/oracle/software/deinstall: ./deinstall -home /u01/app/11.2.0.2/grid_1/
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##
The deinstall tool cannot determine the home type needed to deconfigure the selected home.  Please select the type of Oracle home you are trying to deinstall.
Single Instance database – Enter 1
Real Application Cluster database – Enter 2
Grid Infrastructure for a cluster – Enter 3
Grid Infrastructure for a stand-alone server – Enter 4
Client Oracle Home – Enter 5
Transparent Gateways Oracle Home – Enter 6
3
The product version number of the specified home cannot be determined. Is the product version at least 11.2.0.1.0 (y – yes, n – no)? [n]
y

Specify a comma-separated list of nodes on which to perform the deinstallation task:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08

Checking for existence of the Oracle home location /u01/app/11.2.0.2/grid_1
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is:
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0.4/grid
The following nodes are part of this cluster: dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Checking for sufficient temp space availability on node(s) : ‘dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’

## [END] Install check configuration ##

Traces log file: /u01/app/oraInventory/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2015-08-13_03-56-28-AM.log

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /u01/app/oraInventory/logs/asmcadc_check2015-08-13_03-56-29-AM.log

ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0.4/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Oracle Home selected for deinstall is: /u01/app/11.2.0.2/grid_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
The home being deconfigured is NOT a configured Grid Infrastructure home (/u01/app/11.2.0.4/grid)
ASM was not detected in the Oracle Home
Do you want to continue (y – yes, n – no)? [n]: y
A log of this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-56-01-AM.out’
Any error messages from this session will be written to: ‘/u01/app/oraInventory/logs/deinstall_deconfig2015-08-13_03-56-01-AM.err’

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /u01/app/oraInventory/logs/asmcadc_clean2015-08-13_03-56-47-AM.log
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2015-08-13_03-56-47-AM.log

De-configuring Naming Methods configuration file on all nodes…
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes…
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes…
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes…
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Failed to delete the directory ‘/u01/app/11.2.0.2/grid_1’. The directory is in use.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the local node : Failed <<<<

Failed to delete the directory ‘/u01/app/11.2.0.2/grid_1’. The directory is in use.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the local node : Failed <<<<

Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’ : Failed <<<<

Could not remove listed directories based on ‘/tmp/OraInstall2015-08-13_03-56-54-AM/installRemoveDirFile.lst’ from nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08′. [/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied/bin/rm: cannot remove directory `/u01/app/11.2.0.2/grid_1/’: Permission denied]
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db02’ : Failed <<<<

The directory ‘/u01/app/11.2.0.2/grid_1’ could not be deleted on the nodes ‘dm01db02’.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db03’ : Failed <<<<

The directory ‘/u01/app/11.2.0.2/grid_1’ could not be deleted on the nodes ‘dm01db03’.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db04’ : Failed <<<<

The directory ‘/u01/app/11.2.0.2/grid_1’ could not be deleted on the nodes ‘dm01db04’.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db05’ : Failed <<<<

The directory ‘/u01/app/11.2.0.2/grid_1’ could not be deleted on the nodes ‘dm01db05’.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db06’ : Failed <<<<

The directory ‘/u01/app/11.2.0.2/grid_1’ could not be deleted on the nodes ‘dm01db06’.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db07’ : Failed <<<<

The directory ‘/u01/app/11.2.0.2/grid_1’ could not be deleted on the nodes ‘dm01db07’.
Delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db08’ : Failed <<<<

The directory ‘/u01/app/11.2.0.2/grid_1’ could not be deleted on the nodes ‘dm01db08’.
Oracle Universal Installer cleanup completed with errors.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory ‘/tmp/deinstall2015-08-13_03-56-01AM’ on node ‘dm01db01’
Clean install operation removing temporary directory ‘/tmp/deinstall2015-08-13_03-56-01AM’ on node ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’

## [END] Oracle install clean ##


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
The home being deconfigured is NOT a configured Grid Infrastructure home (/u01/app/11.2.0.4/grid)
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the local node.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the local node.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08’.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db02’.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db03’.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db04’.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db05’.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db06’.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db07’.
Failed to delete directory ‘/u01/app/11.2.0.2/grid_1’ on the remote nodes ‘dm01db08’.
Oracle Universal Installer cleanup completed with errors.

For complete clean up of Oracle Clusterware software from the system, deinstall the following old clusterware home(s). Refer to Clusterware Install guide of respective old release for details.
    /u01/app/11.2.0.3/grid on nodes : dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

Post De-install Steps
  • Verify the OraInventory is update and OLD Oracle Homes are removed.

orcl1@dm01db01-/u01/app/11.2.0.2/grid_1: dcli -g ~/dbs_group -l oracle -s “-q” “grep ‘/u01/app/oracle/product/11.2.0.2/dbhome_1’ /u01/app/oraInventory/ContentsXML/inventory.xml”

orcl1@dm01db01-/u01/app/11.2.0.2/grid_1: dcli -g ~/dbs_group -l oracle -s “-q” “grep ‘/u01/app/11.2.0.2/grid_1’ /u01/app/oraInventory/ContentsXML/inventory.xml”

  • Verify the directories are deleted; if not manually delete them carefully.

 orcl1@dm01db01-/u01/app/oracle/software/deinstall: dcli -g ~/dbs_group -l oracle -s “-q” ‘ls -l /u01/app/oracle/product/11.2.0.2’
dm01db01: ls: /u01/app/oracle/product/11.2.0.2: No such file or directory
dm01db02: total 0
dm01db03: total 0
dm01db04: total 0
dm01db05: total 0
dm01db06: total 0
dm01db07: total 0
dm01db08: total 0
  
orcl1@dm01db01-/u01/app/oracle/software/deinstall: dcli -g ~/dbs_group -l oracle -s “-q” ‘ls -l /u01/app/11.2.0.2’
dm01db01: total 4763768
dm01db01: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:57 grid_1
dm01db02: total 4
dm01db02: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:58 grid_1
dm01db03: total 4
dm01db03: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:58 grid_1
dm01db04: total 4
dm01db04: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:58 grid_1
dm01db05: total 4
dm01db05: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:58 grid_1
dm01db06: total 4
dm01db06: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:58 grid_1
dm01db07: total 4
dm01db07: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:58 grid_1
dm01db08: total 4
dm01db08: drwxr-xr-x 2 oracle oinstall 4096 Aug 13 03:58 grid_1

orcl1@dm01db01-/u01/app/11.2.0.2/grid_1: su – root
[root@dm01db01 ~]#  dcli -g ~/dbs_group -l root -s “-q” ‘cd /u01/app/; rmdir 11.2.0.2’
[root@dm01db01 ~]# dcli -g ~/dbs_group -l root -s “-q” ‘ls -l /u01/app/11.2.0.2’
dm01db01: total 4763764
dm01db02: total 0
dm01db03: total 0
dm01db04: total 0
dm01db05: total 0
dm01db06: total 0
dm01db07: total 0
dm01db08: total 0

[root@dm01db01 ~]# dcli -g ~/dbs_group -l root -s “-q” ‘cd /u01/app/; rmdir 11.2.0.2’
[root@dm01db01 ~]# dcli -g ~/dbs_group -l root -s “-q” ‘cd /u01/app/; ls -l 11.2.0.2’
dm01db01: ls: 11.2.0.2: No such file or directory
dm01db02: ls: 11.2.0.2: No such file or directory
dm01db03: ls: 11.2.0.2: No such file or directory
dm01db04: ls: 11.2.0.2: No such file or directory
dm01db05: ls: 11.2.0.2: No such file or directory
dm01db06: ls: 11.2.0.2: No such file or directory
dm01db07: ls: 11.2.0.2: No such file or directory
dm01db08: ls: 11.2.0.2: No such file or directory
  •  Verify the free space available after de-install completed.

[root@dm01db01 ~]# dcli -g ~/dbs_group -l root ‘df -h /u01’
dm01db01: Filesystem            Size  Used Avail Use% Mounted on
dm01db01: /dev/sda3             193G   81G  103G  45% /u01
dm01db02: Filesystem            Size  Used Avail Use% Mounted on
dm01db02: /dev/sda3             193G   48G  136G  26% /u01
dm01db03: Filesystem            Size  Used Avail Use% Mounted on
dm01db03: /dev/sda3             193G   48G  136G  27% /u01
dm01db04: Filesystem            Size  Used Avail Use% Mounted on
dm01db04: /dev/sda3             193G   58G  126G  32% /u01
dm01db05: Filesystem            Size  Used Avail Use% Mounted on
dm01db05: /dev/sda3             193G   47G  137G  26% /u01
dm01db06: Filesystem            Size  Used Avail Use% Mounted on
dm01db06: /dev/sda3             193G   63G  121G  35% /u01
dm01db07: Filesystem            Size  Used Avail Use% Mounted on
dm01db07: /dev/sda3             193G   42G  142G  23% /u01
dm01db08: Filesystem            Size  Used Avail Use% Mounted on
dm01db08: /dev/sda3             193G   54G  130G  30% /u01

IMPORTANT NOTES
  • For 12.1.0.2.0, please be aware of the known issue with oraInst.loc discussed in “Deinstall Unable to Determine Product Version Number (Doc ID 1966643.1)”
  • Oracle 12c Release 1 (12.1) deinstall tool from Installation Media should not be used to deinstall earlier releases of Oracle Home as it is not supported.
  • The deinstall tool from Oracle Home should be used only if the Oracle Database 12.1 Software is being successfully Installed.
  • For Incomplete or Failed Software Installation, you must only use the deinstall tool from Installation Media.

Conclusion
In this article we have seen how to de-install Oracle Grid and RDBMS software. The external deinstall utility for 11gR2 and built in De-install utility makes the process easier to remove the unused or old Oracle Homes.


0

Overview

As we know Exadata database machine is a combination of Hardware and Software. Over a period of time these hardware and software can failure or give performance issues. In my experience I have seen hardware failures during the Exadata install and as well immediate after completion of Installation. But the newer generations of Exadata machines are more stable and you might find fewer hardware failures.

When you work with Support on hardware, software or performance issues they would request you to run the following Diagnostic utilities and uploaded the diagnostic data.

The example of hardware, software or performance issues are as follows:
  • Hardware failure: Hard disk, Flash disk, mother board, processor, DIMM and so on
  • Software issues: Operating system, firmware, Oracle software and so on
  • Performance issues: Operating system and database

In this article I will demonstrator how to execute these utilities with a live example.

Diagnostic Utilities at a Glance 
Utility Name
Description
SOSREPORT
collects detailed information about the hardware and configuration of Oracle Linux server
SUNDIAG
The utility is used for gathering hardware related information
ILOM SNAPSHOT
The utility is used for gathering hardware related information
EXAWATCHER
It collects the system data and reporting utilities. This information is mostly used for troubleshooting OS or performance issue.


Now let’s take a look at these utilities in little more detail

SOSREPORT UTILITY

SOSREPORT utility collects detailed information about the hardware and configuration of Oracle Linux server.

Steps to run SOSREPORT:
  • Log in to the compute node or storage cell as root user account for which you are running SOSREPORT (example: dm01db01) 

[root@dm01db01 ~]# id
uid=0(root) gid=0(root) groups=0(root), 1(bin), 2(daemon), 3(sys), 4(adm), 6(disk), 10(wheel)
  • You will find the sosreport utility under /usr/sbin location. You also use the Linux command “locate” to search for the utility.

[root@dm01db01 ~]# locate sosreport
/usr/sbin/sosreport
  • Execute the sosreport utility at the shell as follows

[root@dm01db01 ~]# /usr/sbin/sosreport
On the execution of this utility it will ask you for the input.
a.      Press ENTER to continue, or CTRL-C to quit.
Press ENTER on your keyboard

b.      Please enter your first initial and last name [dm01db01]:
Press ENTER to accept the default or enter a value of your choice
                       
c.       Please enter the case number that you are generating this report for:
Enter the SR number

At this time it will take a while (approximately 5-6 minutes) and generate a compressed archive file in /tmp directory.
  • Use WinScp or similar utility to copy the output file to your desktop
  • Upload the output file to Oracle Support for review.

Sample SOSREPORT Run:

[root@dm01db01 ~]# locate sosreport
/usr/sbin/sosreport

[root@dm01db01 ~]# /usr/sbin/sosreport

sosreport (version 2.2)

   This command will collect diagnostic and configuration
information from this Oracle Linux system and installed
applications.

  An archive containing the collected information will be
generated in /tmp and may be provided to a Oracle USA
support representative.

  Any information provided to Oracle USA will be treated
in accordance with the published support policies at:

   https://linux.oracle.com/

  The generated archive may contain data considered
sensitive and its content should be reviewed by the
originating organization before being passed to any third
party.

  No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [dm01db01]:
Please enter the case number that you are generating this report for [None]:  3-1386095xxxx

  Running plugins. Please wait …

  Completed [66/66] …
Creating compressed archive…

Your sosreport has been generated and saved in:
  /tmp/sosreport-dm01db01.3-1386095-20161230023010-9f83.tar.xz

The md5sum is: b1ccc01a773cbd36d463ba07b57c9f83

Please send this file to your support representative.
[root@dm01db01 ~]#


SUNDIAG UTILITY

The utility is used for gathering hardware related information. Oracle Support uses this diagnostic data to assess the hardware failure.

Steps to run SUNDIAG report:

Follow the steps listed below to run the sundiag.sh utility.
  •  Log in to the compute node or storage cell as root user account for which you are running       SUNDIAG (example: dm01db01)
  • You will find the sundiag utility under /opt/oracle.SupportTools location. You also use the Linux           command “locate” to search for the utility.

[root@dm01cel01 ~]# locate sundiag
/opt/oracle.SupportTools/sundiag.sh
  • Run the sundiag.sh utility

 [root@dm01cel01 ~]# /opt/oracle.SupportTools/sundiag.sh
  • Use WinScp or similar utility to copy the output file to your deskto
  • Upload the output file to Oracle Support for review.

Sample sundiag Run:

[root@dm01cel01 ~]# locate sundiag
/opt/oracle.SupportTools/sundiag.sh

 [root@dm01cel01 ~]# /opt/oracle.SupportTools/sundiag.sh

Oracle Exadata Database Machine – Diagnostics Collection Tool

Last alert date is beyond 7 days. Skipping OSW/Metrics collection
Gathering Linux information

Skipping collection of OSWatcher/ExaWatcher logs, Cell Metrics and Traces
Skipping ILOM collection. Use the ilom or snapshot options, or login to ILOM
over the network and run Snapshot separately if necessary.

/var/log/exadatatmp/sundiag_dm01cel01_1605NM70AD_2016_12_30_02_30
Gathering Cell information

==============================================================================
Done. The report files are bzip2 compressed in /var/log/exadatatmp/sundiag_dm01cel01_1605NM70AD_2016_12_30_02_30.tar.bz2
==============================================================================

If you read the output carefully, sundiag utility doesn’t collect the ILOM data and Exawatcher data. That is the reason we need to run separate utilities to gather these data.

ILOM SNAPSHOT

This data is required to troubleshoot a hardware issue by Oracle Support.
Follow the steps listed below to run the snapshot utility in GUI.

Using GUI Interface:
  • For ILOM 2.x and 3.0
  • Open a web browser (use something other than Internet Explorer) and enter the following address

Note:  You may see complaints about security – ignore or override – click I understand the risks/Add exception/Confirm Security Exception
  • Enter root as User Name and its password and click on Log In. This will take you to the Home screen.

  • Select Maintenance -> Snapshot.  (ILOM 2.x and 3.0)

 The Service Snapshot Utility page appears.
ILOM 2.x and 3.0 will look similar to this:
  • From the above Screen, Select Data Set “Normal”, Select Transfer Method as “Browser” and Click “Run”.

Normal – Specifies that ILOM, operating system, and hardware information is collected.
The download file will be saved according to your browser settings.
Important Note:  Do not enable this option:Collect Only Log Files from Data Set‘.  Doing so will limit the snapshot to a much smaller sub-section of log files.
  •  In the dialog box, specify the directory to which to save the file and the file name.

Click OK.
The file is saved to the specified directory.
  • For ILOM version 3.1

If the ILOM version is 3.1 which is the latest version shipped with X3/X4 Exadata. There a little difference in the design.
  • Open a web browser (use something other than Internet Explorer) and enter the following address

Note:  You may see complaints about security – ignore or override – click I understand the risks/Add exception/Confirm Security Exception.
  • Enter root as User Name and its password and click on Log In. This will take you to the Home Screen.

  • Select ILOM Administration -> Maintenance -> Snapshot (ILOM 3.1)

The Service Snapshot Utility page appears.
ILOM 3.1 will look similar to this:
  • Select Data Set “Normal”, Select Transfer Method as “Browser” and Click “Run”.

Normal – Specifies that ILOM, operating system, and hardware information is collected.
The download file will be saved according to your browser settings.
Important Note:  Do not enable this option:Collect Only Log Files from Data Set‘.  Doing so will limit the snapshot to a much smaller sub-section of log files.
  • In the dialog box, specify the directory to which to save the file and the file name.

Click OK.
The file is saved to the specified directory.

  • Using CLI

Follow the steps listed below to run the snapshot utility in command line.

  • Log in to the ILOM CLI interface

[root@dm01db01 ~]# ssh dm01db01-ilom
Password:

  • You will see a similar output

Oracle(R) Integrated Lights Out Manager
Version 3.0.16.15.j r101695
Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.
->
  • After the ‘->’ prompt, type the command in below:

-> set /SP/diag/snapshot dataset=normal
Set ‘dataset’ to ‘normal’
  • Type the following command:

-> set /SP/diag/snapshot dump_uri=sftp://root:welcome@10.10.10.51/tmp
Set ‘dump_uri’ to ‘sftp://root:welcome@10.10.10.51/tmp’
  • Next cd to the snapshot directory and view the status:

-> cd /SP/diag/snapshot
/SP/diag/snapshot

-> show

 /SP/diag/snapshot
    Targets:

    Properties:
        dataset = normal
        dump_uri = (Cannot show property)
        encrypt_output = false
        result = Running

    Commands:
        cd
        set
        show

->

Wait for the snapshot process to complete. It may take several minutes.
Continue to check until the status is shows ‘Snapshot Complete’
Do not use, access, view, copy or move the snapshot file until it has completed.

-> show

 /SP/diag/snapshot
    Targets:

    Properties:
        dataset = normal
        dump_uri = (Cannot show property)
        encrypt_output = false
        result = Collecting data into sftp://root:*****@10.10.10.51/tmp/dm01db01-ilom_10.10.23.56_2016-12-29T08-44-09.zip
Snapshot Complete.
Done.

    Commands:
        cd
        set
        show
  •  You can now exit the CLI interface and find your snapshot in the directory you specified.

-> exit
Connection to dm01db01-ilom closed.
  • The file name will look similar to this example:

 dm01db01-ilom_10.10.10.56_2016-12-29T08-44-09.zip
Do not rename the snapshot file.

Exawatcher
The /opt/oracle.ExWatcher directory contains the Oracle ExaWatcher system data gathering and reporting utilities. This information is mostly used for troubleshooting OS or performance issue.

Steps for ExaWatcher collection:
  •  Navigate to the Exawatcher directory and execute the GetExawatcherResults.sh script.

[root@dm01db01 ~]#  cd /opt/oracle.ExaWatcher/

[root@dm01db01 oracle.ExaWatcher]# ls -ltr GetExaWatcherResults.sh
-rwx—— 1 root root 21012 Oct 21  2015 GetExaWatcherResults.sh

[root@dm01db01 oracle.ExaWatcher]# ./GetExaWatcherResults.sh -h
Usage:
  ./GetExaWatcherResults.sh {–from $FromTime [–to $ToTime] | –at $AtTime [–range $Hours]}
                                               [–archivedir $ArchiveDir]
                                               [–scp $UserName@SrvName]
                                               [–filter $SamplerName]
                                               [–resultdir $ResultDir]

  • To collect from/to a certain date and time:

# ./GetExaWatcherResults.sh –from 07/31/2015_00:00:00 –to 07/31/2015_23:00:00
                                                   mm/dd/yyyy hh:mm:si

Default output location: /opt/oracle.ExaWatcher/archive/ExtractedResults

[root@dm01cel07 ExtractedResults]# cd /opt/oracle.ExaWatcher/archive/ExtractedResults
  • To collect for a time range. In this case, we are collecting for 4 hrs before and after 1300:

# ./GetExaWatcherResults.sh –at 08/05/2015_13:00:00 –range 4
                                                 mm/dd/yyyy hh:mm:si

The default archive directory is /opt/oracle.ExaWatcher/archive/ExtractedResults; however, you can change this using [-d|–archivedir] flag:
Example of changed default archive location to /tmp/ExaWatcherArchive:

# ./GetExaWatcherResults.sh –from 01/25/2014_13:00:00 –to 01/25/2014_14:00:00 –archivedir /tmp/ExaWatcherArchive


Conclusion
In this article we have seen different Exadata diagnostic utilities and how to execute them to collect the diagnostic data. These utilities are used on a daily basis to assess the hardware, software and performance issues.


1


Overview

This Blog contains steps to perform upgrading Oracle GoldenGate from 12.1.2.1.0 to 12.1.2.1.7. This chapter contains minimal upgrade that deploys only the updated core functionality, it is best practice to perform a minimal upgrade first, so that you can trouble shoot more easily in the event that any problems arise.

Environment details

Operating system
Windows server 2012 R2
Environment
production
Oracle Home
D:oracle11204product11.2.0dbhome_1
Golden Gate stage
D:oraclegg_stage
Golden Gate Home
D:oraclegg_homeproduct12.1.2.1ggfor11
OGG Source version
12.1.2.1.0
OGG Target Version
12.1.2.1.7

Steps to perform Oracle Golden Gate upgrade.

1.   Download patch p21078123_121217_MSWIN-x86-64 and unzip it.

After unzip it creates folder 21078123

2.   Login to server and Check Optach version.

D:>cd %oracle_home%OPatch

D:oracle11204product11.2.0dbhome_1OPatch>

D:oracle11204product11.2.0dbhome_1OPatch>opatch version

OPatch Version: 11.2.0.3.4



OPatch succeeded.



D:oracle11204product11.2.0dbhome_1OPatch>



 3.      Set the ORACLE_HOME environment variable to the Oracle GoldenGate installation directory, which is referred to as ORACLE_HOME.

D:>echo %oracle_home%
D:oracle11204product11.2.0dbhome_1

D:>set oracle_home=D:oraclegg_homeproduct12.1.2.1oggfor11

D:>echo %oracle_home%
D:oraclegg_homeproduct12.1.2.1oggfor11



  4. Set the PATH environment variable to include the location of the unzip executable.

D:>echo %path%
D:oracle11204product11.2.0dbhome_1bin;C:Windowssystem32;C:Windows;C:WindowsSystem32System32WindowsPowerShellv1.0
D:>set path=%path%:D:oracle11204product11.2.0dbhome_1OPatch

D:>echo %path%
D:oracle11204product11.2.0dbhome_1bin;C:Windowssystem32;C:Windows;C:WindowsSystem32System32WindowsPowerShellv1.0:D:oracle11204product11.2.0dbhome_1OPatch

D:>

5.      Verify the Oracle Inventory, which OPatch accesses to install the patches. To verify the inventory, run the following command.

D:>cd D:oracle11204product11.2.0dbhome_1OPatch

D:oracle11204product11.2.0dbhome_1OPatch>
D:oracle11204product11.2.0dbhome_1OPatch>opatch lsinventory



6.      Go to staging directory where you unzip then got patch number directory and execute opatch apply.

cd D:oraclegg_stagep21078123_121217_MSWIN-x86-6421078123

D:>cd D:oraclegg_stagep21078123_121217_MSWIN-x86-6421078123

D:oraclegg_stagep21078123_121217_MSWIN-x86-6421078123>D:oracle11204product11.2.0dbhome_1OPatchopatch apply



7.      Verify the version.

goto gg_home directory and type ggsci.exe
D:>cd oraclegg_homeproduct12.1.2.1oggfor11

D:oraclegg_homeproduct12.1.2.1oggfor11>ggsci



Conclusion

In this Blog we have performed Oracle Golden Gate upgradation from 12.1.2.1.0 to 12.1.2.1.7 on Windows operating system. We have seen upgrading Oracle Golden Gate Software on Windows is very simple and straight forward using OPatch Utility.

About Author

Name: Mirza Hidayathullah Baig
Designation: Senior Database Engineer
Organization: NetSoftMate IT Solutions.



0



Exadata was first launched in the year 2008 and it was names as Exadata V1. Since then Exadata has undergone many significant changes.

In this article I will give a quick overview of Exadata history and changes made in each of the model.

         V1
     First Exadata model released in 2008
     It was a combination of HP hardware + Oracle Software
     Exclusively for Data Warehouse workload
     There were too many hardware failures noticed in this model
     Working with Hardware issues was not easy as it involved multiple vendors
     There was no Flash in this version
     Major problem with V1 was the cabinet gets overheated. You can fry Eggs on the cabinetJ

         V2
     V2 was released in 2009
     Combination of Sun hardware and Oracle Software
     Flash Cache was introduced for the first time in V2
     Caches data intelligently on Flash Cache
     Faster components and used solid state disks
     More than just Data Warehouse

             X2
     X2 is the third version of Exadata and it was released in 2010
     This time the naming changed to X series
     Introduced two variants X2-2 and X2-8
     Introduced Exadata Storage Expansion for addition storage capacity
     Faster components and increased capacity (CPU, memory and disk)

         X3
     X3 was released in 2012
     Named as “In-Memory Database machines”
     Introduced Exadata Eight Rack configuration
     Increased Flash cache
     Introduced write back flash cache
     Faster components and increased capacity
     Increased High Capacity disk Size from 2TB to 3TB
     Introduced new Software feature Write-Back Flash Cache


         X4
     It was released in 2013
     Introduced Xeon processors
     Physical memory can be expanded up to 512GB on Compute nodes
     Doubled Flash Cache
     Increased High Performance disk size from 600GB to 1.2TB
     Increased High Capacity disk size from 3TB to 4TB
     Introduced Active/Active Infiniband network connection
  
         X5
     It was released in 2015
     Uses 2-socket CPU
     Physical memory can be expanded up to 768GB on Compute nodes
     Introduced OVM
     Introduced Extreme Flash Cache servers
     Introduced Elastic Exadata configuration
     High performance disk are removed in favor of Extreme Flash Cache servers
     Doubled flash cache and used NVMe


         X6
     X6 was released in 2016
     Increase Flash cache
     Faster components and increased capacity
     Also available in Public Cloud
     Ideal for DBaas

The diagram below gives you an overview of the Exadata hardware evolution.

Conclusion


In this article we have learned the history of Exadata. We have seen how Exadata has undergone significant changes since it’s launched in the year 2008. 
3

What is Exadata Database Machine?

Exadata Database Machine or simply known as Exadata is:
  •           An Engineered System
  •           A pre-configured combination of balanced Hardware and unique software
  •           A unique platform for running Oracle Databases
  •           Consists of Compute Grid, Storage Grid and Network Grid
  •           A fully integrated platform for Oracle Database
  •           Ideal for Database Consolidation platform
  •           It provides High Availability and High Performance for all types of Workloads

The Oracle Exadata Database Machine is an Engineered System designed to deliver extreme performance and high availability for all type of Oracle database workloads (OLTP, OLAP & Mixed Workload).

The Oracle Exadata vision is to deliver dramatically better performance for all database workloads.

Why Exadata?

  •           It provides Extreme Performance and High Availability
  •           Exadata is a flash optimized Oracle database platform by providing intelligent caching algorithms to cache *appropriate* data in flash Cache
  •           For large Data Warehouse databases, it addresses the inability to move large volumes of database from the storage to database servers
  •           For OLTP, it supports larger user population and transaction volumes and provides faster response time
  •           It supports mixed workload in a Consolidation environment and prioritizes workload


Exadata Features

Exadata provides unique performance features such as:
  •           Smart Scan
  •           Smart Flash Cache
  •           Smart Flash Logging
  •           Storage Indexes
  •           I/O Resource Management (IORM)
  •           Hybrid Columnar Compression (EHCC)


Exadata Hardware and Software Components

Exadata Database Machine consists of a storage grid, compute grid, and network grid. Each grid, or hardware layer, is built with multiple high-performing, industry-standard Oracle servers to provide hardware and system fault tolerance.

Exadata Storage Server Hardware & Software

         High Performance storage for Oracle Database
         64 bit Intel-based sun Fire Servers
         Contains either 12 hard disks or 8 flash disks
         Preinstalled Software:
     Exadata Storage Server Software
     Oracle Linux x86_64
     Drivers and utilities
         Only available in conjunction with Database Machines

Exadata Compute Node Hardware & Software

         X5-2 database servers are based on the Intel Sun Fire.
         Each server has 2×18-core Intel Xeon Processors and 256GB of memory.
         Can run Oracle Linux x86_64 or Oracle Solaris.
         Oracle Linux is supported on 2-socket and 8 socket database servers.
         Oracle Solaris is supported only on 2-socket database servers
         12.1.1.1.1 is the terminal version that supports database servers running Oracle Solaris. 
         Run Grid Infrastructure and RDBMS software

       Infiniband Switch

         One of the more important hardware components of Exadata is the InfiniBand network.
         It is used for transferring data between the database tier and the storage tier.
         It is also used for interconnect traffic between the database servers, if they are configured in a RAC cluster.
         In addition, the InfiniBand network may be used to connect to external systems for such uses as backups.

Exadata Configurations

         Eight Rack
     2 Compute node, 3 Storage Cells and 2 Infiniband Switch
         Quarter Rack
     2 Compute node, 3 Storage Cells and 2 Infiniband Switch
         Half Rack
     4 Compute node, 7 Storage Cells and 2 Infiniband Switch
         Full Rack
     8 Compute node, 14 Storage Cells and 2 Infiniband Switch
         Elastic Configuration
     2 – 9 Compute nodes and 3 – 18 Storage cells per Rack

The difference between Eight and Quarter is that, the Eight Rack comes with half of the capacity compared to Quarter Rack. In Eight Rack half of the processor and Disks are disabled.

Exadata Benefits

         High performance
         High availability
         Scalability
         Manage any kind of workload
         Easy to deploy
         Ideal for Consolidation
         Virtualization
         Available on-premises and cloud

Exadata Rack

This image gives an overview of Exadata X5-2 database machine.


Individual Exadata Components


Exadata Storage Cell

Exadata Compute node


Exadata Infiniband Switch


Conclusion

In this article we have learned about Exadata Database Machine and its components. Exadata truly provides extreme performance and highly availability using its faster and balanced hardware and unique Exadata storage software.



1



Overview:
Oracle Golden Gate provides very fast replication of data by reading transaction logs and writing the changes to one or more target databases in a homogeneous and heterogeneous environment. It is useful for High Availability Architectures and especially for Data Warehouse and Decision Support Systems. Thus, the variety of techniques and methods spreads from unidirectional environments for query offloading/reporting to bidirectional or Peer-to-Peer architectures in an active-active fashion.


Environment Details
 
Hostname
PROD-DB1
Ip Address
10.78.101.3
Operating system
Windows server 2012 R2
Environment
production
Oracle Home
D:oracle11204product11.2.0dbhome_1
Golden Gate stage
D:oraclegg_stage
Golden Gate Home
D:oraclegg_homeproduct12.1.2.1ggfor11
OGG Processes
e_shl01s,p_shl01s




Steps to Install Oracle GoldenGate Software



 1. Create Oracle Golden Gate staging directory and home directory.


D:>cd oracle


D:oracle>md gg_stage gg_home

D:oracle>cd gg_home

D:oraclegg_home>md product

D:oraclegg_home>cd product

D:oraclegg_homeproduct>md 12.1.2.1

D:oraclegg_homeproduct>cd 12.1.2.1

D:oraclegg_homeproduct12.1.2.1>md oggfor11



2.      Copy the media files to the Server.



3.      Download and Install Microsoft visual c++


 4.      Unzip the Golden Gate media and click on setup.exe file to initiate installation.



5.      Choose on which database version you installing golden gate.



6.      Provide Oracle Golden Gate Home and Oracle Database Home location and port number.



7.      Verify Summary and click on Install. 


Progress Window

 


8.      Installation successful then click close.




9.      Once Installation done, login to Golden gate from gg home and check the version.





Conclusion: 

In this Article we have performed Oracle Golden Gate Installation on Windows operating system. We have seen Installing Oracle Golden Gate Software on Windows is very simple and straight forward using Graphical Interface.

About Author

Name: Mirza Hidayathullah Baig
Designation: Senior Database Engineer
Organization: NetSoftMate IT Solutions

2

NO OLD POSTSPage 18 of 18NEXT POSTS