Netsoftmate

Tag: Exadata

  • Exadata: How To Mount diag.iso On An Exadata X5 Server From A NFS Server

    Introduction


    I was testing Exadata Compute node snapshot restore procedure on an Exadata X5-2. As per of testing I had to start the machine in special mode using diag.iso, the easiest option is to use WEB ILOM to mount he diag.iso, but for some technical reason or BUG I was unable to mount diag.iso using WEB ILOM. Oracle provided a procedure to mount the diag.iso image using an NFS server.


    In this article I will demonstrate step by step procedure on how mount diag.iso image using NFS Server along with the output.




    Steps to mount diag.iso on an Exadata Compute node from NFS server:


    • First login to the Compute node ILOM CLI or SSH from another Compute node 

    [root@dm01db01 ~]# ssh dm01db01-ilom
    Password:


    Oracle(R) Integrated Lights Out Manager


    Version 3.2.8.24 r114580


    Copyright (c) 2016, Oracle and/or its affiliates. All rights reserved.


    Warning: HTTPS certificate is set to factory default.


    Hostname: dm01db01-ilom


    ->


    • Execute the following command to mount diag.iso from a NFS server

    Syntax:
    set /SP/services/kvms/host_storage_device/remote/ server_URI=NFS_or_Samba_URI_file_location




    -> set /SP/services/kvms/host_storage_device/remote server_URI=nfs://10.10.10.1:/export/dm01/backup1/diag.iso
    Set ‘server_URI’ to ‘nfs://10.10.10.1:/export/dm01/backup1/diag.iso’


    • Verify the server_URI to make it is poing to correct NFS location

    -> show /SP/services/kvms/host_storage_device/remote server_URI


      /SP/services/kvms/host_storage_device/remote
        Properties:
            server_URI = nfs://10.10.10.1:/export/dm01/backup1/diag.iso


    • Execute the following command to Enable storage redirection:

    -> set /SP/services/kvms/host_storage_device/ mode=remote
    Set ‘mode’ to ‘remote’


    • Check the status of redirection:

    -> show /SP/services/kvms/host_storage_device/ status


      /SP/services/kvms/host_storage_device
        Properties:
            status = operational


    Note –  Redirection is active if the status is set to either Operational or Connecting.


    • Now set the next boot device to cdrom

    -> set /HOST boot_device=cdrom
    Set ‘boot_device’ to ‘cdrom’


    • Verify the next boot device is pointing to cdrom:

    -> show /HOST


     /HOST
        Targets:
            console
            diag
            provisioning


        Properties:
            boot_device = cdrom
            generate_host_nmi = (Cannot show property)


        Commands:
            cd
            set
            show


    • Execute the following command to Reboot the Server:

    -> reset /SYS
    Are you sure you want to reset /SYS (y/n)? y
    Performing hard reset on /SYS


    At this time the server boot from diag.iso, enter into interactive mode, restore/recovery the machine or correct the OS configuration that you have and when you are done, disable redirection.





    Steps to Disable steps Server_URI and Storage redirection


    • Execute the following command to disable Storage redirection:

    -> set /SP/services/kvms/host_storage_device/ mode=disabled
    Set ‘mode’ to ‘disabled’


    -> show /SP/services/kvms/host_storage_device/ mode


      /SP/services/kvms/host_storage_device
        Properties:
            mode = disabled


    • Execute the following command to disable Server_URI:

    -> set /SP/services/kvms/host_storage_device/remote server_URI=”
    Set ‘server_URI’ to ”


    -> show /SP/services/kvms/host_storage_device/remote server_URI


      /SP/services/kvms/host_storage_device/remote
        Properties:
            server_URI = (none)


    • Verify the Next boot device:

    -> show /HOST


    /HOST
        Targets:
            console
            diag
            provisioning


        Properties:
            boot_device = default
            generate_host_nmi = (Cannot show property)


        Commands:
            cd
            set
            show


    • Reboot the server to boot with the normal Exadata kernel

    -> reset /SYS
    Are you sure you want to reset /SYS (y/n)? y
    Performing hard reset on /SYS




    Conclusion


    In this article we have learned how to mount diag.iso on Exadata compute node from NFS Server. If the WEB ILOM is not working for some reason then mounting the diag.iso using NFS server comes handy. The process to mount is diag.iso from NFS Server is very simply and much faster.

  • Exadata Database Machine X7-2 – First look

    Exadata Database Machine X7-2


    On October 2nd at Oracle Open World, Oracle announced the next-generation Oracle Exadata Database Machine X7 with significant hardware and software enhancements in overall performance, storage capacity, network bandwidth, and automation. Exadata X7 delivers extreme performance and reliability to run the largest, most business-critical database workloads.


    Exadata X7-2 quick glance:

    • Latest Intel Xeon Scalable Processors (384 core in a Full Rack)
    • 50% increase in Physical memory (upto 12TB in a full Rack)
    • 100% increase in the capacity of Extreme Flash storage (716.8TB in a full Rack)
    • 25% increase in disk capacity (1680TB in a full Rack)
    • A full rack Exadata X7 system has: 
      • Raw capacity of 1.7 petabytes of disk storage or
      • 720 terabytes of NVMe all-Flash storage.
    • Ethernet network bandwidth has increased by 150 percent from 10 to 25Gb/s

    Exadata X7-2 pictorial representation


    For detail on Exadata X7-2 and it’s models visit:

    https://www.oracle.com/corporate/pressrelease/oow17-oracle-exadata-x7-100217.html

    http://www.oracle.com/technetwork/database/exadata/exadata-x7-2-ds-3908482.pdf

  • Different Oracle Health Check Tools

    Introduction

    There are quite a few health check tools provided by Oracle
    for both Engineered and Non-Engineered Systems. This article explains which
    tool is a best choice for a given system.


    Table showing different Health Check Tools available for
    Engineered and Non-Engineered Systems



























    ORAchk (1268927.2)

    Exachk

    Oracle Database Appliance (ODA)

    Exadata Database Machine (1070954.1)

    Non-Engineered Systems

    Exalogic (1449226.1)

     

    Exalytics (1566134.1)

     

    BIG Data Appliance  (BDA)  (1643715.1)

     

    Zero Data Loss Recovery Appliance (ZDLRA) (1643715.1)


    Examples executing
    Health Check utilities


    ORAchk
     
    Download ORAchk using MOS – ORAchk – Health
    Checks for the Oracle Stack (Doc ID 1268927.2)
     
    Non-Engineered
    Systems
     
    Run ORAchk Interactively.

    • Log in to the system as root user
       
    • Stage the appropriate orachk.zip kit in its own
      directory the node on which the tool will be executed. Eg:
      /u01/app/oracle/stage
       
    • Unzip orachk.zip kit, leaving the script and
      driver files together in the same directory
      # unzip orachk.zip –d /tmp/orachk
       
    • Validate the permissions for orachk are 755
      (-rwxr-xr-x). If the permissions are not currently set to 755, set the
      permissions on orachk as follows:
      #cd /tmp/orachk
      # chmod 755 orachk
       
    • Invoke the tool as follows:
      #cd /tmp/orachk
      #./orachk
       
      Follow the prompts while reading and
      understanding all messages.
       
    • Upon completion of ORAchk command the following
      (or similar) will be displayed:
    Detailed report (html) –
    /home/oracle/orachk_oradbnode1_orcl_100715_105241/orachk_oradbnode1_orcl_100715_105241.html
     
    Engineered
    System – Oracle Database Appliance (ODA)
     
    # /opt/oracle/oak/orachk -a

     
    Exachk

    Exadata Database Machine
     
    Download Exachk using MOS Oracle Exadata
    Database Machine exachk or HealthCheck (Doc ID 1070954.1)
     
    # ./exachk -a

     
    Conclusion

    In this article we have seen different Oracle
    Health Check tool available and how to use them.

  • Resize Root File System On Exadata Compute Node

    Introduction

    By default Oracle ACS configure root file system with 30GB space on Exadata Computed nodes X2 and above. In most of the cases this space is sufficient to store operating system, Exadata software, log file and diagnostic files. Over time if you store patches, software and log files are not purged this space will be filled faster. Exadata X2 and above uses volume group and it is easy to extend the logical volume space on which the root file system is mounted.
    Root file system is created on two system partitions LVDb Sys1 and LVDb Sys2 and both system partitions must be size equally at the same time. Only one system partition is active at any time and other is inactive.
    In this article, I will demonstrate how you can extend root file system size on Exadata Compute nodes online without any downtime.

    Environemt

    Exadata X5-2 Half Rack
    Exadata storage software version 12.1.2.3.4

    Current Root File System Allocation

    [root@exa01db01 ~]# df -h /
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VGExaDb-LVDbSys1
                           30G   25G  3.5G  88% /

     

    List Logical Volume and It’s Details

    lvm> lvs
    lvm> lvs -o lv_name,lv_path,vg_name,lv_size
      LV        Path                   VG      LSize
      LVDbOra1  /dev/VGExaDb/LVDbOra1  VGExaDb 200.00g
      LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb  24.00g
      LVDbSys1  /dev/VGExaDb/LVDbSys1  VGExaDb  30.00g
      LVDbSys2  /dev/VGExaDb/LVDbSys2  VGExaDb  30.00g
      perflv    /dev/VGExaDb/perflv    VGExaDb   5.00g


    Get the Current Active System Partition

    [root@exa01db01 ~]# imageinfo


    Kernel version: 2.6.39-400.294.1.el6uek.x86_64 #1 SMP Wed Jan 11 08:46:38 PST 2017 x86_64
    Image kernel version: 2.6.39-400.294.1.el6uek
    Image version: 12.1.2.3.4.170111
    Image activated: 2017-04-08 12:14:23 -0500
    Image status: success
    System partition on device: /dev/mapper/VGExaDb-LVDbSys1


    Steps to Increase Root File System on Compute Nodes:

    • Get the Current Root File System Utilization

    [root@exa01db01 ~]# df -h /

    Filesystem            Size  Used Avail Use% Mounted on

    /dev/mapper/VGExaDb-LVDbSys1
                           30G   25G  3.5G  88% /
    • Get Current Logical Volume Configuration

    lvm> lvs -o lv_name,lv_path,vg_name,lv_size

      LV        Path                   VG      LSize

      LVDbOra1  /dev/VGExaDb/LVDbOra1  VGExaDb 200.00g
      LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb  24.00g
      LVDbSys1  /dev/VGExaDb/LVDbSys1  VGExaDb  30.00g
      LVDbSys2  /dev/VGExaDb/LVDbSys2  VGExaDb  30.00g
      perflv    /dev/VGExaDb/perflv    VGExaDb   5.00g

    • Ensure Root File System Can be Resized Online

    [root@exa01db01 ~]# tune2fs -l /dev/mapper/VGExaDb-LVDbSys1 | grep resize_inode

    Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
    All Nodes:


    [root@exa01db01 ~]# dcli -g dbs_group -l root ‘tune2fs -l /dev/mapper/VGExaDb-LVDbSys1 | grep resize_inode’exa01db01: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isizeexa01db02: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isizeexa01db03: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isizeexa01db04: Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
    • Get the free space available in the Volume Group

    lvm> vgdisplay -s

      “VGExaDb” 1.63 TiB  [295.00 GiB used / 1.34 TiB free]

    • Extend both logical volumes using lvextend command. Here we are extending the root file system by 50GB, so the file system become 80GB in total.

    [root@exa01db01 ~]# lvextend -L +50G /dev/VGExaDb/LVDbSys1

      Size of logical volume VGExaDb/LVDbSys1 changed from 30.00 GiB (7680 extents) to 80.00 GiB (20480 extents).

      Logical volume LVDbSys1 successfully resized
    [root@exa01db01 ~]# lvextend -L +50G /dev/VGExaDb/LVDbSys2

      Size of logical volume VGExaDb/LVDbSys2 changed from 30.00 GiB (7680 extents) to 80.00 GiB (20480 extents).

      Logical volume LVDbSys2 successfully resized
    • Now resize the file system using resize2fs command.

    [root@exa01db01 ~]# resize2fs /dev/VGExaDb/LVDbSys1

    resize2fs 1.43-WIP (20-Jun-2013)

    Filesystem at /dev/VGExaDb/LVDbSys1 is mounted on /; on-line resizing required
    old_desc_blocks = 2, new_desc_blocks = 4
    Performing an on-line resize of /dev/VGExaDb/LVDbSys1 to 15728640 (4k) blocks.
    The filesystem on /dev/VGExaDb/LVDbSys1 is now 15728640 blocks long.
    [root@exa01db02 ~]# e2fsck -f /dev/VGExaDb/LVDbSys1

    e2fsck 1.43-WIP (20-Jun-2013)

    /dev/VGExaDb/LVDbSys1 is mounted.
    e2fsck: Cannot continue, aborting.


    The resize command ro LVDbSys2 is failed as it is inactive. So we must execute the fsck first before resizing.

    [root@exa01db01 ~]# resize2fs /dev/VGExaDb/LVDbSys2

    resize2fs 1.43-WIP (20-Jun-2013)

    Please run ‘e2fsck -f /dev/VGExaDb/LVDbSys2’ first.
    [root@exa01db01 ~]# e2fsck -f /dev/VGExaDb/LVDbSys2

    e2fsck 1.43-WIP (20-Jun-2013)

    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/VGExaDb/LVDbSys2: 122199/3932160 files (0.3% non-contiguous), 5496667/7864320 blocks

    Now execute the resize file system again

    [root@exa01db01 ~]# resize2fs /dev/VGExaDb/LVDbSys2

    resize2fs 1.43-WIP (20-Jun-2013)

    Resizing the filesystem on /dev/VGExaDb/LVDbSys2 to 15728640 (4k) blocks.
    The filesystem on /dev/VGExaDb/LVDbSys2 is now 15728640 blocks long.

    • Validate the root file system

    [root@exa01db01 ~]# df -h /

    Filesystem            Size  Used Avail Use% Mounted on

    /dev/mapper/VGExaDb-LVDbSys1
                           80G   25G   55G  31% /

     

    Conclusion

    In this article we have demonstrated how to resize root file system on Exadata Compute node online without any outage. It is important to note that root file system is create on two system partitions for high availability. 

  • Configure ACFS on Exadata running Oracle 12.2.0.1 on Flex ASM Architecture

    Overview

    In one of previous article I have demonstrated how to configure ACFS on Exadata database running Oracle database 12.1.0.2 using standard ASM. Now I have an opportunity to configure Oracle GoldenGate on Exadata running Oracle database 12.2.0.1. So I have decides to configure ACFS on Exadata running Oracle database 12.2.0.1 using ASM Flex Architecture.

    In this article I will demostrate how to configure Oracle ACFS on Exadata Database machine running Oracle  Database 12.2.0.1 using Flex ASM.

    For details on ACFS and how to configure ACFS on Standard ASM look at my previous article at:
    http://netsoftmate.blogspot.in/2017/02/configure-acfs-on-exadata-database.html

    Prerequisites

    • Verify ASM configuration on Standard Cluster (8 nodes)

    Here the owner of the Grid Home is oracle user.

    dm01db01-+ASM1 {/home/oracle}:id oracle
    uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),101(fuse),1002(dba),1003(oper),1004(asmdba)

    • Determine the ASM architecture

    Here it is an ASM Flex configuration using the default cardinally=3 and the ASM instances are running on node 1, 2 and 4 out of 8 nodes:

    dm01db01-+ASM1 {/home/oracle}:asmcmd showclustermode
    ASM cluster : Flex mode enabled

    dm01db01-orp258c1 {/home/oracle}:srvctl config asm -detail
    ASM home: <CRS home>
    Password file: +DBFS_DG/orapwASM
    Backup of Password file:
    ASM listener: LISTENER
    ASM is enabled.
    ASM is individually enabled on nodes:
    ASM is individually disabled on nodes:
    ASM instance count: 3
    Cluster ASM listener: ASMNET1LSNR_ASM

    dm01db01-+ASM1 {/home/oracle}:srvctl status asm
    ASM is running on dm01db04,dm01db02,dm01db01
    ASM is enabled.

    • Verify if ACFS/ADVM modules is enabled/loaded on every node in a standard cluster or every Hub Node in a Flex Cluster.

    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘lsmod | grep oracle’
    dm01db01: oracleacfs           4609822  3
    dm01db01: oracleadvm            803161  17
    dm01db01: oracleoks             660948  2 oracleacfs,oracleadvm
    dm01db02: oracleacfs           4609822  3
    dm01db02: oracleadvm            803161  17
    dm01db02: oracleoks             660948  2 oracleacfs,oracleadvm
    dm01db03: oracleacfs           4609822  3
    dm01db03: oracleadvm            803161  17
    dm01db03: oracleoks             660948  2 oracleacfs,oracleadvm
    dm01db04: oracleacfs           4609822  3
    dm01db04: oracleadvm            803161  17
    dm01db04: oracleoks             660948  2 oracleacfs,oracleadvm
    dm01db05: oracleacfs           4609822  3
    dm01db05: oracleadvm            803161  17
    dm01db05: oracleoks             660948  2 oracleacfs,oracleadvm
    dm01db06: oracleacfs           4609822  3
    dm01db06: oracleadvm            803161  17
    dm01db06: oracleoks             660948  2 oracleacfs,oracleadvm
    dm01db07: oracleacfs           4609822  3
    dm01db07: oracleadvm            803161  17
    dm01db07: oracleoks             660948  2 oracleacfs,oracleadvm
    dm01db08: oracleacfs           4609822  3
    dm01db08: oracleadvm            803161  17
    dm01db08: oracleoks             660948  2 oracleacfs,oracleadvm

    • Check if ASM ADVM proxy is running on every node in a standard cluster or every Hub Node in a Flex Cluster.

    dm01db01-+ASM1 {/home/oracle}:srvctl status asm -proxy
    ADVM proxy is running on node dm01db06,dm01db05,dm01db04,dm01db03,dm01db02,dm01db01,dm01db08,dm01db07

    dm01db01-+ASM1 {/home/oracle}:srvctl config asm -proxy -detail
    ASM home: <CRS home>
    ADVM proxy is enabled
    ADVM proxy is individually enabled on nodes:
    ADVM proxy is individually disabled on nodes:

    dm01db01-orp258c1 {/home/oracle}:/u01/app/12.2.0.1/grid/bin/crsctl stat res ora.proxy_advm -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.proxy_advm
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ——————————————————————————–

    Step I – Create ASM Disk Group for ACFS

    • Login as Oracle Grid Infrastructure software owner and start asmca.

    dm01db01-orcldb1 {/home/oracle}: id oracle
    uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)

    • Set the ORACLE HOME and SID to ASM

    dm01db01-orcldb1 {/home/oracle}:. oraenv
    ORACLE_SID = [orcldb1] ? +ASM1
    The Oracle base remains unchanged with value /u01/app/oracle

    dm01db01-+ASM1 {/home/oracle}:env | grep ORA
    ORACLE_SID=+ASM1
    ORACLE_BASE=/u01/app/oracle
    ORACLE_HOME=/u01/app/12.2.0.1/grid

    • Set DISPLAY

    dm01db01-+ASM1 {/home/oracle}:export DISPLAY=10.30.20.13:0.0

    • Start asmca utility

    dm01db01-+ASM1 {/home/oracle}:which asmca
    /u01/app/12.1.0.2/grid/bin/asmca
    dm01db01-+ASM1 {/home/oracle}:asmca
     
    ASMCA warming up

    ASMCA Home page  appears

    Click on ASM Instances to view the ASM configuration. We can see that it a Flex ASM Configuration

    Click on “Dis Groups”  and click on “Create” button to create a new ASM Disk Group to be used for ACFS

    On Create Disk Group page, specify:
        – Disk Group Name: ACFS_DG
        – Redundancy: Normal
        – Select Member Disk: Show Eligible
            – Select all candidate disks to be part of ACFS_DG disk group
            – In Exadata each Storage cell is a failure Group
        – Click on Show Advanced Options, specify:
           – Specify the ASM/DB/ADVM Compatibilty
        – Click Ok

    ACFS_DG ASM Disk Group creation in progress

    We can see our ACFS_DG Disk Group created 

    • Verify newly created ACFS_DG Disk Group as follows:

    SQL> select inst_id, name, total_mb, group_number from gv$asm_diskgroup where name like ‘ACFS_DG’;

       INST_ID NAME                             TOTAL_MB GROUP_NUMBER
    ———- —————————— ———- ————
             1 ACFS_DG                          10407936            5
             3 ACFS_DG                          10407936            5
             2 ACFS_DG                          10407936            5

    SQL> col value for a30
    SQL> col name for a30
    SQL> select name, value from v$asm_attribute where GROUP_NUMBER=5 and name like ‘compatible%’;

    NAME                           VALUE
    —————————— ——————————
    compatible.asm                 12.2.0.1.0
    compatible.rdbms               11.2.0.4.0
    compatible.advm                12.2.0.1.0

    dm01db01-+ASM1 {/home/oracle}:/u01/app/12.2.0.1/grid/bin/crsctl stat res ora.ACFS_DG.dg -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.ACFS_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  OFFLINE      dm01db05                 STABLE
                   ONLINE  OFFLINE      dm01db06                 STABLE
                   ONLINE  OFFLINE      dm01db07                 STABLE
                   ONLINE  OFFLINE      dm01db08                 STABLE
    ——————————————————————————–

    Step II – Create ASM Volume
    Click on Volumes

    On Volumes page click on “Create”

    On Create Volume page, specify:
        – Volumn name: acfsvol
        – Disk Group name: ACFS_DG
        – Size: 4927.56 (it can be anything based on your requirement from usable space avaialble.)
        – Click Ok

    acfsvol creation is in progress

    We can see our Volume created

    •  Verify newly created Volume as follows:

    ASMCMD> volinfo –all
    Diskgroup Name: ACFS_DG

             Volume Name: ACFSVOL
             Volume Device: /dev/asm/acfsvol-223
             State: ENABLED
             Size (MB): 5045824
             Resize Unit (MB): 64
             Redundancy: MIRROR
             Stripe Columns: 8
             Stripe Width (K): 1024
             Usage: ACFS
             Mountpath: /acfs_ogg

    dm01db01-+ASM1 {/home/oracle}:/u01/app/12.2.0.1/grid/bin/crsctl stat res ora.ACFS_DG.ACFSVOL.advm -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.ACFS_DG.ACFSVOL.advm
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ——————————————————————————–

    dm01db01-+ASM1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘ls -l /dev/asm/*’
    dm01db01: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223
    dm01db02: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223
    dm01db03: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223
    dm01db04: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223
    dm01db05: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223
    dm01db06: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223
    dm01db07: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223
    dm01db08: brwxrwx— 1 root asmdba 251, 114177 Mar 23 08:53 /dev/asm/acfsvol-223

    III – Create ASM Cluster File system

    Click on “ASM Cluster File Systems” on left pane

    On ASM Cluster File Systems page click on “Create”


    On Create ASM Cluster File System page, specify:
        – Type of ACFS: Cluster File System
        – Mount point: /acfs_ogg
        – Auto Mount: check
        – User Name: oracle
        – Group Name: oinstall
        – Description: acfs for OGG
        – Select Volume: ACFSVOL – /dev/asm/acfsvol-223 – 4927.5625G
        – Click Ok

    ACFS creation in progress

    Run the ACFS script on Node 1 only to register ACFS with Grid Infrastructure and to mount the ACFS file system.

    • Contents of ACFS script looks like this:

    [root@dm01db01 ~]# cat /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.sh | more
    #!/bin/sh
    /u01/app/12.2.0.1/grid/bin/srvctl add filesystem -d /dev/asm/acfsvol-223 -m /acfs_ogg -u oracle -fstype ACFS –

    description ‘acfs for OGG’ -autostart ALWAYS
    if [ $? = “0” -o $? = “2” ]; then
       /u01/app/12.2.0.1/grid/bin/srvctl start filesystem -d /dev/asm/acfsvol-223
       if [ $? = “0” ]; then
          chown oracle:oinstall /acfs_ogg
          chmod 775 /acfs_ogg
          /u01/app/12.2.0.1/grid/bin/srvctl status filesystem -d /dev/asm/acfsvol-223
          exit 0
       else
          exit $?
       fi
       /u01/app/12.2.0.1/grid/bin/srvctl status filesystem -d /dev/asm/acfsvol-223
    fi

    • Execute the script

    [root@dm01db01 ~]# /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.sh
    ACFS file system /acfs_ogg is mounted on nodes
    dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08

    Click Close on the Run ACFS script window

    We can see our Volume created

    •  Verify newly created Volume as follows:

    dm01db01-+ASM1 {/home/oracle}:/u01/app/12.2.0.1/grid/bin/crsctl stat res ora.acfs_dg.acfsvol.acfs -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.acfs_dg.acfsvol.acfs
                   ONLINE  ONLINE       dm01db01                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db02                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db03                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db04                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db05                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db06                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db07                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db08                 mounted on /acfs_ogg
                                                                 ,STABLE
    ——————————————————————————–

    dm01db01-+ASM1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘df -h /acfs_ogg’
    dm01db01: Filesystem            Size  Used Avail Use% Mounted on
    dm01db01: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg
    dm01db02: Filesystem            Size  Used Avail Use% Mounted on
    dm01db02: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg
    dm01db03: Filesystem            Size  Used Avail Use% Mounted on
    dm01db03: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg
    dm01db04: Filesystem            Size  Used Avail Use% Mounted on
    dm01db04: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg
    dm01db05: Filesystem            Size  Used Avail Use% Mounted on
    dm01db05: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg
    dm01db06: Filesystem            Size  Used Avail Use% Mounted on
    dm01db06: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg
    dm01db07: Filesystem            Size  Used Avail Use% Mounted on
    dm01db07: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg
    dm01db08: Filesystem            Size  Used Avail Use% Mounted on
    dm01db08: /dev/asm/acfsvol-223  4.9T   12G  4.9T   1% /acfs_ogg

    dm01db01-orp258c1 {/home/oracle}:/u01/app/12.2.0.1/grid/bin/crsctl stat res -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.ACFS_DG.ACFSVOL.advm
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.ACFS_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  OFFLINE      dm01db05                 STABLE
                   ONLINE  OFFLINE      dm01db06                 STABLE
                   ONLINE  OFFLINE      dm01db07                 STABLE
                   ONLINE  OFFLINE      dm01db08                 STABLE
    ora.ASMNET1LSNR_ASM.lsnr
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.DATA.ACFSVOL1.advm
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.DATA.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  OFFLINE      dm01db05                 STABLE
                   ONLINE  OFFLINE      dm01db06                 STABLE
                   ONLINE  OFFLINE      dm01db07                 STABLE
                   ONLINE  OFFLINE      dm01db08                 STABLE
    ora.DATA1.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  OFFLINE      dm01db05                 STABLE
                   ONLINE  OFFLINE      dm01db06                 STABLE
                   ONLINE  OFFLINE      dm01db07                 STABLE
                   ONLINE  OFFLINE      dm01db08                 STABLE
    ora.DBFS_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   OFFLINE OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   OFFLINE OFFLINE      dm01db05                 STABLE
                   OFFLINE OFFLINE      dm01db06                 STABLE
                   OFFLINE OFFLINE      dm01db07                 STABLE
                   OFFLINE OFFLINE      dm01db08                 STABLE
    ora.LISTENER.lsnr
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.MGMT_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   OFFLINE OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   OFFLINE OFFLINE      dm01db05                 STABLE
                   OFFLINE OFFLINE      dm01db06                 STABLE
                   OFFLINE OFFLINE      dm01db07                 STABLE
                   OFFLINE OFFLINE      dm01db08                 STABLE
    ora.acfs_dg.acfsvol.acfs
                   ONLINE  ONLINE       dm01db01                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db02                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db03                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db04                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db05                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db06                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db07                 mounted on /acfs_ogg
                                                                 ,STABLE
                   ONLINE  ONLINE       dm01db08                 mounted on /acfs_ogg
                                                                 ,STABLE
    ora.chad
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.data.acfsvol1.acfs
                   ONLINE  ONLINE       dm01db01                 mounted on /acfs_ogg
                                                                 1,STABLE
                   ONLINE  ONLINE       dm01db02                 mounted on /acfs_ogg
                                                                 1,STABLE
                   ONLINE  ONLINE       dm01db03                 mounted on /acfs_ogg
                                                                 1,STABLE
                   ONLINE  ONLINE       dm01db04                 mounted on /acfs_ogg
                                                                 1,STABLE
                   ONLINE  ONLINE       dm01db05                 mounted on /acfs_ogg
                                                                 1,STABLE
                   ONLINE  ONLINE       dm01db06                 mounted on /acfs_ogg
                                                                 1,STABLE
                   ONLINE  ONLINE       dm01db07                 mounted on /acfs_ogg
                                                                 1,STABLE
                   ONLINE  ONLINE       dm01db08                 mounted on /acfs_ogg
                                                                 1,STABLE
    ora.net1.network
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.ons
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.proxy_advm
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.LISTENER_SCAN1.lsnr
          1        ONLINE  ONLINE       dm01db02                 STABLE
    ora.LISTENER_SCAN2.lsnr
          1        ONLINE  ONLINE       dm01db04                 STABLE
    ora.LISTENER_SCAN3.lsnr
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.MGMTLSNR
          1        ONLINE  ONLINE       dm01db01                 192.168.2.1,STABLE
    ora.asm
          1        ONLINE  ONLINE       dm01db01                 Started,STABLE
          2        ONLINE  ONLINE       dm01db02                 Started,STABLE
          3        ONLINE  ONLINE       dm01db04                 Started,STABLE
    ora.dm01db01.vip
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.dm01db02.vip
          1        ONLINE  ONLINE       dm01db02                 STABLE
    ora.dm01db03.vip
          1        ONLINE  ONLINE       dm01db03                 STABLE
    ora.dm01db04.vip
          1        ONLINE  ONLINE       dm01db04                 STABLE
    ora.dm01db05.vip
          1        ONLINE  ONLINE       dm01db05                 STABLE
    ora.dm01db06.vip
          1        ONLINE  ONLINE       dm01db06                 STABLE
    ora.dm01db07.vip
          1        ONLINE  ONLINE       dm01db07                 STABLE
    ora.dm01db08.vip
          1        ONLINE  ONLINE       dm01db08                 STABLE
    ora.cvu
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.mgmtdb
          1        ONLINE  ONLINE       dm01db01                 Open,STABLE
    ora.orcldb.db
          1        ONLINE  ONLINE       dm01db01                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          2        ONLINE  ONLINE       dm01db02                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          3        ONLINE  ONLINE       dm01db04                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          4        ONLINE  ONLINE       dm01db05                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          5        ONLINE  ONLINE       dm01db07                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          6        ONLINE  ONLINE       dm01db06                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          7        ONLINE  ONLINE       dm01db03                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          8        ONLINE  ONLINE       dm01db08                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
    ora.prod.db
          1        ONLINE  ONLINE       dm01db01                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          2        ONLINE  ONLINE       dm01db02                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          3        ONLINE  ONLINE       dm01db04                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          4        ONLINE  ONLINE       dm01db05                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          5        ONLINE  ONLINE       dm01db07                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          6        ONLINE  ONLINE       dm01db06                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          7        ONLINE  ONLINE       dm01db03                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
          8        ONLINE  ONLINE       dm01db08                 Open,HOME=/u01/app/o
                                                                 racle/product/12.2.0
                                                                 .1/dbhome,STABLE
    ora.qosmserver
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.scan1.vip
          1        ONLINE  ONLINE       dm01db02                 STABLE
    ora.scan2.vip
          1        ONLINE  ONLINE       dm01db04                 STABLE
    ora.scan3.vip
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ——————————————————————————–

    Conclusion:
    In this article we have learned how to quickly configure ACFS on Exadata Database machine running Oracle 12.2.0.1 on ASM Flex Architecture.

  • Installing Oracle Grid Infrastructure 12.2.0.1 on Exadata X2 Full Rack

    Overview
    Oracle has release Oracle database 12.2 for on-premises just a few days ago. It is time to Install the brand new Oracle 12.2 Grid Infrastructure and database software installation on Exadata database.

    With Oracle 12.2 the way Grid Infrastructure is installed has changed little bit. Here are those changes:

    • Unzip Grid Infrastructure software in the GI Home directory
    • Use gridSetup.sh to install and configure GI.

    Note: GI software installation no more uses runInstaller to install the GI software

    In this article I will demonstrate how to install Oracle 12.2 GI and Database software on a full Rack Oracle Exadata X2-2 Database machine.

    Environment

    • Exadata X2-2 Full Rack
    • 8 Compute nodes, 14 Storage cell and 3 IB Switch
    • Exadata Storage Software version 12.1.2.1.1

    Steps to install Oracle Grid Infrastructure 12.2.0.1 on Exadata X2 Full Rack (OEL 6.8)

    • Download Oracle Grid and Database software from OTN.

    http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html

    •  Create the new Grid Infrastructure (GI_HOME) directory where 12.2.0.1 will be installed

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root mkdir -p /u01/app/grid

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root chown oracle:oinstall /u01/app/grid

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root mkdir -p /u01/app/12.2.0.1/grid

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root chown oracle:oinstall /u01/app/12.2.0.1/grid

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ls -l  /u01/app/12.2.0.1
    dm01db01: total 4
    dm01db01: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid
    dm01db02: total 4
    dm01db02: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid
    dm01db03: total 4
    dm01db03: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid
    dm01db04: total 4
    dm01db04: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid
    dm01db05: total 4
    dm01db05: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid
    dm01db06: total 4
    dm01db06: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid
    dm01db07: total 4
    dm01db07: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid
    dm01db08: total 4
    dm01db08: drwxr-xr-x 2 oracle oinstall 4096 Mar  4 08:13 grid

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ls -ld /u01/app/grid

    dm01db01: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid
    dm01db02: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid
    dm01db03: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid
    dm01db04: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid
    dm01db05: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid
    dm01db06: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid
    dm01db07: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid
    dm01db08: drwxr-xr-x 2 oracle oinstall 4096 Mar  9 03:02 /u01/app/grid

    • Unzip installation Oracle software

    Grid Infrastructure software

    dm01db01-orcldb1 {/home/oracle}:cd /u01/app/oracle/software/

    dm01db01-orcldb1 {/u01/app/oracle/software}:unzip -q /u01/app/oracle/software/linuxx64_12201_grid_home.zip -d /u01/app/12.2.0.1/grid

    dm01db01-orcldb1 {/u01/app/oracle/software}:ls -l /u01/app/12.2.0.1/grid
    total 296
    drwxr-xr-x  2 oracle oinstall 4096 Jan 26 10:12 addnode
    drwxr-xr-x 11 oracle oinstall 4096 Jan 26 10:10 assistants
    drwxr-xr-x  2 oracle oinstall 8192 Jan 26 10:12 bin
    drwxr-xr-x  3 oracle oinstall 4096 Jan 26 10:12 cdata
    drwxr-xr-x  3 oracle oinstall 4096 Jan 26 10:10 cha
    drwxr-xr-x  4 oracle oinstall 4096 Jan 26 10:12 clone
    drwxr-xr-x 16 oracle oinstall 4096 Jan 26 10:12 crs
    drwxr-xr-x  6 oracle oinstall 4096 Jan 26 10:12 css
    drwxr-xr-x  7 oracle oinstall 4096 Jan 26 10:10 cv
    drwxr-xr-x  3 oracle oinstall 4096 Jan 26 10:10 dbjava
    drwxr-xr-x  2 oracle oinstall 4096 Jan 26 10:11 dbs
    drwxr-xr-x  2 oracle oinstall 4096 Jan 26 10:12 dc_ocm
    drwxr-xr-x  5 oracle oinstall 4096 Jan 26 10:12 deinstall
    drwxr-xr-x  3 oracle oinstall 4096 Jan 26 10:10 demo
    drwxr-xr-x  3 oracle oinstall 4096 Jan 26 10:10 diagnostics
    drwxr-xr-x  8 oracle oinstall 4096 Jan 26 10:11 dmu
    -rw-r–r–  1 oracle oinstall  852 Aug 19  2015 env.ora
    drwxr-xr-x  7 oracle oinstall 4096 Jan 26 10:12 evm
    drwxr-xr-x  5 oracle oinstall 4096 Jan 26 10:10 gpnp
    -rwxr-x—  1 oracle oinstall 5395 Jul 21  2016 gridSetup.sh
    drwxr-xr-x  4 oracle oinstall 4096 Jan 26 10:10 has
    drwxr-xr-x  3 oracle oinstall 4096 Jan 26 10:10 hs
    ….

    • Setup User equivalance

    dm01db01-orcldb1 {/home/oracle}: dcli -g dbs_group -l oracle -k

    • Automatic Configure prerequisites using Oracle RPMs

    Oracle recommends that you install Oracle Linux 6 or Oracle Linux 7 and use Oracle RPMs to configure your operating systems for Oracle Database and Oracle Grid Infrastructure installations.

    [root@dm01db01 ~]# yum install oracle-database-server-12cR2-preinstall

    • Manually prerequisites steps

    Perform the following steps if you are not planning to use the Oracle RPM to perform prerequisites.

    Set kernel parameters
    Minimum requirement for GI installation. Edit the /etc/sysctl.conf file, and add or edit the following lines.

    [root@dm01db01 ~]# vi /etc/sysctl.conf

    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    kernel.shmall = 2097152
    kernel.shmmax = 4294967295
    kernel.shmmni = 4096
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default = 262144
    net.core.rmem_max = 4194304
    net.core.wmem_default = 262144
    net.core.wmem_max = 1048576

    Update the current values of the kernel parameters:

    [root@dm01db01 ~]# /sbin/sysctl -p

    Confirm that the values are set correctly:

    [root@dm01db01 ~]# /sbin/sysctl -a

    Install required OS packages
    The following packages (or later versions) are required for OEL 6:

    binutils-2.20.51.0.2-5.36.el6 (x86_64)
    compat-libcap1-1.10-1 (x86_64)
    compat-libstdc++-33-3.2.3-69.el6 (x86_64)
    compat-libstdc++-33-3.2.3-69.el6 (i686)
    e2fsprogs-1.41.12-14.el6 (x86_64)
    e2fsprogs-libs-1.41.12-14.el6 (x86_64)
    glibc-2.12-1.7.el6 (i686)
    glibc-2.12-1.7.el6 (x86_64)
    glibc-devel-2.12-1.7.el6 (x86_64)
    glibc-devel-2.12-1.7.el6 (i686)
    ksh
    libgcc-4.4.4-13.el6 (i686)
    libgcc-4.4.4-13.el6 (x86_64)
    libs-1.42.8-1.0.2.el6.x86_64
    libstdc++-4.4.4-13.el6 (x86_64)
    libstdc++-4.4.4-13.el6 (i686)
    libstdc++-devel-4.4.4-13.el6 (x86_64)
    libstdc++-devel-4.4.4-13.el6 (i686)
    libaio-0.3.107-10.el6 (x86_64)
    libaio-0.3.107-10.el6 (i686)
    libaio-devel-0.3.107-10.el6 (x86_64)
    libaio-devel-0.3.107-10.el6 (i686)
    libXtst-1.0.99.2 (x86_64)
    libXtst-1.0.99.2 (i686)
    libX11-1.5.0-4.el6 (i686)
    libX11-1.5.0-4.el6 (x86_64)
    libXau-1.0.6-4.el6 (i686)
    libXau-1.0.6-4.el6 (x86_64)
    libxcb-1.8.1-1.el6 (i686)
    libxcb-1.8.1-1.el6 (x86_64)
    libXi-1.3 (x86_64)
    libXi-1.3 (i686)
    make-3.81-19.el6
    net-tools-1.60-110.el6_2.x86_64 (for Oracle RAC and Oracle Clusterware)
    nfs-utils-1.2.3-15.0.1 (for Oracle ACFS)
    sysstat-9.0.4-11.el6 (x86_64)
    smartmontools-5.43-1.el6.x86_64

    Create OS groups and users
    Create the following groups and user

    [root@dm01db01 ~]# groupadd -g 1001 oinstall
    [root@dm01db01 ~]# groupadd -g 1002 dba
    [root@dm01db01 ~]# groupadd -g 1003 oper

    [root@dm01db01 ~]# groupadd -g 1004 asmdba
    [root@dm01db01 ~]# groupadd -g 1005 asmadmin
    [root@dm01db01 ~]# groupadd -g 1006 asmoper

    [root@dm01db01 ~]# groupadd -g 1007 backupdba
    [root@dm01db01 ~]# groupadd -g 1008 dgdba
    [root@dm01db01 ~]# groupadd -g 1009 kmdba
    [root@dm01db01 ~]# groupadd -g 1010 racdba

    [root@dm01db01 ~]# useradd -u 1000 -g oinstall -G dba,oper,asmdba,asmoper,asmadmin,backupdba,dgdba,kmdba,racdba oracle


    Ensure the IPs are allocated
    Following IPs are required for Oracle GI installation. There are taken care by Exadata DBA or Networking Engineer at the time of Exadata installation.

    • Management IP
    • Public IP
    • Private IP
    • Virtual IP
    • Scan IP
    Register hostname/IPs in DNS
    This is done as part Exadata installation by Exadata DBA or Network Engineer.

    Check hostname resolution
    [root@dm01db01 ~]# ping dm01db01
    [root@dm01db01 ~]# ping dm01db01.nsm.com
    [root@dm01db01 ~]# nslookup dm01db01

    Disable firewall
    [root@dm01db01 ~]# service iptables status
    [root@dm01db01 ~]# service iptables stop

    Create user’s profile
    dm01db01-orcldb1 {/u01/app/oracle/software}: vi .bash_profile

    export TMP=/tmp
    export TMPDIR=$TMP

    export ORACLE_BASE=/u01/app/oracle
    export GRID_HOME=/u01/app/12.2.0.1/grid
    export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome
    export ORACLE_TERM=xterm
    export PATH=$PATH:$ORACLE_HOME/bin
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

    • Validate Readiness for Oracle Clusterware upgrade using CVU

    dm01db01-orcldb1 {/u01/app/oracle/software}:/u01/app/12.2.0.1/grid/runcluvfy.sh stage -pre crsinst -n

    dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08 -r 12.2 -orainv oinstall -fixupnoexec -verbose

    Verifying Physical Memory …
      Node Name     Available                 Required                  Status
      ————  ————————  ————————  ———-
      dm01db05      70.5587GB (7.3986188E7KB)  8GB (8388608.0KB)         passed
      dm01db04      70.5587GB (7.3986188E7KB)  8GB (8388608.0KB)         passed
      dm01db03      62.6681GB (6.5712308E7KB)  8GB (8388608.0KB)         passed
      dm01db02      70.5587GB (7.3986188E7KB)  8GB (8388608.0KB)         passed
      dm01db01      70.5587GB (7.3986188E7KB)  8GB (8388608.0KB)         passed
      dm01db08      70.5587GB (7.3986188E7KB)  8GB (8388608.0KB)         passed
      dm01db07      70.5587GB (7.3986188E7KB)  8GB (8388608.0KB)         passed
      dm01db06      70.5587GB (7.3986188E7KB)  8GB (8388608.0KB)         passed
    Verifying Physical Memory …PASSED
    Verifying Available Physical Memory …
      Node Name     Available                 Required                  Status
      ————  ————————  ————————  ———-
      dm01db05      64.6698GB (6.7811208E7KB)  50MB (51200.0KB)          passed
      dm01db04      65.1885GB (6.8355116E7KB)  50MB (51200.0KB)          passed
      dm01db03      59.677GB (6.257588E7KB)   50MB (51200.0KB)           passed
      dm01db02      66.6203GB (6.9856476E7KB)  50MB (51200.0KB)          passed
      dm01db01      65.2446GB (6.8413932E7KB)  50MB (51200.0KB)          passed
      dm01db08      64.9811GB (6.8137596E7KB)  50MB (51200.0KB)          passed
      dm01db07      65.0432GB (6.820278E7KB)  50MB (51200.0KB)           passed
      dm01db06      65.0759GB (6.823698E7KB)  50MB (51200.0KB)           passed
    Verifying Available Physical Memory …PASSED
    Verifying Swap Size …
      Node Name     Available                 Required                  Status
      ————  ————————  ————————  ———-
      dm01db05      15.9949GB (1.6771856E7KB)  16GB (1.6777216E7KB)      passed
      dm01db04      15.9949GB (1.6771856E7KB)  16GB (1.6777216E7KB)      passed
      dm01db03      15.9949GB (1.6771856E7KB)  15.9689GB (1.6744628E7KB)  passed
      dm01db02      15.9949GB (1.6771856E7KB)  16GB (1.6777216E7KB)      passed
      dm01db01      15.9949GB (1.6771856E7KB)  16GB (1.6777216E7KB)      passed
      dm01db08      15.9949GB (1.6771856E7KB)  16GB (1.6777216E7KB)      passed
      dm01db07      15.9949GB (1.6771856E7KB)  16GB (1.6777216E7KB)      passed
      dm01db06      15.9949GB (1.6771856E7KB)  16GB (1.6777216E7KB)      passed
    Verifying Swap Size …PASSED
    Verifying Free Space: dm01db05:/usr,dm01db05:/var,dm01db05:/etc,dm01db05:/sbin,dm01db05:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db05      /             31.1182GB     25MB          passed
      /var              dm01db05      /             31.1182GB     5MB           passed
      /etc              dm01db05      /             31.1182GB     25MB          passed
      /sbin             dm01db05      /             31.1182GB     10MB          passed
      /tmp              dm01db05      /             31.1182GB     1GB           passed
    Verifying Free Space: dm01db05:/usr,dm01db05:/var,dm01db05:/etc,dm01db05:/sbin,dm01db05:/tmp …PASSED
    Verifying Free Space: dm01db04:/usr,dm01db04:/var,dm01db04:/etc,dm01db04:/sbin,dm01db04:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db04      /             30.8779GB     25MB          passed
      /var              dm01db04      /             30.8779GB     5MB           passed
      /etc              dm01db04      /             30.8779GB     25MB          passed
      /sbin             dm01db04      /             30.8779GB     10MB          passed
      /tmp              dm01db04      /             30.8779GB     1GB           passed
    Verifying Free Space: dm01db04:/usr,dm01db04:/var,dm01db04:/etc,dm01db04:/sbin,dm01db04:/tmp …PASSED
    Verifying Free Space: dm01db03:/usr,dm01db03:/var,dm01db03:/etc,dm01db03:/sbin,dm01db03:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db03      /             32.7207GB     25MB          passed
      /var              dm01db03      /             32.7207GB     5MB           passed
      /etc              dm01db03      /             32.7207GB     25MB          passed
      /sbin             dm01db03      /             32.7207GB     10MB          passed
      /tmp              dm01db03      /             32.7207GB     1GB           passed
    Verifying Free Space: dm01db03:/usr,dm01db03:/var,dm01db03:/etc,dm01db03:/sbin,dm01db03:/tmp …PASSED
    Verifying Free Space: dm01db02:/usr,dm01db02:/var,dm01db02:/etc,dm01db02:/sbin,dm01db02:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db02      /             24.959GB      25MB          passed
      /var              dm01db02      /             24.959GB      5MB           passed
      /etc              dm01db02      /             24.959GB      25MB          passed
      /sbin             dm01db02      /             24.959GB      10MB          passed
      /tmp              dm01db02      /             24.959GB      1GB           passed
    Verifying Free Space: dm01db02:/usr,dm01db02:/var,dm01db02:/etc,dm01db02:/sbin,dm01db02:/tmp …PASSED
    Verifying Free Space: dm01db01:/usr,dm01db01:/var,dm01db01:/etc,dm01db01:/sbin,dm01db01:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db01      /             27.3477GB     25MB          passed
      /var              dm01db01      /             27.3477GB     5MB           passed
      /etc              dm01db01      /             27.3477GB     25MB          passed
      /sbin             dm01db01      /             27.3477GB     10MB          passed
      /tmp              dm01db01      /             27.3477GB     1GB           passed
    Verifying Free Space: dm01db01:/usr,dm01db01:/var,dm01db01:/etc,dm01db01:/sbin,dm01db01:/tmp …PASSED
    Verifying Free Space: dm01db08:/usr,dm01db08:/var,dm01db08:/etc,dm01db08:/sbin,dm01db08:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db08      /             36.6592GB     25MB          passed
      /var              dm01db08      /             36.6592GB     5MB           passed
      /etc              dm01db08      /             36.6592GB     25MB          passed
      /sbin             dm01db08      /             36.6592GB     10MB          passed
      /tmp              dm01db08      /             36.6592GB     1GB           passed
    Verifying Free Space: dm01db08:/usr,dm01db08:/var,dm01db08:/etc,dm01db08:/sbin,dm01db08:/tmp …PASSED
    Verifying Free Space: dm01db07:/usr,dm01db07:/var,dm01db07:/etc,dm01db07:/sbin,dm01db07:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db07      /             38.7031GB     25MB          passed
      /var              dm01db07      /             38.7031GB     5MB           passed
      /etc              dm01db07      /             38.7031GB     25MB          passed
      /sbin             dm01db07      /             38.7031GB     10MB          passed
      /tmp              dm01db07      /             38.7031GB     1GB           passed
    Verifying Free Space: dm01db07:/usr,dm01db07:/var,dm01db07:/etc,dm01db07:/sbin,dm01db07:/tmp …PASSED
    Verifying Free Space: dm01db06:/usr,dm01db06:/var,dm01db06:/etc,dm01db06:/sbin,dm01db06:/tmp …
      Path              Node Name     Mount point   Available     Required      Status
      —————-  ————  ————  ————  ————  ————
      /usr              dm01db06      /             38.2168GB     25MB          passed
      /var              dm01db06      /             38.2168GB     5MB           passed
      /etc              dm01db06      /             38.2168GB     25MB          passed
      /sbin             dm01db06      /             38.2168GB     10MB          passed
      /tmp              dm01db06      /             38.2168GB     1GB           passed
    Verifying Free Space: dm01db06:/usr,dm01db06:/var,dm01db06:/etc,dm01db06:/sbin,dm01db06:/tmp …PASSED
    Verifying User Existence: oracle …
      Node Name     Status                    Comment
      ————  ————————  ————————
      dm01db05      passed                    exists(1000)
      dm01db04      passed                    exists(1000)
      dm01db03      passed                    exists(1000)
      dm01db02      passed                    exists(1000)
      dm01db01      passed                    exists(1000)
      dm01db08      passed                    exists(1000)
      dm01db07      passed                    exists(1000)
      dm01db06      passed                    exists(1000)

    Verifying Users With Same UID: 1000 …PASSED
    Verifying User Existence: oracle …PASSED
    …
    ******************************************************************************************
    Following is the list of fixable prerequisites selected to fix in this session
    ******************************************************************************************
    ————–                —————     —————-
    Check failed.                 Failed on nodes     Reboot required?
    ————–                —————     —————-
    Soft Limit: maximum stack     dm01db05,dm01db04,  no
    size                          dm01db03,dm01db02,
                                  dm01db01,dm01db08,
                                  dm01db07,dm01db06
    Group Existence: asmadmin     dm01db05,dm01db04,  no
                                  dm01db03,dm01db02,
                                  dm01db01,dm01db08,
                                  dm01db07,dm01db06

    Execute “/tmp/CVU_12.2.0.1.0_oracle/runfixup.sh” as root user on nodes “dm01db06,dm01db05,dm01db04,dm01db03,dm01db02,dm01db01,dm01db08,dm01db07” to perform the fix up operations manually

    • Unset the environment variables

    dm01db01-orcldb1 {/u01/app/oracle/software}:unset ORACLE_HOME ORACLE_BASE ORACLE_SID

    • Set the display and execute gridSetup.sh

    dm01db01-orcldb1 {/home/oracle}:export DISPLAY=10.30.221.39:0.0

    dm01db01-orcldb1 {/home/oracle}:cd /u01/app/12.2.0.1/grid

    dm01db01-orcldb1 {/u01/app/12.2.0.1/grid}:ls -l gridSetup.sh
    -rwxr-x— 1 oracle oinstall 5395 Jul 21  2016 gridSetup.sh

    dm01db01-orcldb1 {/u01/app/12.2.0.1/grid}:./gridSetup.sh
    Launching Oracle Grid Infrastructure Setup Wizard…


    OUI starting …

    Select “Configure Oracle Grid Infrastructure for a New Cluster”



    Select “Configure an Oracle Standalone Cluster”. With Oracle 12.2 Oracle has introduced different Cluster option. For more information please refer to Oracle Grid Infrastructure Install and Upgrade document.

    Specify the Cluster name, SCAN Name and the SCAN port to be used for Installation

    Provide the list of nodes. Click Add button

     On this page you can define one node at a time or range of nodes. Here I am specifying one node.

     To specify a range of node, select Add a range of nodes and use the pattern below.

    Verify all the nodes and VIP and click next.

    Select the correct interface for ASM & Private network and Public network.

     Select “Configure ASM using Block Devices”.

     Select Yes, if you are planning to create a separate ASM disk group for GIMR. Else select No.

     Select “Change Discovery Path” to enter the correct string to discover the candidate disks.

    Enter “o/*/SYSTEMDG* to search for all ASM disks with name “SYSTEMDG”

    We can see all the ASM disk with the prefix SYSTEMDG 

    Click Specify Failure Groups to create the ASM failure groups. Here I have 14 storage cells, so each cell act as a failure group in Exadata

    Select the ASM disks to create DBFS_DG ASM disk group to store OCR and Voting disks. Here I am creating DBFS_DG disk group using normal redundancy and disks from 7 storage cells.

     Create MGMT_DG for storing GIMR repository. Here I am creating MGMT_DG disk group using normal redundancy and disks from 7 storage cells.



    Specify the ASM password for SYS and ASMSNMP users

    Specify a strong password to avoid this warning. Click YES to continue.

     Select “Do not user IPMI” and click Next.

    Click next. This step can be performed at a later time.

    Specify the OS groups for ASM authentication

    Click Yes to continue.

    Specify the Oracle Base. GI HOME is selected by default as we unzipped the GI software and running gridSetup from GI HOME.

    Click Next. We can run the root script manually at the end of the installation.

    Prerequisite checks started

    Prerequisite checks in progress

    Prerequisite check failed for Soft Limit. You can ignore it and move forward or use “Fix & Check Again” 
    Click Fix & Check Again



    A pop-up box appears asking you to run the “runfixup.sh” script



    Login as root user and run the fixup script

    If you don’t want to run the fixup script. Click Ignore All and click next.
    Click Yes to continue.

    On Installation summary page verify the details and click Install



    Grid Infrastructure Installation progress

    The following screen appears to run the root.sh script as root user.

    Open a new terminal and run the /u01/app/12.2.0.1/grid/root.sh on all the nodes. Run the script on node 1 first. Once it is completed successfully on node 1, you can run the script on all other nodes in parallel.

    Node 1:
    [root@dm01db01 ~]# /u01/app/12.2.0.1/grid/root.sh
    Performing root user operation.

    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /u01/app/12.2.0.1/grid

    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    The contents of “dbhome” have not changed. No need to overwrite.
    The file “oraenv” already exists in /usr/local/bin.  Overwrite it? (y/n)
    [n]: y
       Copying oraenv to /usr/local/bin …
    The contents of “coraenv” have not changed. No need to overwrite.

    Creating /etc/oratab file…
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Relinking oracle with rac_on option
    Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
    The log of current session can be found at:
      /u01/app/grid/crsdata/dm01db01/crsconfig/rootcrs_dm01db01_2017-03-09_03-33-02AM.log
    2017/03/09 03:33:04 CLSRSC-594: Executing installation step 1 of 19: ‘SetupTFA’.
    2017/03/09 03:33:05 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
    2017/03/09 03:33:36 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
    2017/03/09 03:33:36 CLSRSC-594: Executing installation step 2 of 19: ‘ValidateEnv’.
    2017/03/09 03:33:43 CLSRSC-363: User ignored prerequisites during installation
    2017/03/09 03:33:43 CLSRSC-594: Executing installation step 3 of 19: ‘CheckFirstNode’.
    2017/03/09 03:33:45 CLSRSC-594: Executing installation step 4 of 19: ‘GenSiteGUIDs’.
    2017/03/09 03:33:47 CLSRSC-594: Executing installation step 5 of 19: ‘SaveParamFile’.
    2017/03/09 03:33:54 CLSRSC-594: Executing installation step 6 of 19: ‘SetupOSD’.
    2017/03/09 03:33:54 CLSRSC-594: Executing installation step 7 of 19: ‘CheckCRSConfig’.
    2017/03/09 03:33:54 CLSRSC-594: Executing installation step 8 of 19: ‘SetupLocalGPNP’.
    2017/03/09 03:34:16 CLSRSC-594: Executing installation step 9 of 19: ‘ConfigOLR’.
    2017/03/09 03:34:24 CLSRSC-594: Executing installation step 10 of 19: ‘ConfigCHMOS’.
    2017/03/09 03:34:24 CLSRSC-594: Executing installation step 11 of 19: ‘CreateOHASD’.
    2017/03/09 03:34:30 CLSRSC-594: Executing installation step 12 of 19: ‘ConfigOHASD’.
    2017/03/09 03:34:45 CLSRSC-330: Adding Clusterware entries to file ‘oracle-ohasd.conf’
    2017/03/09 03:35:05 CLSRSC-594: Executing installation step 13 of 19: ‘InstallAFD’.
    2017/03/09 03:35:11 CLSRSC-594: Executing installation step 14 of 19: ‘InstallACFS’.
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’ has completed
    CRS-4133: Oracle High Availability Services has been stopped.
    CRS-4123: Oracle High Availability Services has been started.
    2017/03/09 03:35:43 CLSRSC-594: Executing installation step 15 of 19: ‘InstallKA’.
    2017/03/09 03:35:56 CLSRSC-594: Executing installation step 16 of 19: ‘InitConfig’.
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’ has completed
    CRS-4133: Oracle High Availability Services has been stopped.
    CRS-4123: Oracle High Availability Services has been started.
    CRS-2672: Attempting to start ‘ora.evmd’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.mdnsd’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.evmd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.gpnpd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.gipcd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.cssdmonitor’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.gipcd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.cssd’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.diskmon’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.diskmon’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.cssd’ on ‘dm01db01’ succeeded

    Disk groups created successfully. Check /u01/app/grid/cfgtoollogs/asmca/asmca-170309AM033635.log for details.

    2017/03/09 03:37:26 CLSRSC-482: Running command: ‘/u01/app/12.2.0.1/grid/bin/ocrconfig -upgrade oracle oinstall’
    CRS-2672: Attempting to start ‘ora.crf’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.storage’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.storage’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.crf’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.crsd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.crsd’ on ‘dm01db01’ succeeded
    CRS-4256: Updating the profile
    Successful addition of voting disk 78c7111b10e44f2dbfed007a03782cef.
    Successful addition of voting disk 42f1ce36d82b4f43bf7802841a0511d4.
    Successful addition of voting disk a43a383b20ff4f50bf690962ce8e9bb5.
    Successfully replaced voting disk group with +DBFS_DG.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    —  —–    —————–                ——— ———
     1. ONLINE   78c7111b10e44f2dbfed007a03782cef (o/192.168.2.11/SYSTEMDG_CD_10_dm01cel03) [DBFS_DG]
     2. ONLINE   42f1ce36d82b4f43bf7802841a0511d4 (o/192.168.2.18/SYSTEMDG_CD_06_dm01cel10) [DBFS_DG]
     3. ONLINE   a43a383b20ff4f50bf690962ce8e9bb5 (o/192.168.2.16/SYSTEMDG_CD_05_dm01cel08) [DBFS_DG]
    Located 3 voting disk(s).
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.storage’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.crf’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.storage’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.asm’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db01’ succeeded
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’ has completed
    CRS-4133: Oracle High Availability Services has been stopped.
    2017/03/09 03:39:09 CLSRSC-594: Executing installation step 17 of 19: ‘StartCluster’.
    CRS-4123: Starting Oracle High Availability Services-managed resources
    CRS-2672: Attempting to start ‘ora.evmd’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.mdnsd’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.evmd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.gpnpd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.gipcd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.gipcd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.cssdmonitor’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.cssd’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.diskmon’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.diskmon’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.cssd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.ctssd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.ctssd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.asm’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.asm’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.storage’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.storage’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.crf’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.crf’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.crsd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.crsd’ on ‘dm01db01’ succeeded
    CRS-6023: Starting Oracle Cluster Ready Services-managed resources
    CRS-6017: Processing resource auto-start for servers: dm01db01
    CRS-6016: Resource auto-start has completed for server dm01db01
    CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
    CRS-4123: Oracle High Availability Services has been started.
    2017/03/09 03:41:28 CLSRSC-343: Successfully started Oracle Clusterware stack
    2017/03/09 03:41:28 CLSRSC-594: Executing installation step 18 of 19: ‘ConfigNode’.
    CRS-2672: Attempting to start ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.asm’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.asm’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.DBFS_DG.dg’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.DBFS_DG.dg’ on ‘dm01db01’ succeeded
    2017/03/09 03:43:32 CLSRSC-594: Executing installation step 19 of 19: ‘PostConfig’.

    2017/03/09 03:44:19 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster … succeeded 
    Now you can run the script on all other nodes in parallel.

    Go back to the installation screen and click ok.

    Installation continues

    Click Ok to finish the Installation.

    • Verify Installation

    [root@dm01db01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.ASMNET1LSNR_ASM.lsnr
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   OFFLINE OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   OFFLINE OFFLINE      dm01db05                 STABLE
                   OFFLINE OFFLINE      dm01db06                 STABLE
                   OFFLINE OFFLINE      dm01db07                 STABLE
                   OFFLINE OFFLINE      dm01db08                 STABLE
    ora.DBFS_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   OFFLINE OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   OFFLINE OFFLINE      dm01db05                 STABLE
                   OFFLINE OFFLINE      dm01db06                 STABLE
                   OFFLINE OFFLINE      dm01db07                 STABLE
                   OFFLINE OFFLINE      dm01db08                 STABLE
    ora.LISTENER.lsnr
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.MGMT_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   OFFLINE OFFLINE      dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   OFFLINE OFFLINE      dm01db05                 STABLE
                   OFFLINE OFFLINE      dm01db06                 STABLE
                   OFFLINE OFFLINE      dm01db07                 STABLE
                   OFFLINE OFFLINE      dm01db08                 STABLE
    ora.chad
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.net1.network
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.ons
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.proxy_advm
                   OFFLINE OFFLINE      dm01db01                 STABLE
                   OFFLINE OFFLINE      dm01db02                 STABLE
                   OFFLINE OFFLINE      dm01db03                 STABLE
                   OFFLINE OFFLINE      dm01db04                 STABLE
                   OFFLINE OFFLINE      dm01db05                 STABLE
                   OFFLINE OFFLINE      dm01db06                 STABLE
                   OFFLINE OFFLINE      dm01db07                 STABLE
                   OFFLINE OFFLINE      dm01db08                 STABLE
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.LISTENER_SCAN1.lsnr
          1        ONLINE  ONLINE       dm01db02                 STABLE
    ora.LISTENER_SCAN2.lsnr
          1        ONLINE  ONLINE       dm01db04                 STABLE
    ora.LISTENER_SCAN3.lsnr
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.MGMTLSNR
          1        ONLINE  ONLINE       dm01db01                 192.168.2.1,STABLE
    ora.asm
          1        ONLINE  ONLINE       dm01db01                 Started,STABLE
          2        ONLINE  ONLINE       dm01db02                 Started,STABLE
          3        ONLINE  ONLINE       dm01db04                 Started,STABLE
    ora.dm01db01.vip
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.dm01db02.vip
          1        ONLINE  ONLINE       dm01db02                 STABLE
    ora.dm01db03.vip
          1        ONLINE  ONLINE       dm01db03                 STABLE
    ora.dm01db04.vip
          1        ONLINE  ONLINE       dm01db04                 STABLE
    ora.dm01db05.vip
          1        ONLINE  ONLINE       dm01db05                 STABLE
    ora.dm01db06.vip
          1        ONLINE  ONLINE       dm01db06                 STABLE
    ora.dm01db07.vip
          1        ONLINE  ONLINE       dm01db07                 STABLE
    ora.dm01db08.vip
          1        ONLINE  ONLINE       dm01db08                 STABLE
    ora.cvu
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.mgmtdb
          1        ONLINE  ONLINE       dm01db01                 Open,STABLE
    ora.qosmserver
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.scan1.vip
          1        ONLINE  ONLINE       dm01db02                 STABLE
    ora.scan2.vip
          1        ONLINE  ONLINE       dm01db04                 STABLE
    ora.scan3.vip
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ——————————————————————————–
    [root@dm01db01 ~]#

    ASMCMD> lsdg
    State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB 

    Usable_file_MB  Offline_disks  Voting_files  Name
    MOUNTED  NORMAL  N         512             512   4096  4194304   2136960  2135156            30528        

    1052314              0             Y  DBFS_DG/
    MOUNTED  NORMAL  N         512             512   4096  4194304    824256   755900            30528         

    362686              0             N  MGMT_DG/

    ASMCMD> showclustermode
    ASM cluster : Flex mode enabled

    ASMCMD> showversion
    ASM version         : 12.2.0.1.0


    Conclusion
    In this article we have learned how to Install Oracle Grid Infrastructure 12.2.0.1 on Exadata X2 Full Rack. The GI installation has change little bit 12.2. GI 12.2 installation now uses gridSetup.sh to installation GI software.

  • How To Add Exadata Storage Server To Existing Exadata Cluster

    Overview

    You purchased a new Storage cell for expansion or removed Storage from an
    existing Cluster and want to add it to the new a Cluster due to space demand.

    In this article I will demonstrate how to add a storage cell to an
    existing Exadata Database Machine.

    Here we are
    adding new storage cell with 600 GB disks to existing Exadata database machine using
    600 GB disks.


    Environment

    • Exadata Database Machine X2-2 Full Rack
    • Exadata Storage Software version 12.1.2.2.0
    • Oracle Grid/Database Version 11.2.0.4


    Pre
    Checks:

    • Create
      a new file named “new_cell_group” containing only the hostnames for
      the new cells.

    [root@dm01db01 ~]# vi
    new_cell_group
    [root@dm01db01 ~]# cat
    new_cell_group
    dm01cel05
    • It is assumed that the
      file “cell_group” only contains the original cells, not the new ones.
       If you have already modified “cell_group” to contain the new
      cells, just comment it out or remove it from the file.

    [root@dm01db01 ~]# cat cell_group
    dm01cel01
    dm01cel02
    dm01cel03
    dm01cel04
    dm01cel05
    dm01cel06
    dm01cel07
    dm01cel08
    dm01cel09
    dm01cel10
    dm01cel11
    dm01cel12
    dm01cel13
    dm01cel14
    [root@dm01db01 ~]# vi cell_group
    [root@dm01db01 ~]# cat cell_group
    dm01cel01
    dm01cel02
    dm01cel03
    dm01cel04
    #dm01cel05
    dm01cel06
    dm01cel07
    dm01cel08
    dm01cel09
    dm01cel10
    dm01cel11
    dm01cel12
    dm01cel13
    dm01cel14
    • Here
      we use the root user to run the cellcli commands; ensure that user equivalency setup
      for root user between Storage cells and Compute nodes.

    [root@dm01db01
    ~]# dcli -g dbs_group -l root ‘uptime’
    dm01db01:
    09:55:05 up 425 days, 23:16,  1
    user,  load average: 3.26, 3.37, 3.39
    dm01db02:
    09:55:05 up 117 days, 18:20,  0
    users,  load average: 1.19, 1.42, 1.56
    dm01db03:
    09:55:05 up 85 days, 10:07,  0
    users,  load average: 6.25, 6.20, 6.22
    dm01db04:
    09:55:05 up 519 days, 11:33,  0
    users,  load average: 1.53, 1.48, 1.47
    dm01db05:
    09:55:05 up 519 days, 12:45,  0
    users,  load average: 1.36, 1.35, 1.47
    dm01db06:
    09:55:05 up 515 days, 21:40,  0
    users,  load average: 1.47, 1.36, 1.36
    dm01db07:
    09:55:05 up 519 days, 12:03,  0
    users,  load average: 1.44, 1.64, 1.71
    dm01db08:
    09:55:05 up 519 days, 11:29,  0
    users,  load average: 1.78, 1.90, 1.78
    [root@dm01db01
    ~]# dcli -g cell_group -l root ‘uptime’
    dm01cel01:
    09:55:15 up 466 days, 20:25,  0
    users,  load average: 1.44, 1.35, 1.26
    dm01cel02:
    09:55:15 up 519 days, 14:48,  0
    users,  load average: 1.49, 1.44, 1.51
    dm01cel03:
    09:55:15 up 519 days, 14:47,  0
    users,  load average: 1.01, 0.96, 1.10
    dm01cel04:
    09:55:15 up 519 days, 14:47,  0
    users,  load average: 1.40, 1.32, 1.24
    dm01cel06:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 1.10, 1.25, 1.30
    dm01cel07:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 0.99, 1.14, 1.18
    dm01cel08:
    09:55:15 up 466 days, 19:37,  0
    users,  load average: 1.52, 1.24, 1.17
    dm01cel09:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 1.00, 1.40, 1.56
    dm01cel10:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 1.09, 1.22, 1.22
    dm01cel11:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 1.24, 1.27, 1.22
    dm01cel12:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 1.11, 1.14, 1.14
    dm01cel13:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 1.13, 1.27, 1.16
    dm01cel14:
    09:55:15 up 519 days, 14:46,  0
    users,  load average: 1.20, 1.12, 1.15
    Ensure Cell Disks are Ready

    • First verify if any cell disks
      already exist:

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list celldisk attributes
    name,status,freespace where disktype=harddisk”
    dm01cel05: CD_00_dm01cel05       normal 
    0
    dm01cel05: CD_01_dm01cel05       normal 
    0
    dm01cel05: CD_02_dm01cel05       normal 
    0
    dm01cel05: CD_03_dm01cel05       normal 
    0
    dm01cel05: CD_04_dm01cel05       normal 
    0
    dm01cel05: CD_05_dm01cel05       normal 
    0
    dm01cel05: CD_06_dm01cel05       normal 
    0
    dm01cel05: CD_07_dm01cel05       normal 
    0
    dm01cel05: CD_08_dm01cel05       normal 
    0
    dm01cel05: CD_09_dm01cel05       normal 
    0
    dm01cel05: CD_10_dm01cel05       normal 
    0
    dm01cel05: CD_11_dm01cel05       normal 
    0
    In my case the cell disks exists.
    • If no cell disks are created.
      There will be no output.

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list celldisk attributes
    name,status,freespace where disktype=harddisk”
    [root@dm01db01 ~]#
    • If
      this command returns a list of cell disks, then you should verify if any grid
      disks exist:

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list griddisk where
    disktype=harddisk”
    dm01cel05: DATA_CD_00_dm01cel05          active
    dm01cel05: DATA_CD_01_dm01cel05          active
    dm01cel05: DATA_CD_02_dm01cel05          active
    dm01cel05: DATA_CD_03_dm01cel05          active
    dm01cel05: DATA_CD_04_dm01cel05          active
    dm01cel05: DATA_CD_05_dm01cel05          active
    dm01cel05: DATA_CD_06_dm01cel05          active
    dm01cel05: DATA_CD_07_dm01cel05          active
    dm01cel05: DATA_CD_08_dm01cel05          active
    dm01cel05: DATA_CD_09_dm01cel05          active
    dm01cel05: DATA_CD_10_dm01cel05          active
    dm01cel05: DATA_CD_11_dm01cel05          active
    dm01cel05: RECO_CD_00_dm01cel05          active
    dm01cel05: RECO_CD_01_dm01cel05          active
    dm01cel05: RECO_CD_02_dm01cel05          active
    dm01cel05: RECO_CD_03_dm01cel05          active
    dm01cel05: RECO_CD_04_dm01cel05          active
    dm01cel05: RECO_CD_05_dm01cel05          active
    dm01cel05: RECO_CD_06_dm01cel05          active
    dm01cel05: RECO_CD_07_dm01cel05          active
    dm01cel05: RECO_CD_08_dm01cel05          active
    dm01cel05: RECO_CD_09_dm01cel05          active
    dm01cel05: RECO_CD_10_dm01cel05          active
    dm01cel05: RECO_CD_11_dm01cel05          active
    dm01cel05: DBFS_DG_CD_02_dm01cel05      active
    dm01cel05: DBFS_DG_CD_03_dm01cel05      active
    dm01cel05: DBFS_DG_CD_04_dm01cel05      active
    dm01cel05: DBFS_DG_CD_05_dm01cel05      active
    dm01cel05: DBFS_DG_CD_06_dm01cel05      active
    dm01cel05: DBFS_DG_CD_07_dm01cel05      active
    dm01cel05: DBFS_DG_CD_08_dm01cel05      active
    dm01cel05: DBFS_DG_CD_09_dm01cel05      active
    dm01cel05: DBFS_DG_CD_10_dm01cel05      active
    dm01cel05: DBFS_DG_CD_11_dm01cel05      active
    In my case grid disks exists.
    • If no grid disks are created.
      There will be no output.

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list griddisk where
    disktype=harddisk”
    [root@dm01db01 ~]#

    • If any grid disks are
      present, then you must ensure they are not in use by looking in V$ASM_DISK and
      verifying the mount status is “closed” and the header status is
      “former” or “candidate”:

    dm01db01-orcldb1 {/home/oracle}:.
    oraenv
    ORACLE_SID = [orcldb1] ? +ASM1
    The Oracle base remains unchanged
    with value /u01/app/oracle
    dm01db01-+ASM1 {/home/oracle}:sqlplus
    / as sysasm
    SQL*Plus: Release 11.2.0.4.0
    Production on Mon Oct 17 10:01:15 2016
    Copyright (c) 1982, 2013,
    Oracle.  All rights reserved.
    Connected to:
    Oracle Database 11g Enterprise
    Edition Release 11.2.0.4.0 – 64bit Production
    With the Real Application
    Clusters and Automatic Storage Management options
    SQL> select name, path, state,
    mode_status,header_status, mount_status from v$asm_disk where header_status
    <> ‘MEMBER’ order by path;
    no rows selected
    • If it is OK to drop the grid
      disks, you can use the following command to drop set of grid disks (by prefix
      like, RECO):

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e drop griddisk all harddisk
    prefix=RECO”
    dm01cel05: GridDisk RECO_CD_00_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_01_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_02_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_03_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_04_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_05_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_06_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_07_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_08_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_09_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_10_dm01cel05
    successfully dropped
    dm01cel05: GridDisk RECO_CD_11_dm01cel05
    successfully dropped
    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e drop griddisk all harddisk
    prefix=DATA”
    dm01cel05: GridDisk DATA_CD_00_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_01_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_02_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_03_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_04_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_05_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_06_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_07_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_08_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_09_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_10_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DATA_CD_11_dm01cel05
    successfully dropped
    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e drop griddisk all harddisk prefix=DBFS_DG”
    dm01cel05: GridDisk DBFS_DG_CD_02_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_03_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_04_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_05_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_06_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_07_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_08_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_09_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_10_dm01cel05
    successfully dropped
    dm01cel05: GridDisk DBFS_DG_CD_11_dm01cel05
    successfully dropped
    • Verify Grid disks are dropped

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list griddisk where
    disktype=harddisk”
    [root@dm01db01 ~]#
    • Drop cell disks

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e drop celldisk all harddisk force”
    dm01cel05: CellDisk CD_00_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_01_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_02_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_03_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_04_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_05_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_06_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_07_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_08_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_09_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_10_dm01cel05
    successfully dropped
    dm01cel05: CellDisk CD_11_dm01cel05
    successfully dropped
    • If
      no cell disks exist, create cell disks on all hard disks (output will be
      similar to the following): 

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e create celldisk all harddisk”
    dm01cel05: CellDisk CD_00_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_01_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_02_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_03_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_04_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_05_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_06_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_07_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_08_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_09_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_10_dm01cel05
    successfully created
    dm01cel05: CellDisk CD_11_dm01cel05
    successfully created
    • Verify that celldisks were
      created (output should be similar to the following)

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list celldisk attributes
    name,status,freespace where disktype=harddisk”
    dm01cel05: CD_00_dm01cel05       normal 
    528.6875G
    dm01cel05: CD_01_dm01cel05       normal 
    528.6875G
    dm01cel05: CD_02_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_03_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_04_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_05_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_06_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_07_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_08_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_09_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_10_dm01cel05       normal 
    557.8125G
    dm01cel05: CD_11_dm01cel05       normal 
    557.8125G


    Create
    DATA and RECO grid disks at the same offsets found on the existing disks (outer
    tracks)

    • Verify the current sizes and
      offsets for existing grid disks (on the existing cells):

    [root@dm01db01
    ~]# dcli -g cell_group -l root “cellcli -e list griddisk attributes
    name,size | grep ‘CD_03’ | grep -i DBFS_DG”
    dm01cel01:
    DBFS_DG_CD_03_dm01cel01      29.125G
    dm01cel02:
    DBFS_DG_CD_03_dm01cel02      29.125G
    dm01cel03:
    DBFS_DG_CD_03_dm01cel03      29.125G
    dm01cel04:
    DBFS_DG_CD_03_dm01cel04      29.125G
    dm01cel06:
    DBFS_DG_CD_03_dm01cel06      29.125G
    dm01cel07:
    DBFS_DG_CD_03_dm01cel07      29.125G
    dm01cel08:
    DBFS_DG_CD_03_dm01cel08      29.125G
    dm01cel09:
    DBFS_DG_CD_03_dm01cel09      29.125G
    dm01cel10:
    DBFS_DG_CD_03_dm01cel10      29.125G
    dm01cel11:
    DBFS_DG_CD_03_dm01cel11      29.125G
    dm01cel12:
    DBFS_DG_CD_03_dm01cel12      29.125G
    dm01cel13:
    DBFS_DG_CD_03_dm01cel13      29.125G
    dm01cel14:
    DBFS_DG_CD_03_dm01cel14      29.125G
    [root@dm01db01
    ~]# dcli -g cell_group -l root “cellcli -e list griddisk attributes
    name,size,offset | grep ‘CD_03’ | grep -i data”
    dm01cel01:
    DATA_CD_03_dm01cel01          260G            32M
    dm01cel02:
    DATA_CD_03_dm01cel02          260G            32M
    dm01cel03:
    DATA_CD_03_dm01cel03          260G            32M
    dm01cel04:
    DATA_CD_03_dm01cel04          260G            32M
    dm01cel06:
    DATA_CD_03_dm01cel06          260G            32M
    dm01cel07:
    DATA_CD_03_dm01cel07          260G            32M
    dm01cel08:
    DATA_CD_03_dm01cel08          260G            32M
    dm01cel09:
    DATA_CD_03_dm01cel09          260G            32M
    dm01cel10:
    DATA_CD_03_dm01cel10          260G            32M
    dm01cel11:
    DATA_CD_03_dm01cel11          260G            32M
    dm01cel12:
    DATA_CD_03_dm01cel12          260G            32M
    dm01cel13:
    DATA_CD_03_dm01cel13          260G            32M
    dm01cel14:
    DATA_CD_03_dm01cel14          260G            32M
    [root@dm01db01
    ~]# dcli -g cell_group -l root “cellcli -e list griddisk attributes
    name,size,offset | grep ‘CD_03’ | grep -i reco”
    dm01cel01:
    RECO_CD_03_dm01cel01         
    268.6875G       260.046875G
    dm01cel02:
    RECO_CD_03_dm01cel02         
    268.6875G       260.046875G
    dm01cel03:
    RECO_CD_03_dm01cel03         
    268.6875G       260.046875G
    dm01cel04:
    RECO_CD_03_dm01cel04         
    268.6875G       260.046875G
    dm01cel06:
    RECO_CD_03_dm01cel06         
    268.6875G       260.046875G
    dm01cel07:
    RECO_CD_03_dm01cel07         
    268.6875G       260.046875G
    dm01cel08:
    RECO_CD_03_dm01cel08          268.6875G       260.046875G
    dm01cel09:
    RECO_CD_03_dm01cel09         
    268.6875G       260.046875G
    dm01cel10:
    RECO_CD_03_dm01cel10         
    268.6875G       260.046875G
    dm01cel11:
    RECO_CD_03_dm01cel11         
    268.6875G       260.046875G
    dm01cel12:
    RECO_CD_03_dm01cel12         
    268.6875G       260.046875G
    dm01cel13:
    RECO_CD_03_dm01cel13         
    268.6875G       260.046875G
    dm01cel14:
    RECO_CD_03_dm01cel14         
    268.6875G       260.046875G

    • Create similar grid disks on the
      new cells with the same size; be sure to create them in order of offset
      position, from smallest to largest offset. 

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e CREATE GRIDDISK ALL HARDDISK
    PREFIX=’DATA’, size=260G”
    dm01cel05: GridDisk DATA_CD_00_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_01_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_02_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_03_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_04_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_05_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_06_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_07_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_08_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_09_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_10_dm01cel05
    successfully created
    dm01cel05: GridDisk DATA_CD_11_dm01cel05
    successfully created

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e CREATE GRIDDISK ALL HARDDISK
    PREFIX=’RECO’, size=268.6875G”
    dm01cel05: GridDisk RECO_CD_00_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_01_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_02_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_03_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_04_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_05_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_06_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_07_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_08_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_09_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_10_dm01cel05
    successfully created
    dm01cel05: GridDisk RECO_CD_11_dm01cel05
    successfully created

    • Validate that grid disks were
      created properly on the new cells (size and offset matches the original cells)

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list griddisk attributes
    name,size,offset | grep -i data”
    dm01cel05: DATA_CD_00_dm01cel05  260G            32M
    dm01cel05: DATA_CD_01_dm01cel05  260G            32M
    dm01cel05: DATA_CD_02_dm01cel05  260G            32M
    dm01cel05: DATA_CD_03_dm01cel05  260G            32M
    dm01cel05: DATA_CD_04_dm01cel05  260G            32M
    dm01cel05: DATA_CD_05_dm01cel05  260G            32M
    dm01cel05: DATA_CD_06_dm01cel05  260G            32M
    dm01cel05: DATA_CD_07_dm01cel05  260G            32M
    dm01cel05: DATA_CD_08_dm01cel05  260G            32M
    dm01cel05: DATA_CD_09_dm01cel05  260G            32M
    dm01cel05: DATA_CD_10_dm01cel05  260G            32M
    dm01cel05: DATA_CD_11_dm01cel05  260G            32M

    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e list griddisk attributes
    name,size,offset | grep -i reco”
    dm01cel05: RECO_CD_00_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_01_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_02_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_03_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_04_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_05_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_06_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_07_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_08_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_09_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_10_dm01cel05  268.6875G       260.046875G
    dm01cel05: RECO_CD_11_dm01cel05  268.6875G       260.046875G

    Create
    the new DBFS_DG grid disks only on disks 3 through 12 of the new storage
    servers

    • List
      the size of the DBFS_DG grid disks on the original cells (they should all be
      the same size) – we’ll call this size, “dbfs_dg_size”:

    [root@dm01db01 ~]# dcli -c
    dm01cel01 -l root “cellcli -e list griddisk attributes name,size | grep ‘DBFS_DG’”
    dm01cel01: DBFS_DG_CD_02_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_03_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_04_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_05_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_06_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_07_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_08_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_09_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_10_dm01cel01      29.125G
    dm01cel01: DBFS_DG_CD_11_dm01cel01      29.125G

    • The DBFS_DG grid disks will be
      automatically placed just after the RECO grid disks on the innermost tracks.

    NOTE: This
    command will “fail” on the first and second disks of each cell, but
    that is expected since there is no more free space on those disks due to the
    system partitions. The “error” message is similar to: 
    “Cell disks were skipped because they had no freespace for grid disks:
    CD_00_dm01cel05, CD_01_dm01cel05.”


    [root@dm01db01 ~]# dcli -g
    new_cell_group -l root “cellcli -e CREATE GRIDDISK ALL HARDDISK PREFIX=’DBFS_DG’,
    size=29.125G”
    dm01cel05: Cell disks were
    skipped because they had no freespace for grid disks: CD_00_dm01cel05, CD_01_dm01cel05.
    dm01cel05: GridDisk DBFS_DG_CD_02_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_03_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_04_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_05_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_06_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_07_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_08_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_09_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_10_dm01cel05
    successfully created
    dm01cel05: GridDisk DBFS_DG_CD_11_dm01cel05
    successfully created

    Validate
    that Grid Disks on all Cells are the Same Size

    • All Cells at once:

    [root@dm01db01 ~]# dcli -g
    cell_group -l root “cellcli -e list griddisk attributes name,size,offset |
    grep -i data”
    dm01cel01: DATA_CD_00_dm01cel01          260G            32M
    dm01cel01: DATA_CD_01_dm01cel01          260G            32M
    dm01cel01: DATA_CD_02_dm01cel01          260G            32M
    dm01cel01: DATA_CD_03_dm01cel01          260G            32M
    dm01cel01: DATA_CD_04_dm01cel01          260G            32M
    dm01cel01: DATA_CD_05_dm01cel01          260G            32M
    dm01cel01: DATA_CD_06_dm01cel01          260G            32M
    dm01cel01: DATA_CD_07_dm01cel01          260G            32M
    dm01cel01: DATA_CD_08_dm01cel01          260G            32M
    dm01cel01: DATA_CD_09_dm01cel01          260G            32M
    dm01cel01: DATA_CD_10_dm01cel01          260G            32M
    dm01cel01: DATA_CD_11_dm01cel01          260G            32M
    …..
    dm01cel14: DATA_CD_00_dm01cel14          260G            32M
    dm01cel14: DATA_CD_01_dm01cel14          260G            32M
    dm01cel14: DATA_CD_02_dm01cel14          260G            32M
    dm01cel14: DATA_CD_03_dm01cel14          260G            32M
    dm01cel14: DATA_CD_04_dm01cel14          260G            32M
    dm01cel14: DATA_CD_05_dm01cel14          260G            32M
    dm01cel14: DATA_CD_06_dm01cel14          260G            32M
    dm01cel14: DATA_CD_07_dm01cel14          260G            32M
    dm01cel14: DATA_CD_08_dm01cel14          260G            32M
    dm01cel14: DATA_CD_09_dm01cel14          260G            32M
    dm01cel14: DATA_CD_10_dm01cel14          260G            32M
    dm01cel14: DATA_CD_11_dm01cel14          260G            32M

    [root@dm01db01 ~]# dcli -g
    cell_group -l root “cellcli -e list griddisk attributes name,size,offset |
    grep -i reco”
    dm01cel01: RECO_CD_00_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_01_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_02_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_03_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_04_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_05_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_06_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_07_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_08_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_09_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_10_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_11_dm01cel01          268.6875G       260.046875G
    ……
    dm01cel13: RECO_CD_11_dm01cel13          268.6875G       260.046875G
    dm01cel14: RECO_CD_00_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_01_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_02_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_03_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_04_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_05_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_06_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_07_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_08_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_09_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_10_dm01cel14          268.6875G       260.046875G
    dm01cel14: RECO_CD_11_dm01cel14          268.6875G       260.046875G

    [root@dm01db01 ~]# dcli -g
    cell_group -l root “cellcli -e list griddisk attributes name,size,offset |
    grep -i DBFS_DG”
    dm01cel01: DBFS_DG_CD_02_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_03_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_04_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_05_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_06_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_07_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_08_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_09_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_10_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_11_dm01cel01      29.125G         528.734375G
    ….
    dm01cel14: DBFS_DG_CD_02_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_03_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_04_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_05_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_06_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_07_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_08_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_09_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_10_dm01cel14      29.125G         528.734375G
    dm01cel14: DBFS_DG_CD_11_dm01cel14      29.125G         528.734375G

    • One cell at a time:

    [root@dm01db01 ~]# dcli -c dm01cel01
    -l root “cellcli -e list griddisk attributes name,size,offset | grep -i
    data”
    dm01cel01: DATA_CD_00_dm01cel01          260G            32M
    dm01cel01: DATA_CD_01_dm01cel01          260G            32M
    dm01cel01: DATA_CD_02_dm01cel01          260G            32M
    dm01cel01: DATA_CD_03_dm01cel01          260G            32M
    dm01cel01: DATA_CD_04_dm01cel01          260G            32M
    dm01cel01: DATA_CD_05_dm01cel01          260G            32M
    dm01cel01: DATA_CD_06_dm01cel01          260G            32M
    dm01cel01: DATA_CD_07_dm01cel01          260G            32M
    dm01cel01: DATA_CD_08_dm01cel01          260G            32M
    dm01cel01: DATA_CD_09_dm01cel01          260G            32M
    dm01cel01: DATA_CD_10_dm01cel01          260G            32M
    dm01cel01: DATA_CD_11_dm01cel01          260G            32M

    [root@dm01db01 ~]# dcli -c dm01cel01
    -l root “cellcli -e list griddisk attributes name,size,offset | grep -i
    reco”
    dm01cel01: RECO_CD_00_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_01_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_02_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_03_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_04_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_05_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_06_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_07_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_08_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_09_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_10_dm01cel01          268.6875G       260.046875G
    dm01cel01: RECO_CD_11_dm01cel01          268.6875G       260.046875G

    [root@dm01db01 ~]# dcli -c dm01cel01
    -l root “cellcli -e list griddisk attributes name,size,offset | grep -i DBFS_DG”
    dm01cel01: DBFS_DG_CD_02_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_03_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_04_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_05_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_06_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_07_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_08_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_09_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_10_dm01cel01      29.125G         528.734375G
    dm01cel01: DBFS_DG_CD_11_dm01cel01      29.125G         528.734375G

    Add
    the DBFS_DG grid disks to the existing DBFS_DG disk group

    • Update the cellip.ora file

    [root@dm01db01 ~]# vi
    /etc/oracle/cell/network-config/cellip.ora

    [root@dm01db01 network-config]#
    cat /etc/oracle/cell/network-config/cellip.ora
    cell=”192.168.2.9″
    cell=”192.168.2.10″
    cell=”192.168.2.11″
    cell=”192.168.2.12″
    cell=”192.168.2.13″
    cell=”192.168.2.14″
    cell=”192.168.2.15″
    cell=”192.168.2.16″
    cell=”192.168.2.17″
    cell=”192.168.2.18″
    cell=”192.168.2.19″
    cell=”192.168.2.20″
    cell=”192.168.2.21″
    cell=”192.168.2.22″

    [root@dm01db01 ~]# cd
    /etc/oracle/cell/network-config/

    [root@dm01db01 network-config]#
    dcli -g ~/dbs_group -l root -d /etc/oracle/cell/network-config -f cellip.ora

    [root@dm01db01 network-config]#
    dcli -g ~/dbs_group -l root ls -l  /etc/oracle/cell/network-config/cellip.ora
    dm01db01: -rwxr-xr-x 1 oracle
    oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
    dm01db02: -rwx—— 1 oracle
    oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
    dm01db03: -rwx—— 1 oracle
    oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
    dm01db04: -rwx—— 1 oracle
    oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
    dm01db05: -rwx—— 1 oracle
    oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
    dm01db06: -rwx—— 1 oracle
    oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
    dm01db07: -rwx—— 1 oracle
    oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
    dm01db08: -rwx—— 1 oracle oinstall
    279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora

    dm01db01-orcldb1 {/home/oracle}:.
    oraenv
    ORACLE_SID = [orcldb1] ? +ASM1
    The Oracle base remains unchanged
    with value /u01/app/oracle
    dm01db01-+ASM1
    {/home/oracle}:sqlplus / as sysasm

    SQL*Plus: Release 11.2.0.4.0
    Production on Mon Oct 17 11:18:34 2016

    Copyright (c) 1982, 2013,
    Oracle.  All rights reserved.

    Connected to:
    Oracle Database 11g Enterprise
    Edition Release 11.2.0.4.0 – 64bit Production
    With the Real Application
    Clusters and Automatic Storage Management options

    SQL> alter diskgroup DBFS_DG
    add disk ‘o/*/DBFS_DG*dm01cel05’ 
    rebalance power 32 NOWAIT;

    Diskgroup altered.

    Add
    the DATA and RECO grid disks to the existing DATA and RECO disk groups

    • From
      one DB node add the new DATA disks to the DATA disk group (replace the pattern
      below ” ‘o/*/DATA_DM01*new_cel01’, ‘o/*/DATA_DM01*new_cel02’,
      ‘o/*/DATA_DM01*new_cel03′” with list of actual new cells):

    SQL>
    alter diskgroup DATA add disk ‘o/*/DATA*dm01cel05’ rebalance power 32 NOWAIT;

    Diskgroup
    altered.

    •  From
      another DB node add the new RECO disks to the RECO disk group (replace the
      pattern below, ”
      ‘o/*/RECO_DM01*new_cel01′,’o/*/RECO_DM01*new_cel02′,’o/*/RECO_DM01*new_cel03’
      ” with list of actual new cells):

    SQL> alter diskgroup RECO add
    disk ‘o/*/RECO*dm01cel05’ rebalance power 32 NOWAIT;

    Diskgroup altered.

    • Monitor the progress of the
      rebalance activities using this query (you can proceed with the rest of the
      steps while this is continuing):

    SQL> select * from
    gv$asm_operation;

      
    INST_ID GROUP_NUMBER OPERA STAT     
    POWER     ACTUAL      SOFAR  
    EST_WORK   EST_RATE EST_MINUTES
    ERROR_CODE
    ———- ———— —–
    —- ———- ———- ———- ———- ———- ———–
    ——————————————–
             7            1 REBAL WAIT         32
             3            1 REBAL WAIT         32
             2            1 REBAL WAIT         32
             1            1 REBAL WAIT         32
             6            1 REBAL WAIT         32
             8            1 REBAL WAIT         32
             4            1 REBAL WAIT         32
             5            1 REBAL RUN          32         32     
    41666      45014      17263           0

    8 rows selected.

    SQL> select * from
    gv$asm_operation;

    no rows selected


    Validate
    that DATA, RECO, and DBFS_DG contain disks from new cells


    Before:
    dm01db01-orcldb1 {/home/oracle}:.
    oraenv
    ORACLE_SID = [orcldb1] ? +ASM1
    The Oracle base remains unchanged
    with value /u01/app/oracle
    dm01db01-+ASM1
    {/home/oracle}:sqlplus / as sysasm

    SQL*Plus: Release 11.2.0.4.0
    Production on Mon Oct 17 10:43:16 2016

    Copyright (c) 1982, 2013,
    Oracle.  All rights reserved.

    Connected to:
    Oracle Database 11g Enterprise
    Edition Release 11.2.0.4.0 – 64bit Production
    With the Real Application Clusters
    and Automatic Storage Management options

    SQL> set lines 200
    SQL> set pages 200
    SQL> select
    d.failgroup,dg.name, sum(d.total_mb) cell_total_mb, count(1) cell_disk_count
    from v$asm_disk d, v$asm_diskgroup dg where d.group_number = dg.group_number
    group by d.failgroup, dg.name order by d.failgroup, dg.name;

    FAILGROUP                      NAME                           CELL_TOTAL_MB
    CELL_DISK_COUNT
    ——————————
    —————————— ————- —————
    DM01CEL01                      DATA                                 3194880              12
    DM01CEL01                      RECO                                 3301632              12
    DM01CEL01                      DBFS_DG                              298240              10
    DM01CEL02                      DATA                                 2928640              12
    DM01CEL02                      RECO                                 3026496              12
    DM01CEL02                      DBFS_DG                              268416              10
    DM01CEL03                      DATA                                 3194880              12
    DM01CEL03                      RECO                                 3301632              12
    DM01CEL03                      DBFS_DG                              298240              10
    DM01CEL04                      DATA                                 3194880              12
    DM01CEL04                      RECO                                 3301632              12
    DM01CEL04                      DBFS_DG                              298240              10
    DM01CEL06                      DATA                                 3194880              12
    DM01CEL06                      RECO                                 3301632              12
    DM01CEL06                      DBFS_DG                              298240              10
    DM01CEL07                      DATA                                 2928640              11
    DM01CEL07                      RECO                                 3026496              11
    DM01CEL07                      DBFS_DG                              268416              10
    DM01CEL08                      DATA                                 3194880              12
    DM01CEL08                      RECO                                 3301632              12
    DM01CEL08                      DBFS_DG                              298240              10
    DM01CEL09                      DATA                                 2396160               12
    DM01CEL09                      RECO                                 2476224               12
    DM01CEL09                      DBFS_DG                              238592               10
    DM01CEL10                      DATA                                 3194880              12
    DM01CEL10                      RECO                                 3301632              12
    DM01CEL10                      DBFS_DG                              298240              10
    DM01CEL11                      DATA                                 2928640              12
    DM01CEL11                      RECO                                 3026496              12
    DM01CEL11                      DBFS_DG                              298240              10
    DM01CEL12                      DATA                                 3194880              12
    DM01CEL12                      RECO                                 3301632              12
    DM01CEL12                      DBFS_DG                              298240              10
    DM01CEL13                      DATA                                 3194880              12
    DM01CEL13                      RECO                                 3301632              12
    DM01CEL13                      DBFS_DG                              298240              10
    DM01CEL14                      DATA                                 3194880              12
    DM01CEL14                      RECO                                 3301632              12
    DM01CEL14                      DBFS_DG                              298240              10

    39 rows selected.

    After
    SQL> set lines 200
    SQL> set pages 200
    SQL> select
    d.failgroup,dg.name, sum(d.total_mb) cell_total_mb, count(1) cell_disk_count
    from v$asm_disk d, v$asm_diskgroup dg where d.group_number = dg.group_number
    group by d.failgroup, dg.name order by d.failgroup, dg.name;

    FAILGROUP                      NAME                           CELL_TOTAL_MB
    CELL_DISK_COUNT
    ——————————
    —————————— ————- —————
    DM01CEL01                      DATA                                 3194880              12
    DM01CEL01                      RECO                                 3301632              12
    DM01CEL01                      DBFS_DG                              298240              10
    DM01CEL02                      DATA                                 2928640              12
    DM01CEL02                      RECO                                 3026496              12
    DM01CEL02                      DBFS_DG                              268416               10
    DM01CEL03                      DATA                                 3194880              12
    DM01CEL03                      RECO                                 3301632              12
    DM01CEL03                      DBFS_DG                              298240              10
    DM01CEL04                      DATA                                 2928640              12
    DM01CEL04                      RECO                                 3026496              12
    DM01CEL04                      DBFS_DG                              268416               10
    DM01CEL05                      DATA                                 3194880              12
    DM01CEL05                      RECO                                 3301632              12
    DM01CEL05                      DBFS_DG                              298240              10
    DM01CEL06                      DATA                                 3194880              12
    DM01CEL06                      RECO                                 3301632              12
    DM01CEL06                      DBFS_DG                              298240              10
    DM01CEL07                      DATA                                 2928640              12
    DM01CEL07                      RECO                                 3026496              12
    DM01CEL07                      DBFS_DG                              268416               10
    DM01CEL08                      DATA                                 3194880              12
    DM01CEL08                      RECO                                 3301632              12
    DM01CEL08                      DBFS_DG                              298240              10
    DM01CEL09                      DATA                                 2396160               12
    DM01CEL09                      RECO                                 2476224               12
    DM01CEL09                      DBFS_DG                              238592               10
    DM01CEL10                      DATA                                 3194880              12
    DM01CEL10                      RECO                                 3301632              12
    DM01CEL10                      DBFS_DG                              298240              10
    DM01CEL11                      DATA                                 2928640              12
    DM01CEL11                      RECO                                 3026496              12
    DM01CEL11                      DBFS_DG                              298240              10
    DM01CEL12                      DATA                                 3194880              12
    DM01CEL12                      RECO                                 3301632              12
    DM01CEL12                      DBFS_DG                              298240              10
    DM01CEL13                      DATA                                 3194880              12
    DM01CEL13                      RECO                                 3301632              12
    DM01CEL13                      DBFS_DG                              298240              10
    DM01CEL14                      DATA                                 3194880              12
    DM01CEL14                      RECO                                 3301632              12
    DM01CEL14                      DBFS_DG                              298240              10

    42 rows selected.

    Ensure that there are no offline
    disks in any disk group:
    SQL> select
    name,total_mb,free_mb,offline_disks from v$asm_diskgroup;
    NAME                             TOTAL_MB    FREE_MB OFFLINE_DISKS
    ——————————
    ———- ———- ————-
    DATA                             42864640   41713708             0
    RECO                             44296896   44293768             0
    DBFS_DG                          4026240    4023232             0
    Conclusion
    In this article we have learned
    how to add a storage cell to an existing Exadata Database machine of same disk
    size.

  • Move OCR/Voting Disk/SPFILE to New ASM Disk Group on Exadata

    Overview
    In Oracle database 11.2 and above it is recommended to place the OCR and voting disks in ASM disk group. In my case the OCR, Voting disk and SPFILE are stored in +DBFS_DG disk group. I would like to move them to +DATA disk group. The ASM disk groups are configured using ASM Normal redundancy for protection.


    In this article I will demonstrate how move/replace OCR, Voting disk and SPFILE from one disk group to another ASM disk group.


    Environment

    • Oracle Exadata X2-2
    • ASM Disk Groups +DATA, +RECO, +DBFS_DG
    • Oracle GI/RDBMS version 11.2.0.4

    Steps to move/replacec OCR/Voting disk/SPFILE to new disk group.

    • Locate the current OCR, Voting disk  and SPFILE locations

    OCR file location:


    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/ocrcheck
    Status of Oracle Cluster Registry is as follows :
             Version                  :          3
             Total space (kbytes)     :     262120
             Used space (kbytes)      :       4808
             Available space (kbytes) :     257312
             ID                       : 1556021409
             Device/File Name         :   +DBFS_DG
                                        Device/File integrity check succeeded


                                        Device/File not configured


                                        Device/File not configured


                                        Device/File not configured


                                        Device/File not configured


             Cluster registry integrity check succeeded


             Logical corruption check succeeded


    Voting Disk location:


    dm01db01-+ASM1 {/home/oracle}:$ORACLE_HOME/bin/crsctl query css votedisk
    ##  STATE    File Universal Id                File Name Disk group
    —  —–    —————–                ——— ———
     1. ONLINE   8f84a639c9ac4fbabf581bb6972f3f8f (o/192.168.1.9/DBFS_DG_CD_02_dm01cel01) [DBFS_DG]
     2. ONLINE   3b079cf1cba24f42bfda7a8b18f0e22f (o/192.168.1.10/DBFS_DG_CD_02_dm01cel02) [DBFS_DG]
     3. ONLINE   ad837de087dd4f6bbf72800ecd028573 (o/192.168.1.11/DBFS_DG_CD_07_dm01cel03) [DBFS_DG]


    ASM SPFILE location:


    SQL> show parameter spfile


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    spfile                               string      +DBFS_DG/dm01-cluster/asmparameterfile/registry.253.790181847

    • Create New ASM diskgroup with suitable redundancy for OCR and Voting files.

    Here I am moving OCR/Voting disks from +DBFS_DG to +DATA with NORMAL redundancy
    The disk group +DATA already exists.

    • Ensure that the new diskgroup is mounted on all cluster nodes.

    ASMCMD> lsdg
    State    Type    Rebal  Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  
    Offline_disks  Voting_files  Name
    MOUNTED  NORMAL  N         512   4096  4194304  71470080  58440508           433152        29003678              
    0             N  DATA/
    MOUNTED  NORMAL  N         512   4096  4194304   4085888   4068444            29824         2019310              
    0             Y  DBFS_DG/
    MOUNTED  NORMAL  N         512   4096  4194304   9460160   7874420           108224         3883098              
    0             N  RECO/


    SQL> select name, state, type from v$asm_diskgroup;


    NAME                           STATE       TYPE
    —————————— ———– ——
    DATA                           MOUNTED     NORMAL
    DBFS_DG                        MOUNTED     NORMAL
    RECO                           MOUNTED     NORMAL

    • Move OCR and Vote file from +DBFS_DG to +DATA

    dm01db01-+ASM1 {/home/oracle}:$ORACLE_HOME/bin/ocrconfig -add +DATA
    PROT-20: Insufficient permission to proceed. Require privileged user


    Switch to root user and execute the command again.


    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/ocrconfig -add +DATA


    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/ocrcheck
    Status of Oracle Cluster Registry is as follows :
             Version                  :          3
             Total space (kbytes)     :     262120
             Used space (kbytes)      :       4808
             Available space (kbytes) :     257312
             ID                       : 1556021409
             Device/File Name         :   +DBFS_DG
                                        Device/File integrity check succeeded
             Device/File Name         : +DATA
                                        Device/File integrity check succeeded


                                        Device/File not configured


                                        Device/File not configured


                                        Device/File not configured


             Cluster registry integrity check succeeded


             Logical corruption check succeeded


    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/ocrconfig -delete +DBFS_DG


    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/ocrcheck
    Status of Oracle Cluster Registry is as follows :
             Version                  :          3
             Total space (kbytes)     :     262120
             Used space (kbytes)      :       4808
             Available space (kbytes) :     257312
             ID                       : 1556021409
             Device/File Name         : +DATA
                                        Device/File integrity check succeeded


                                        Device/File not configured


                                        Device/File not configured


                                        Device/File not configured


                                        Device/File not configured


             Cluster registry integrity check succeeded


             Logical corruption check succeeded


    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/crsctl replace votedisk +DATA
    Successful addition of voting disk 560494cd61f64fd3bfeca18aa876e5bc.
    Successful addition of voting disk 15b8c8b5b80c4f7cbf3eadd516af60bc.
    Successful addition of voting disk 7cff6119f09b4f99bfd0d6a1fd399a00.
    Successful deletion of voting disk 8f84a639c9ac4fbabf581bb6972f3f8f.
    Successful deletion of voting disk 3b079cf1cba24f42bfda7a8b18f0e22f.
    Successful deletion of voting disk ad837de087dd4f6bbf72800ecd028573.
    Successfully replaced voting disk group with +DATA.
    CRS-4266: Voting file(s) successfully replaced

    [root@dm01db01 ~]# /u01/app/11.2.0.4/grid/bin/crsctl query css votedisk
    ##  STATE    File Universal Id                File Name Disk group
    —  —–    —————–                ——— ———
     1. ONLINE   560494cd61f64fd3bfeca18aa876e5bc (o/192.168.3.10/DATA_CD_00_dm01cel02) [DATA]
     2. ONLINE   15b8c8b5b80c4f7cbf3eadd516af60bc (o/192.168.3.11/DATA_CD_10_dm01cel03) [DATA]
     3. ONLINE   7cff6119f09b4f99bfd0d6a1fd399a00 (o/192.168.3.12/DATA_CD_00_dm01cel04) [DATA]
    Located 3 voting disk(s).

    • Change ASM SPFILE location from +DBFS_DG to +DATA Diskgroup.

    SQL> show parameter spfile


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    spfile                               string      +DBFS_DG/dm01-cluster/asmparameterfile/registry.253.790181847


    SQL> create pfile=’/tmp/initasm.ora’ from spfile;


    File created.


    SQL> create spfile=’+DATA’ from pfile=’/tmp/initasm.ora’;


    File created.


    Now GPNPTOOL will get updated with new ASM SPFILE location.


    That can be verified by below command :


    dm01db01-+ASM1 {/home/oracle}:$ORACLE_HOME/bin/gpnptool get
    Warning: some command line parameters were defaulted. Resulting command line:
             /u01/app/11.2.0.4/grid/bin/gpnptool.bin get -o-<?xml version=”1.0″ encoding=”UTF-8″?><gpnp:GPnP-Profile Version=”1.0″ xmlns=”http://www.grid-pnp.org/2005/11/gpnp-profile” xmlns:gpnp=”http://www.grid-pnp.org/2005/11/gpnp-profile”xmlns:orcl=”http://www.oracle.com/gpnp/2005/11/gpnp-profile” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd” ProfileSequence=”5″ 
    ClusterUId=”154e3eadf63effb1bf594bae93566a45″ ClusterName=”dm01-cluster” PALocation=””><gpnp:Network-Profile><gpnp:HostNetwork id=”gen” HostName=”*”><gpnp:Network id=”net1″ IP=”10.204.26.1″ Adapter=”bondeth0″ Use=”public”/><gpnp:Network id=”net2″ IP=”192.168.0.0″ Adapter=”bondib0″ Use=”cluster_interconnect”/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id=”css” DiscoveryString=”+asm” LeaseDuration=”400″/><orcl:ASM-Profile id=”asm” DiscoveryString=”o/*/*” SPFile=”+DATA/dm01-cluster/asmparameterfile/registry.253.934093287″/><ds:Signature xmlns:ds=”http://www.w3.org/2000/09/xmldsig#”><ds:SignedInfo><ds:CanonicalizationMethod 
    Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”/><ds:SignatureMethod Algorithm=”http://www.w3.org/2000/09/xmldsig#rsa-sha1″/><ds:Reference URI=””><ds:Transforms><ds:Transform Algorithm=”http://www.w3.org/2000/09/xmldsig#enveloped-signature”/><ds:Transform Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”> <InclusiveNamespaces xmlns=”http://www.w3.org/2001/10/xml-exc-c14n#” PrefixList=”gpnp orcl xsi”/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm=”http://www.w3.org/2000/09/xmldsig#sha1″/>ds:DigestValue>SEdtgDVCAmvafpr2sb2YbZh0NvE=</ds:DigestValue></ds:Reference></ds:SignedInfo> ds:SignatureValue>AcWa+ZnR6F+qAoCAy3uqZPSdxPbYnXpMbaTozjfeQb0HEtQsK7nT3r
    +jO8AZkFUXqvSZa77D9d18/mYPXiPdQKJndpr+4cIM1Yg8K+fSC4HgD8hUgh5vl7YYJ2T9YoYVyfSbX6EQFOKZMKHM5IGZk0d8ls+4AD/1OO+LhijtEM4=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>
    Success.

    • Shutdown and Start the CRS on all the nodes

    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘/u01/app/11.2.0.4/grid/bin/crsctl stop crs’
    dm01db01: CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db01’
    dm01db01: CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.DATA_dm01.dg’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.DBFS_DG.dg’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.RECO_dm01.dg’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.dm01db01.vip’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ttsrv_dm01db01’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.LISTENER_IB.lsnr’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ttsrv_dm01db01’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ttrmi_dm01db01’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.LISTENER_IB.lsnr’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.dm01db01-ibvip.vip’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ttrmi_dm01db01’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.dm01db01.vip’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.RECO_dm01.dg’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.dm01db01-ibvip.vip’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.registry.acfs’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.DATA_dm01.dg’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.DBFS_DG.dg’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2679: Attempting to clean ‘ora.DBFS_DG.dg’ on ‘dm01db01’
    dm01db01: CRS-2681: Clean of ‘ora.DBFS_DG.dg’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.net2.network’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.net2.network’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.ons’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db01’ has completed
    dm01db01: CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db01’
    dm01db01: CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.crf’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db01’
    dm01db01: CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db01’ succeeded
    dm01db01: CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db01’ has completed
    dm01db01: CRS-4133: Oracle High Availability Services has been stopped.
    dm01db02: CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db02’
    dm01db02: CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.LISTENER_IB.lsnr’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ttsrv_dm01db02’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.DATA_dm01.dg’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.DBFS_DG.dg’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.RECO_dm01.dg’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ttsrv_dm01db02’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ttrmi_dm01db02’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ttrmi_dm01db02’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.LISTENER_IB.lsnr’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.dm01db02-ibvip.vip’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.dm01db02.vip’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.RECO_dm01.dg’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.registry.acfs’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.dm01db02.vip’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.scan1.vip’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.dm01db02-ibvip.vip’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.DATA_dm01.dg’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.DBFS_DG.dg’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2679: Attempting to clean ‘ora.DBFS_DG.dg’ on ‘dm01db02’
    dm01db02: CRS-2681: Clean of ‘ora.DBFS_DG.dg’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.net2.network’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.net2.network’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.ons’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db02’ has completed
    dm01db02: CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db02’
    dm01db02: CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.crf’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db02’
    dm01db02: CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db02’ succeeded
    dm01db02: CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db02’ has completed
    dm01db02: CRS-4133: Oracle High Availability Services has been stopped.
    dm01db03: CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db03’
    dm01db03: CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ttrmi_dm01db03’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.DATA_dm01.dg’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.DBFS_DG.dg’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.RECO_dm01.dg’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.LISTENER_IB.lsnr’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ttrmi_dm01db03’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.RECO_dm01.dg’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.dm01db03.vip’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.LISTENER_IB.lsnr’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.dm01db03-ibvip.vip’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.dm01db03-ibvip.vip’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.dm01db03.vip’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.registry.acfs’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.DATA_dm01.dg’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.DBFS_DG.dg’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2679: Attempting to clean ‘ora.DBFS_DG.dg’ on ‘dm01db03’
    dm01db03: CRS-2681: Clean of ‘ora.DBFS_DG.dg’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.net2.network’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.net2.network’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.ons’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db03’ has completed
    dm01db03: CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db03’
    dm01db03: CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.crf’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db03’
    dm01db03: CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db03’ succeeded
    dm01db03: CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db03’ has completed
    dm01db03: CRS-4133: Oracle High Availability Services has been stopped.
    dm01db04: CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db04’
    dm01db04: CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ttrmi_dm01db04’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.LISTENER_IB.lsnr’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.dm01db04-ibvip.vip’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.DATA_dm01.dg’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.DBFS_DG.dg’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.RECO_dm01.dg’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ttrmi_dm01db04’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.RECO_dm01.dg’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.dm01db04-ibvip.vip’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.dm01db04.vip’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.LISTENER_IB.lsnr’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.dm01db07-ibvip.vip’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.dm01db04.vip’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.dm01db07-ibvip.vip’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.registry.acfs’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.DATA_dm01.dg’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.DBFS_DG.dg’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2679: Attempting to clean ‘ora.DBFS_DG.dg’ on ‘dm01db04’
    dm01db04: CRS-2681: Clean of ‘ora.DBFS_DG.dg’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.net2.network’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.net2.network’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.ons’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db04’ has completed
    dm01db04: CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.crf’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db04’
    dm01db04: CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db04’
    dm01db04: CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db04’ succeeded
    dm01db04: CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db04’ has completed
    dm01db04: CRS-4133: Oracle High Availability Services has been stopped.
    dm01db05: CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db05’
    dm01db05: CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.cvu’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.DATA_dm01.dg’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.DBFS_DG.dg’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.RECO_dm01.dg’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ttrmi_dm01db05’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.LISTENER_IB.lsnr’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ttrmi_dm01db05’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.cvu’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2672: Attempting to start ‘ora.cvu’ on ‘dm01db08’
    dm01db05: CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.LISTENER_IB.lsnr’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.dm01db05-ibvip.vip’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.dm01db05.vip’ on ‘dm01db05’
    dm01db05: CRS-2676: Start of ‘ora.cvu’ on ‘dm01db08’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.RECO_dm01.dg’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.scan3.vip’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.dm01db05-ibvip.vip’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.registry.acfs’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.dm01db05.vip’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.DATA_dm01.dg’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.DBFS_DG.dg’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2679: Attempting to clean ‘ora.DBFS_DG.dg’ on ‘dm01db05’
    dm01db05: CRS-2681: Clean of ‘ora.DBFS_DG.dg’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.net2.network’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.net2.network’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.ons’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db05’ has completed
    dm01db05: CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.crf’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db05’
    dm01db05: CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db05’
    dm01db05: CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db05’ succeeded
    dm01db05: CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db05’ has completed
    dm01db05: CRS-4133: Oracle High Availability Services has been stopped.
    dm01db06: CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db06’
    dm01db06: CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.DATA_dm01.dg’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.DBFS_DG.dg’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.RECO_dm01.dg’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ttrmi_dm01db06’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.LISTENER_IB.lsnr’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ttrmi_dm01db06’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.LISTENER_IB.lsnr’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.dm01db06-ibvip.vip’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.dm01db06.vip’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.RECO_dm01.dg’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.dm01db06.vip’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.dm01db06-ibvip.vip’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.registry.acfs’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.DATA_dm01.dg’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.DBFS_DG.dg’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2679: Attempting to clean ‘ora.DBFS_DG.dg’ on ‘dm01db06’
    dm01db06: CRS-2681: Clean of ‘ora.DBFS_DG.dg’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.net2.network’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.net2.network’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.ons’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db06’ has completed
    dm01db06: CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db06’
    dm01db06: CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.crf’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db06’
    dm01db06: CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db06’ succeeded
    dm01db06: CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db06’ has completed
    dm01db06: CRS-4133: Oracle High Availability Services has been stopped.
    dm01db08: CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db08’
    dm01db08: CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.dm01db08.vip’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ttrmi_dm01db08’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.LISTENER_IB.lsnr’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.DATA_dm01.dg’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.DBFS_DG.dg’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.RECO_dm01.dg’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ttrmi_dm01db08’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.dm01db08.vip’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.RECO_dm01.dg’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.LISTENER_IB.lsnr’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.dm01db08-ibvip.vip’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.dm01db07.vip’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.cvu’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.cvu’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.dm01db08-ibvip.vip’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.dm01db07.vip’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.registry.acfs’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.scan2.vip’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.oc4j’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.DATA_dm01.dg’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.DBFS_DG.dg’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2679: Attempting to clean ‘ora.DBFS_DG.dg’ on ‘dm01db08’
    dm01db08: CRS-2681: Clean of ‘ora.DBFS_DG.dg’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.net2.network’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.net2.network’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.ons’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db08’ has completed
    dm01db08: CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.mdnsd’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.asm’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db08’
    dm01db08: CRS-2673: Attempting to stop ‘ora.crf’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.crf’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2677: Stop of ‘ora.gipcd’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘dm01db08’
    dm01db08: CRS-2677: Stop of ‘ora.gpnpd’ on ‘dm01db08’ succeeded
    dm01db08: CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘dm01db08’ has completed
    dm01db08: CRS-4133: Oracle High Availability Services has been stopped.


    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘/u01/app/11.2.0.4/grid/bin/crsctl start crs’
    dm01db01: CRS-4123: Oracle High Availability Services has been started.
    dm01db02: CRS-4123: Oracle High Availability Services has been started.
    dm01db03: CRS-4123: Oracle High Availability Services has been started.
    dm01db04: CRS-4123: Oracle High Availability Services has been started.
    dm01db05: CRS-4123: Oracle High Availability Services has been started.
    dm01db06: CRS-4123: Oracle High Availability Services has been started.
    dm01db08: CRS-4123: Oracle High Availability Services has been started.

    • Ensure ALL Cluster resources are started successfully using below sample commands :

    dm01db01-+ASM1 {/home/oracle}:$ORACLE_HOME/bin/crsctl stat res -init -t
    ——————————————————————————–
    NAME           TARGET  STATE        SERVER                   STATE_DETAILS
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.asm
          1        ONLINE  ONLINE       dm01db01                 Started
    ora.cluster_interconnect.haip
          1        ONLINE  ONLINE       dm01db01
    ora.crf
          1        ONLINE  ONLINE       dm01db01
    ora.crsd
          1        ONLINE  ONLINE       dm01db01
    ora.cssd
          1        ONLINE  ONLINE       dm01db01
    ora.cssdmonitor
          1        ONLINE  ONLINE       dm01db01
    ora.ctssd
          1        ONLINE  ONLINE       dm01db01                 OBSERVER
    ora.diskmon
          1        ONLINE  ONLINE       dm01db01
    ora.drivers.acfs
          1        ONLINE  ONLINE       dm01db01
    ora.evmd
          1        ONLINE  ONLINE       dm01db01
    ora.gipcd
          1        ONLINE  ONLINE       dm01db01
    ora.gpnpd
          1        ONLINE  ONLINE       dm01db01
    ora.mdnsd
          1        ONLINE  ONLINE       dm01db01


    dm01db01-+ASM1 {/home/oracle}:$ORACLE_HOME/bin/crsctl check cluster -all
    **************************************************************
    dm01db01:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************
    dm01db02:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************
    dm01db03:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************
    dm01db04:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************
    dm01db05:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************
    dm01db06:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************
    dm01db07:
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************

    dm01db08:

    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    **************************************************************
    dm01db01-+ASM1 {/home/oracle}:


    dm01db01-+ASM1 {/home/oracle}:$ORACLE_HOME/bin/crsctl stat res -t
    ——————————————————————————–
    NAME           TARGET  STATE        SERVER                   STATE_DETAILS
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.DATA_dm01.dg
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   ONLINE  ONLINE       dm01db07
                   ONLINE  ONLINE       dm01db08
    ora.DBFS_DG.dg
                   ONLINE  OFFLINE      dm01db01
                   ONLINE  OFFLINE      dm01db02
                   ONLINE  OFFLINE      dm01db03
                   ONLINE  OFFLINE      dm01db04
                   ONLINE  OFFLINE      dm01db05
                   ONLINE  OFFLINE      dm01db06
                   ONLINE  ONLINE       dm01db07
                   ONLINE  OFFLINE      dm01db08
    ora.LISTENER.lsnr
                   OFFLINE OFFLINE      dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   ONLINE  ONLINE       dm01db07
                   ONLINE  ONLINE       dm01db08
    ora.LISTENER_IB.lsnr
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   ONLINE  ONLINE       dm01db07
                   ONLINE  ONLINE       dm01db08
    ora.RECO_dm01.dg
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   ONLINE  ONLINE       dm01db07
                   ONLINE  ONLINE       dm01db08
    ora.asm
                   ONLINE  ONLINE       dm01db01                 Started
                   ONLINE  ONLINE       dm01db02                 Started
                   ONLINE  ONLINE       dm01db03                 Started
                   ONLINE  ONLINE       dm01db04                 Started
                   ONLINE  ONLINE       dm01db05                 Started
                   ONLINE  ONLINE       dm01db06                 Started
                   ONLINE  ONLINE       dm01db07                Started
                   ONLINE  ONLINE       dm01db08                 Started
    ora.gsd
                   OFFLINE OFFLINE      dm01db01
                   OFFLINE OFFLINE      dm01db02
                   OFFLINE OFFLINE      dm01db03
                   OFFLINE OFFLINE      dm01db04
                   OFFLINE OFFLINE      dm01db05
                   OFFLINE OFFLINE      dm01db06
                   OFFLINE OFFLINE      dm01db07
                   OFFLINE OFFLINE      dm01db08
    ora.net1.network
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   OFFLINE OFFLINE      dm01db07
                   ONLINE  ONLINE       dm01db08
    ora.net2.network
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   OFFLINE OFFLINE      dm01db07
                   ONLINE  ONLINE       dm01db08
    ora.ons
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   OFFLINE OFFLINE      dm01db07
                   ONLINE  ONLINE       dm01db08
    ora.registry.acfs
                   ONLINE  ONLINE       dm01db01
                   ONLINE  ONLINE       dm01db02
                   ONLINE  ONLINE       dm01db03
                   ONLINE  ONLINE       dm01db04
                   ONLINE  ONLINE       dm01db05
                   ONLINE  ONLINE       dm01db06
                   OFFLINE OFFLINE      dm01db07
                   ONLINE  ONLINE       dm01db08
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.LISTENER_SCAN1.lsnr
          1        ONLINE  ONLINE       dm01db08
    ora.LISTENER_SCAN2.lsnr
          1        ONLINE  ONLINE       dm01db06
    ora.LISTENER_SCAN3.lsnr
          1        ONLINE  ONLINE       dm01db05
    ora.dm01db01-ibvip.vip
          1        ONLINE  ONLINE       dm01db01
    ora.dm01db01.vip
          1        ONLINE  ONLINE       dm01db01
    ora.dm01db02-ibvip.vip
          1        ONLINE  ONLINE       dm01db02
    ora.dm01db02.vip
          1        ONLINE  ONLINE       dm01db02
    ora.dm01db03-ibvip.vip
          1        ONLINE  ONLINE       dm01db03
    ora.dm01db03.vip
          1        ONLINE  ONLINE       dm01db03
    ora.dm01db04-ibvip.vip
          1        ONLINE  ONLINE       dm01db04
    ora.dm01db04.vip
          1        ONLINE  ONLINE       dm01db04
    ora.dm01db05-ibvip.vip
          1        ONLINE  ONLINE       dm01db05
    ora.dm01db05.vip
          1        ONLINE  ONLINE       dm01db05
    ora.dm01db06-ibvip.vip
          1        ONLINE  ONLINE       dm01db06
    ora.dm01db06.vip
          1        ONLINE  ONLINE       dm01db06
    ora.dm01db08-ibvip.vip
          1        ONLINE  ONLINE       dm01db08
    ora.dm01db08.vip
          1        ONLINE  ONLINE       dm01db08
    ora.cvu
          1        ONLINE  ONLINE       dm01db03
    ora.oc4j
          1        ONLINE  ONLINE       dm01db03
    ora.scan1.vip
          1        ONLINE  ONLINE       dm01db08
    ora.scan2.vip
          1        ONLINE  ONLINE       dm01db06
    ora.scan3.vip
          1        ONLINE  ONLINE       dm01db05


    SQL> show parameter spfile


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    spfile                               string      +DATA/dm01-cluster/asmpar
                                                     ameterfile/registry.253.934093
                                                     287


    Conclusion
    In this article we have learned how to move/replace OCR, Voting disks and ASM SPFILE from one ASM disk group to another.

  • Configure ZFS on Exadata Database Machine

    Overview
    Once you migrate databases to Exadata Database Machine, its time to configure backups for them. Since Exadata uses ASM for storing database files, you can use RMAN to backup your databases. You can backup databases running on Exadata to FRA or external backup media such as ZFS. Customers purchase Exadata also purchase ZFS for backup and restore their databases. ZFS storage appliance is idea for Exadata as it provides faster backup solution.


    About ZFS Storage Appliance
    Oracle ZFS backup Appliance is a high performance backup solution for Oracle engineered systems (Exadata, SuperCluster and ODA) that reduces backup and recovery time objectives. 


    The Oracle ZFS Backup Appliance is fully tested, validated and supported backup appliance specifically tuned for Oracle engineered systems backup and recovery.


    Benefits of using ZFS Storage Appliance:

    • Greater Performance
    • Cost effectiveness
    • Simplified management
    • Reduced risk

    The Oracle ZFS Storage Appliance is ideal for backing up the databases running on Exadata Database machine, Oracle SuperCluster and Oracle Database Appliance (ODA). Many Customers purchased Exadata uses ZFS for backing up databases.


    About ESBU Utility
    The Oracle Engineered Systems Backup Utility (ESBU) formerly known as “Exadata Backup Configuration Utility” for Oracle ZFS Storage Appliance configures the Oracle ZFS Storage Appliance and the database compute nodes of a particular engineered system (Exadata, SuperCluster or ODA) through a command-line wizard. 


    Download ESBU utility from OTN at:


    http://www.oracle.com/technetwork/server-storage/sun-unified-storage/downloads/zfssa-plugins-1489830.html



    ESBU utility performs the entire Configuration process in three steps.

    • Oracle ZFS Storage Appliance configuration
    • Engineered System nodes configuration
    • RMAN run block script generation



    In this article I will demonstrate how to Configure ZFS shares on Exadata Database machines using ESBU utility. 


    Note: The ZFS Storage Appliance is Installed and Configured by Storage Administrator or Oracle ACS. 


    Environment
    Exadata Half Rack X6-2


    Steps to Configure ZFS shares on Exadata Database Machine

    • Create a staging area on Exadata compute node 1 and copy the ESBU utility to node 1 using Winscp

    • Unzip ESBU utility

    • Ensure that the resolv.conf contains the domain name to reach the ZFS controllers.

    • Install and Configure SSH Keys

    • Ensure you can connect to ZFS without password.

    • On Exadata Compute nodes, ensure that “rcpbind” and nfslock” Linux services are enabled and running.

    • Navigate to staging area and execute the ESBU utility

    Enter 0 for Exadata

     Enter y to continue

    Verify the ZFS appliance name, enter root for user name, enter project name (example: dm01). Enter y to continue

    Enter the IP for accessing pool (Example: 192.168.1.200), enter 0 and hit enter

    Enter y to select another pool. Enter y to continue.

    Enter IP to access second pool, verify details, enter y to continue

    At this point the ESBU completed the step 1 of configure ZFS pools. Enter y to continue with step 2.

    Enter y to continue

    Enter DB name, hostname of node 1, hit return to accept default user and group. Enter y to continue.

    Enter y to continue

    Enter y to confirm Oracle Home. Enter y to continue

    Enter y to continue

    Enter y if you have more than one network interface

    Specify the pair of private IPs on Exadata compute node 1

    Specify the pair of private IPs on Exadata compute node 2

    Specify the pair of private IPs on Exadata compute node 3

    Specify the pair of private IPs on Exadata compute node 4

    Enter y to continue. This will bounce your dm01 database

    At this point ESBU completed step 2. Enter y to continue to step 3.

    Enter y to continue

    This completes the ESBU configuration for Exadata Database machine.

    • Verify the ZFS shares are mounted on all Exadata compute nodes. 
    Conclusion
    In this article we have learnt about
    ESBU utility and how use it to configure the ZFS shares on Exadata Database
    Machines. ESBU utility is a fully automated tool for configuring ZFS shares on Exadata. It replaces the manual steps that were required to
    configure ZFS Storage Appliance and enforces the best practices and
    configuration.

  • Convert Standard ASM to Flex ASM on Exadata

    About Flex ASM

    • In earlier Oracle database release, ASM will be running on each node in the cluster and ASM clients such as database and ACFS can only access ASM by using an ASM on the same host.
    • Oracle Database 12c introduced a new feature called Oracle Flex ASM which provides remote storage access.
    • Oracle Flex ASM is an ASM instance which will be running in the cluster based on cardinality defined.
    • The default cardinality is 3, but this can be modified using srvctl modify asm command.
    • ASM is not required to run on each node in the cluster.
    • With Oracle Flex ASM,the clients can connect to remote ASM using network connection (ie ASM network ).
    • If a server running an ASM instance fails, Oracle Clusterware will start a new ASM instance on a different server to maintain the cardinality. If a 12c database instance is using a particular ASM instance, and that instance is lost because of a server crash or ASM instance failure, then the Oracle 12c database instance will reconnect to an existing ASM instance on another node.These features are collectively called Oracle Flex ASM.



    Steps to Convert Standard ASM to Flex ASM

    • You can convert an Oracle ASM configuration to an Oracle Flex ASM using ASMCA. This functionality is only available in an Oracle Grid Infrastructure 12c configuration.

    Before you convert an Oracle ASM configuration to an Oracle Flex ASM, you must ensure the following:

    • Oracle Cluster Registry (OCR) is stored in a disk group

    [root@dm01db01 ~]# /u01/app/12.1.0.2/grid/bin/ocrcheck
    Status of Oracle Cluster Registry is as follows :
             Version                  :          4
             Total space (kbytes)     :     409568
             Used space (kbytes)      :      10684
             Available space (kbytes) :     398884
             ID                       : 1119121028
             Device/File Name         :      +DATA
                                        Device/File integrity check succeeded


                                        Device/File not configured


                                        Device/File not configured


                                        Device/File not configured


                                        Device/File not configured


             Cluster registry integrity check succeeded


             Logical corruption check succeeded

    • The server parameter file (SPFILE) is stored in a disk group.

    SQL> show parameter spfile


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    spfile                               string      +DATA/dm01-cluster/ASMPARAMETE
                                                     RFILE/registry.253.934096549

    SQL> show parameter spfile


    NAME                                 TYPE        VALUE
    ———————————— ———– ——————————
    spfile                               string      +DATA/orcl/spfileorcl.ora

    • The password file (ORAPWD file) is stored in a disk group.

    dm01db01-orcldb1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘ls -ltr
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl1
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl2
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl3
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl4
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl5
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl7
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl6
    dm01db01: -rw-r—– 1 oracle oinstall 8192 Jan 17 03:45 /u01/app/oracle/product/12.1.0.2/dbhome/dbs/orapworcl8


    dm01db01-orcl1 {/home/oracle}:orapwd file=’+DATA/orcl/orapworcl’ entries=10 dbuniquename=’orcl’ format=12


    Enter password for SYS:


    ASMCMD> ls -l +DATA/ORCL/orapworcl
    Type      Redund  Striped  Time             Sys  Name
    PASSWORD  HIGH    COARSE   JAN 26 06:00:00  N    orapworcl => +DATA/ORCL/PASSWORD/pwdorcl.272.934265865


    dm01db01-+ASM1 {/home/oracle}:orapwd file=’+DATA/orapwASM’ entries=10 dbuniquename=’ASM’ format=12


    Enter password for SYS:


    OPW-00019: Failed to update the CRS resource with DB password file location


    dm01db01-+ASM1 {/home/oracle}:srvctl modify asm -pwfile +DATA/orapwASM


    dm01db01-+ASM1 {/home/oracle}:srvctl config asm -detail
    ASM home: <CRS home>
    Password file: +DATA/orapwASM
    ASM listener: LISTENER
    ASM is enabled.
    ASM is individually enabled on nodes:
    ASM is individually disabled on nodes:

    ASMCMD [+DATA] > ls -l orapwasm
    Type      Redund  Striped  Time             Sys  Name
    PASSWORD  HIGH    COARSE   JAN 27 04:00:00  N    orapwasm => +DATA/ASM/PASSWORD/pwdasm.399.934344779

    • The disk group where the OCR, SPFILE, and ORAPWD files are stored has Oracle ASM compatibility (COMPATIBLE.ASM) set to 12.1 or higher.

    SQL> col name for a40
    SQL> col value for a40
    SQL> set lines 200
    SQL> set pages 200
    SQL> select a.group_number,b.name dgname,a.name,a.value from v$asm_attribute a, v$asm_diskgroup b where a.name in (‘au_size’,’disk_repair_time’,’compatible.rdbms’,’compatible.asm’) and a.group_number=b.group_number order by b.name,a.name;


    GROUP_NUMBER DGNAME                         NAME                                     VALUE
    ———— —————————— ————————–               ————————
               3 ACFS_DG                        au_size                                  4194304
               3 ACFS_DG                        compatible.asm                           12.1.0.0.0
               3 ACFS_DG                        compatible.rdbms                         12.1.0.0.0
               3 ACFS_DG                        disk_repair_time                         3.6h
               1 DATA                           au_size                                  4194304
               1 DATA                           compatible.asm                           12.1.0.2.0
               1 DATA                           compatible.rdbms                         11.2.0.4.0
               1 DATA                           disk_repair_time                         3.6h
               2 RECO                           au_size                                  4194304
               2 RECO                           compatible.asm                           12.1.0.2.0
               2 RECO                           compatible.rdbms                         11.2.0.4.0
               2 RECO                           disk_repair_time                         3.6h


    dm01db01-+ASM1 {/home/oracle}:/u01/app/12.1.0.2/grid/bin/crsctl stat res -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.ACFS_DG.ACFSVOL.advm
                   ONLINE  ONLINE       dm01db01                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
                   ONLINE  ONLINE       dm01db02                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
                   ONLINE  ONLINE       dm01db03                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
                   ONLINE  ONLINE       dm01db04                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
                   ONLINE  ONLINE       dm01db05                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
                   ONLINE  ONLINE       dm01db06                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
                   ONLINE  ONLINE       dm01db07                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
                   ONLINE  ONLINE       dm01db08                 Volume device /dev/a
                                                                 sm/acfsvol-228 is online,STABLE
    ora.ACFS_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.DATA.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.LISTENER.lsnr
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.RECO.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.acfs_dg.acfsvol.acfs
                   ONLINE  ONLINE       dm01db01                 mounted on /acfs,STABLE
                   ONLINE  ONLINE       dm01db02                 mounted on /acfs,STABLE
                   ONLINE  ONLINE       dm01db03                 mounted on /acfs,STABLE
                   ONLINE  ONLINE       dm01db04                 mounted on /acfs,STABLE
                   ONLINE  ONLINE       dm01db05                 mounted on /acfs,STABLE
                   ONLINE  ONLINE       dm01db06                 mounted on /acfs,STABLE
                   ONLINE  ONLINE       dm01db07                 mounted on /acfs,STABLE
                   ONLINE  ONLINE       dm01db08                 mounted on /acfs,STABLE
    ora.asm
                   ONLINE  ONLINE       dm01db01                 Started,STABLE
                   ONLINE  ONLINE       dm01db02                 Started,STABLE
                   ONLINE  ONLINE       dm01db03                 Started,STABLE
                   ONLINE  ONLINE       dm01db04                 Started,STABLE
                   ONLINE  ONLINE       dm01db05                 Started,STABLE
                   ONLINE  ONLINE       dm01db06                 Started,STABLE
                   ONLINE  ONLINE       dm01db07                 Started,STABLE
                   ONLINE  ONLINE       dm01db08                 Started,STABLE
    ora.net1.network
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ora.ons
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.LISTENER_SCAN1.lsnr
          1        ONLINE  ONLINE       dm01db08                 STABLE
    ora.LISTENER_SCAN2.lsnr
          1        ONLINE  ONLINE       dm01db07                 STABLE
    ora.LISTENER_SCAN3.lsnr
          1        ONLINE  ONLINE       dm01db05                 STABLE
    ora.dm01db01.vip
          1        ONLINE  ONLINE       dm01db01                 STABLE
    ora.dm01db02.vip
          1        ONLINE  ONLINE       dm01db02                 STABLE
    ora.dm01db03.vip
          1        ONLINE  ONLINE       dm01db03                 STABLE
    ora.dm01db04.vip
          1        ONLINE  ONLINE       dm01db04                 STABLE
    ora.dm01db05.vip
          1        ONLINE  ONLINE       dm01db05                 STABLE
    ora.dm01db06.vip
          1        ONLINE  ONLINE       dm01db06                 STABLE
    ora.dm01db07.vip
          1        ONLINE  ONLINE       dm01db07                 STABLE
    ora.dm01db08.vip
          1        ONLINE  ONLINE       dm01db08                 STABLE
    ora.cvu
          1        ONLINE  ONLINE       dm01db06                 STABLE
    ora.oc4j
          1        OFFLINE OFFLINE                               STABLE
    ora.orcl.db
          1        ONLINE  ONLINE       dm01db03                 Open,STABLE
          2        ONLINE  ONLINE       dm01db04                 Open,STABLE
          3        ONLINE  ONLINE       dm01db01                 Open,STABLE
          4        ONLINE  ONLINE       dm01db05                 Open,STABLE
          5        ONLINE  ONLINE       dm01db06                 Open,STABLE
          6        ONLINE  ONLINE       dm01db02                 Open,STABLE
          7        ONLINE  ONLINE       dm01db07                 Open,STABLE
          8        ONLINE  ONLINE       dm01db08                 Open,STABLE
    ora.scan1.vip
          1        ONLINE  ONLINE       dm01db08                 STABLE
    ora.scan2.vip
          1        ONLINE  ONLINE       dm01db07                 STABLE
    ora.scan3.vip
          1        ONLINE  ONLINE       dm01db05                 STABLE




    ASMCMD> showclustermode
    ASM cluster : Flex mode disabled


    dm01db01-+ASM1 {/home/oracle}:srvctl status asm -detail
    ASM is running on dm01db01,dm01db03,dm01db02,dm01db05,dm01db04,dm01db07,dm01db06,dm01db08
    ASM is enabled.


    Conversion Steps

    • Set the display and Start ASMCA



    • Click on “ASM Instances” and then “Convert to Oracle Flex ASM”



    • On Convert to Oracle Flex ASM page, specify:
      • ASM Listener Port number: 1521
      • Select network to be used for ASM: bond0 (cluster_interconnect)
      • Check the box if you plan on using ASM for pre 12c databases.
      • Click Ok



    • Click Yes



    • The processing page will be displayed.





    • Open the script and view the script contents.

    [root@dm01db01 ~]# cat /u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh  | more
    #!/bin/sh


    cd /u01/app/oracle/cfgtoollogs/asmca/scripts
    su oracle -c “/u01/app/12.1.0.2/grid/bin/srvctl add asm -proxy”
    if [ “$?” != “0” ]; then
            echo “Error adding ASM Proxy”
            exit 1
    fi
    /u01/app/12.1.0.2/grid/srvm/admin/srvmhelper -updateDepASM
    if [ “$?” != “0” ]; then
            echo “Error updating ASM resource dependencies”
            exit 1
    fi
    for node in dm01db01 dm01db02 dm01db03 dm01db04 dm01db05 dm01db06 dm01db07 dm01db08
    do
            /u01/app/12.1.0.2/grid/bin/crsctl stop cluster -n $node
            if [ “$?” != “0” ]; then
                    echo “Error stopping Oracle Grid Infrastructure in node $node”
                    exit 1
            fi
            /u01/app/12.1.0.2/grid/bin/crsctl start cluster -n $node


            stat=1
            cmd=”/u01/app/12.1.0.2/grid/bin/crsctl status server $node”
            while [ $stat = 1 ]
            do
                    sleep 5
                    op=`$cmd`
                    op=`echo $op | /bin/cut -d’ ‘ -f2  | /bin/cut -d’=’ -f2`
                    if [ “$op” = “ONLINE” ]
                    then
                            stat=0
                            echo “Oracle Grid Infrastructure restarted in node $node”
                    fi
            done
            if [ “$node” = dm01db01 ]; then
                    for lsnr in ASMNET1LSNR_ASM
                    do
                            /u01/app/12.1.0.2/grid/bin/srvctl start listener -listener $lsnr
                            lsnrret=$?
                            if [ “$lsnrret” = “2” ]; then
                                    echo “ASM listener $lsnr running already”
                            elif [ “$lsnrret” != “0” ]; then
                                    echo “Error starting ASM listener $lsnr”
                                    exit 1
                            fi
                    done
            fi
    done


    • Execute the script on node 1 as root user as follows:

    dm01db01-+ASM1 {/dev/asm}:df -k | grep asm
    /dev/asm/acfsvol-228


    Stop the ACFS file system before executing the script.


    dm01db01-+ASM1 {/dev/asm}:srvctl status filesystem -device /dev/asm/acfsvol-228
    ACFS file system /acfs is mounted on nodes


    dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08


    dm01db01-+ASM1 {/home/oracle}:srvctl stop filesystem -d /dev/asm/acfsvol-228 -f


    dm01db01-+ASM1 {/home/oracle}:srvctl status filesystem -d /dev/asm/acfsvol-228
    ACFS file system /acfs is not mounted


    dm01db01-+ASM1 {/dev/asm}:df -k | grep asm


    Finally execute the script


    [root@dm01db01 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
    CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db01’
    CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.orcldb.db’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ttsrv_dm01db01’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.acfs_dg.acfsvol.acfs’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.RECO.dg’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.orcl.db’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.ACFS_DG.dg’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.ACFS_DG.dg’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.RECO.dg’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.DATA.dg’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.dm01db01.vip’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.dm01db01.vip’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.dm01db01.vip’ on ‘dm01db05’
    CRS-2677: Stop of ‘ora.orcldb.db’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.orcl.db’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.dm01db01.vip’ on ‘dm01db05’ succeeded
    CRS-2673: Attempting to stop ‘ora.ons’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.ons’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.net1.network’ on ‘dm01db01’ succeeded
    CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dm01db01’ has completed
    CRS-2677: Stop of ‘ora.crsd’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dm01db01’
    CRS-2673: Attempting to stop ‘ora.storage’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.storage’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.asm’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.ctssd’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.evmd’ on ‘dm01db01’ succeeded
    CRS-2677: Stop of ‘ora.asm’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.cssd’ on ‘dm01db01’ succeeded
    CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘dm01db01’
    CRS-2677: Stop of ‘ora.diskmon’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.evmd’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.cssdmonitor’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.cssd’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.diskmon’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.evmd’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.diskmon’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.cssd’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.ctssd’ on ‘dm01db01’
    CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.ctssd’ on ‘dm01db01’ succeeded
    CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.asm’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.asm’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.storage’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.storage’ on ‘dm01db01’ succeeded
    CRS-2672: Attempting to start ‘ora.crsd’ on ‘dm01db01’
    CRS-2676: Start of ‘ora.crsd’ on ‘dm01db01’ succeeded
    Oracle Grid Infrastructure restarted in node dm01db01
    CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dm01db02’
    CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dm01db02’
    CRS-2673: Attempting to stop ‘ora.acfs_dg.acfsvol.acfs’ on ‘dm01db02’
    CRS-2673: Attempting to stop ‘ora.orcl.db’ on ‘dm01db02’
    CRS-2673: Attempting to stop ‘ora.orcldb.db’ on ‘dm01db02’
    CRS-2673: Attempting to stop ‘ora.RECO.dg’ on ‘dm01db02’
    CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘dm01db02’
    CRS-2677: Stop of ‘ora.RECO.dg’ on ‘dm01db02’ succeeded
    CRS-2677: Stop of ‘ora.DATA.dg’ on ‘dm01db02’ succeeded
    CRS-2677: Stop of ‘ora.orcl.db’ on ‘dm01db02’ succeeded
    CRS-2677: Stop of ‘ora.orcldb.db’ on ‘dm01db02’ succeeded
    …
    …


    [root@dm01db01 ~]#

    • Go back to ASMCA and click close.

    Verify the cluster mode now.


    ASMCMD> showclustermode
    ASM cluster : Flex mode enabled

    Conclusion
    In this article we have learned about Flex ASM in Oracle database 12c and how to convert a Standard ASM to Flex ASM to take advantage of 12c new feature.