Netsoftmate

Tag: Create disk group

  • Configure ACFS on Exadata Database Machine

    About ACFS:

    • ACFS stands for Oracle Automatic Storage Management Cluster File System
    • ACFS extends ASM functionality to support all kind of files like database files, application files and general purpose files.
    • Oracle ACFS is a High Availability Cluster File system and it is tightly coupled with Oracle Clusterware technology.
    • It can manage large number of file systems and files.
    • One can manage ACFS/ADVM using: ASMCA, ASMCMD, SQL*PLUS, OEM and ACFS Plug-ins

    In this article I will demostrate how to configure Oracle ACFS on Exadata Database machine. Before start with the actual configuration, let’s discuss the few important points about ACFS and configure it on Exadata.


    Oracle Automatic Storage Management Cluster File System (ACFS) on Exadata Database Machine:


    Starting with Oracle Grid Infrastructure version 12.1.0.2, Oracle ACFS supports all database files and general purpose files on Oracle Exadata Database Machine running Oracle Linux on database servers.


    The following database versions are supported by Oracle ACFS on Exadata Database Machine:

    • Oracle Database 10g Rel. 2 (10.2.0.4 and 10.2.0.5)
    • Oracle Database 11g (11.2.0.4 and higher)
    • Oracle Database 12c (12.1.0.1 and higher)



    Some important information about using Oracle ACFS with Exadata


    • Oracle ACFS supports all database file types and general purpose files
    • All Oracle ACFS advanced functionality is supported on Exadata including:
      • Snapshots, replication, tagging, security, encryption, audit and HANFS.



    Patches required

    • Please refer to the MOS note for important fixed required for Oracle ACFS on Exadata Database Machine.
      • 2022172.1



    Oracle ACFS restriction

    • Smart Scan, Storage Indexes, IO Resource Manager (IORM) are currently not supported.
    • Smart FC read operation are cached.
    • Smart FC write operations is not supported
    • Smart Flash Logging is not supported.
    • HCC is supported with fix for bug 19136936
    • Hardware Assisted Resilient Data (HARD) checks are not performed.
    • Oracle ACFS replication or security/encryption/audit is only supported with general purpose files.
    • Database files in ACFS doesn’t support replication or security/encryption/audit



    Performance Optimization

    • When storing database data files on Oracle ACFS, Initialization parameter FILESYSTEMIO_OPTIONS must be set to ‘setall’. Other settings are not supported.
    • For each disk group that contains an Oracle ADVM volume that is intended to hold data files, set ASM and ADVM compatibility attributes to 12.1 or higher as this will ensure optimized performance.

    Uses of ACFS on Exadata Database Machine

    • Since Oracle ACFS support database version 10.2.0.4 and 10.2.0.5, Customers can migrate their 10g database to Exadata. Later they upgrade their 10g database to 11g or higher database version.
    • Oracle ACFS can be used as a shared Oracle Home for RDBMS, Middleware, GoldenGate binaries.
    • Oracle ACFS can be used to store Oracle GoldenGate related files such as trail file, checkpoint files, parameter files and so on.



    Steps to configure ACFS


    It consists of 4 steps:

    1. Load ACFS/ADVM module on all the nodes
    2. Create ASM diskgroup for ACFS
    3. Create ASM Volume
    4. Create ASM Cluster File system



    Current Environment Details:

    • Exadata Database Machine X5-2 Full Rack
    • Exadata Storage Software version 12.1.2.1.1
    • Oracle Grid/Database Version 12.1.0.2



    I – Load ACFS/ADVM module on all the nodes

    • Login as Oracle Grid Infrastructure software owner and start asmca.

    uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)

    • Set the ORACLE HOME and SID to ASM

    dm01db01-orcldb1 {/home/oracle}:. oraenv
    ORACLE_SID = [orcldb1] ? +ASM1
    The Oracle base remains unchanged with value /u01/app/oracle


    dm01db01-+ASM1 {/home/oracle}:env | grep ORA
    ORACLE_SID=+ASM1
    ORACLE_BASE=/u01/app/oracle
    ORACLE_HOME=/u01/app/12.1.0.2/grid

    • Set DISPLAY

    dm01db01-+ASM1 {/home/oracle}:export DISPLAY=10.30.201.137:0.0

    • Start asmca utility

    dm01db01-+ASM1 {/home/oracle}:which asmca
    /u01/app/12.1.0.2/grid/bin/asmca
    dm01db01-+ASM1 {/home/oracle}:asmca





    • Make sure ACFS/ADVM modules loaded on every node in a standard cluster.



    Here ACFS/ADVM module is not load on node 1.
    Execute the following command to load acfs module.


    [root@dm01db01 ~]# /u01/app/12.1.0.2/grid/bin/acfsload -h
    ACFS-9228: usage: acfsload [-h] {start|stop} [-s]


    [root@dm01db01 ~]# /u01/app/12.1.0.2/grid/bin/acfsload start
    ACFS-9391: Checking for existing ADVM/ACFS installation.
    ACFS-9392: Validating ADVM/ACFS installation files for operating system.
    ACFS-9393: Verifying ASM Administrator setup.
    ACFS-9308: Loading installed ADVM/ACFS drivers.
    ACFS-9327: Verifying ADVM/ACFS devices.
    ACFS-9156: Detecting control device ‘/dev/asm/.asm_ctl_spec’.
    ACFS-9156: Detecting control device ‘/dev/ofsctl’.
    ACFS-9322: completed

    • Verify ACFS/ADVM module is loaded.

    [root@dm01db01 ~]# lsmod | grep oracle
    oracleacfs           3310765  0
    oracleadvm            506070  0
    oracleoks             508392  2 oracleacfs,oracleadvm


    On all nodes:


    [root@dm01db01 ~]# dcli -g dbs_group -l root ‘lsmod | grep oracle’
    dm01db01: oracleacfs           3310765  0
    dm01db01: oracleadvm            506070  0
    dm01db01: oracleoks             508392  2 oracleacfs,oracleadvm
    dm01db02: oracleacfs           3310765  0
    dm01db02: oracleadvm            506070  0
    dm01db02: oracleoks             508392  2 oracleacfs,oracleadvm
    dm01db03: oracleacfs           3310765  0
    dm01db03: oracleadvm            506070  0
    dm01db03: oracleoks             508392  2 oracleacfs,oracleadvm
    dm01db04: oracleacfs           3310765  0
    dm01db04: oracleadvm            506070  0
    dm01db04: oracleoks             508392  2 oracleacfs,oracleadvm
    dm01db05: oracleacfs           3310765  0
    dm01db05: oracleadvm            506070  0
    dm01db05: oracleoks             508392  2 oracleacfs,oracleadvm
    dm01db06: oracleacfs           3310765  0
    dm01db06: oracleadvm            506070  0
    dm01db06: oracleoks             508392  2 oracleacfs,oracleadvm
    dm01db07: oracleacfs           3310765  0
    dm01db07: oracleadvm            506070  0
    dm01db07: oracleoks             508392  2 oracleacfs,oracleadvm
    dm01db08: oracleacfs           3310765  0
    dm01db08: oracleadvm            506070  0
    dm01db08: oracleoks             508392  2 oracleacfs,oracleadvm


    [root@dm01db01 ~]# vi acfs_validate_module.sh


    [root@dm01db01 ~]# cat acfs_validate_module.sh
    #################################################
    ###
    ### ACFS/ADVM modules validation
    ###
    #################################################
    ###
    ### Author: Esteban D Bernal
    ###
    #################################################


    /u01/app/12.1.0.2/grid/bin/acfsdriverstate loaded
    /u01/app/12.1.0.2/grid/bin/acfsdriverstate installed
    /u01/app/12.1.0.2/grid/bin/acfsdriverstate supported
    /u01/app/12.1.0.2/grid/bin/acfsdriverstate version
    /u01/app/12.1.0.2/grid/bin/acfsroot version_check


    lsmod | grep oracle


    ## Where: “/u01/app/12.1.0.2/grid” is the Grid Infrastructure Home directory,
    ## thus you need to replace this value with your Grid Infrastructure Home directory path.


    [root@dm01db01 ~]# chmod +x acfs_validate_module.sh


    [root@dm01db01 ~]# ls -l acfs_validate_module.sh
    -rwxr-xr-x 1 root root 673 Jan 25 05:02 acfs_validate_module.sh


    [root@dm01db01 ~]# sh -x acfs_validate_module.sh
    + /u01/app/12.1.0.2/grid/bin/acfsdriverstate loaded
    ACFS-9203: true
    + /u01/app/12.1.0.2/grid/bin/acfsdriverstate installed
    ACFS-9203: true
    + /u01/app/12.1.0.2/grid/bin/acfsdriverstate supported
    ACFS-9200: Supported
    + /u01/app/12.1.0.2/grid/bin/acfsdriverstate version
    ACFS-9325:     Driver OS kernel version = 2.6.39-400.3.0.el6uek.x86_64(x86_64).
    ACFS-9326:     Driver Oracle version = 140611.5.
    + /u01/app/12.1.0.2/grid/bin/acfsroot version_check
    ACFS-9316: Valid ADVM/ACFS distribution media detected at: ‘/u01/app/12.1.0.2/grid/usm/install/Oracle/EL6UEK/x86_64/2.6.39-400/2.6.39-400-x86_64/bin’
    + lsmod
    + grep oracle
    oracleacfs           3310765  0
    oracleadvm            506070  0
    oracleoks             508392  2 oracleacfs,oracleadvm


    • Refresh or Exit and start ASMCA again.



    I still see that ACFS/ADVM is “Not Installed” on node 1 though we have enabled/loaded it manually on node 1 and verified. Let’s move on and see if it is just a BUG. If there is no issue we should be able to configure ASM Volume and ACFS without any issues.


    II – Create ASM diskgroup for ACFS

    • On Disk Groups page click on “Create”



    • On Create Disk Group page, specify:
      • Disk Group Name: ACFS_DG
      • Redundancy: Normal
      • Select Member Disk: Show Eligible
      • Select all candidate disks to be part of ACFS_DG disk group
      • In Exadata each Storage cell is a failure Group
      • Click on Show Advanced Options, specify:
      • Allocation Unit Size: 4 (MB)
      • Minimun software version for ASM/Database/ADVM: 12.1.0.0.0
      • Click Ok



    • Click Ok



    • We can see our ACFS_DG Disk Group created



    • Verify newly created ACFS_DG Disk Group as follows:

    SQL> select inst_id, name, total_mb, group_number from gv$asm_diskgroup where name like ‘ACFS_DG’;


       INST_ID NAME                             TOTAL_MB GROUP_NUMBER
    ———- —————————— ———- ————
             1 ACFS_DG                           4145536            3
             3 ACFS_DG                           4145536            3
             6 ACFS_DG                           4145536            3
             5 ACFS_DG                           4145536            3
             7 ACFS_DG                           4145536            3
             4 ACFS_DG                           4145536            3
             2 ACFS_DG                           4145536            3
             8 ACFS_DG                           4145536            3


    8 rows selected.


    SQL> col value for a30
    SQL> col name for a30
    SQL> select name, value from v$asm_attribute where GROUP_NUMBER=3 and name like ‘compatible%’;


    NAME                           VALUE
    —————————— ——————————
    compatible.asm                 12.1.0.0.0
    compatible.rdbms               12.1.0.0.0
    compatible.advm                12.1.0.0.0


    dm01db01-+ASM1 {/home/oracle}:/u01/app/12.1.0.2/grid/bin/crsctl stat res ora.ACFS_DG.dg -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.ACFS_DG.dg
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ——————————————————————————–


    III – Create ASM Volume

    • Click on “Volumes” tab



    • On Volumes page click on “Create”
      • On Create Volume page, specify:
      • Volumn name: acfsvol
      • Disk Group name: ACFS_DG
      • Size: 2008.25 (it can be anything based on your requirement from usable space avaialble.)
      • Click Ok




    • Click Ok



    • We can see our Volume created



    • Verify newly created Volume as follows:

    ASMCMD> volinfo –all
    Diskgroup Name: ACFS_DG


             Volume Name: ACFSVOL
             Volume Device: /dev/asm/acfsvol-228
             State: ENABLED
             Size (MB): 2056448
             Resize Unit (MB): 64
             Redundancy: MIRROR
             Stripe Columns: 8
             Stripe Width (K): 1024
             Usage:
             Mountpath:


    dm01db01-+ASM1 {/home/oracle}:/u01/app/12.1.0.2/grid/bin/crsctl stat res ora.ACFS_DG.ACFSVOL.advm -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.ACFS_DG.ACFSVOL.advm
                   ONLINE  ONLINE       dm01db01                 STABLE
                   ONLINE  ONLINE       dm01db02                 STABLE
                   ONLINE  ONLINE       dm01db03                 STABLE
                   ONLINE  ONLINE       dm01db04                 STABLE
                   ONLINE  ONLINE       dm01db05                 STABLE
                   ONLINE  ONLINE       dm01db06                 STABLE
                   ONLINE  ONLINE       dm01db07                 STABLE
                   ONLINE  ONLINE       dm01db08                 STABLE
    ——————————————————————————–


    dm01db01-+ASM1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘ls -l /dev/asm/*’
    dm01db01: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228
    dm01db02: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228
    dm01db03: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228
    dm01db04: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228
    dm01db05: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228
    dm01db06: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228
    dm01db07: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228
    dm01db08: brwxrwx— 1 root dba 251, 116737 Jan 25 06:14 /dev/asm/acfsvol-228


    IV – Create ASM Cluster File system

    • Create ACFS mount point directory on all nodes

    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ‘mkdir /acfs’


    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ‘chown oracle:oinstall /acfs’


    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ‘chmod 775 /acfs’


    [root@dm01db01 ~]# dcli -g ~/dbs_group -l root ‘ls -ld /acfs’
    dm01db01: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs
    dm01db02: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs
    dm01db03: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs
    dm01db04: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs
    dm01db05: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs
    dm01db06: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs
    dm01db07: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs
    dm01db08: drwxrwxr-x 2 oracle oinstall 4096 Jan 25 06:28 /acfs


    • Click on “ASM Cluster File Systems” tab



    • On ASM Cluster File Systems page click on “Create”



    • On Create ASM Cluster File System page, specify:
      • Type of ACFS: Cluster File System
      • Mount point: /acfs
      • Auto Mount: check
      • User Name: oracle
      • Group Name: oinstall
      • Description: ACFS File System
      • Select Volume: ACFSVOL – /dev/asm/acfsvol-228 – 2008.25
      • Click Ok



    • Run the ACFS script on Node 1 only to register ACFS with Grid Infrastructure and to mount the ACFS file system.

    • Contents of ACFS script looks like this:

    [root@dm01db01 ~]# cat /u01/app/oracle/cfgtoollogs/asmca/scripts/acfs_script.sh
    #!/bin/sh


    /u01/app/12.1.0.2/grid/bin/srvctl add filesystem -d /dev/asm/acfsvol-228 -m /acfs -u oracle -fstype ACFS -description ‘ACFS File System’ -autostart ALWAYS
    if [ $? = “0” -o $? = “2” ]; then
       /u01/app/12.1.0.2/grid/bin/srvctl start filesystem -d /dev/asm/acfsvol-228
       if [ $? = “0” ]; then
          chown oracle:oinstall /acfs
          chmod 775 /acfs
          /u01/app/12.1.0.2/grid/bin/srvctl status filesystem -d /dev/asm/acfsvol-228
          exit 0
       fi
       /u01/app/12.1.0.2/grid/bin/srvctl status filesystem -d /dev/asm/acfsvol-228
    fi


    • Execute the script

    [root@dm01db01 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/acfs_script.sh
    ACFS file system /acfs is mounted on nodes dm01db01,dm01db02,dm01db03,dm01db04,dm01db05,dm01db06,dm01db07,dm01db08


    • Click Close



    • We can see our Volume created



    • Verify newly created Volume as follows:

    dm01db01-+ASM1 {/home/oracle}:/u01/app/12.1.0.2/grid/bin/crsctl stat res ora.acfs_dg.acfsvol.acfs -t
    ——————————————————————————–
    Name           Target  State        Server                   State details
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.acfs_dg.acfsvol.acfs
                   ONLINE  ONLINE       dm01db01                 mounted on /acfs,STA
                                                                 BLE
                   ONLINE  ONLINE       dm01db02                 mounted on /acfs,STA
                                                                 BLE
                   ONLINE  ONLINE       dm01db03                 mounted on /acfs,STA
                                                                 BLE
                   ONLINE  ONLINE       dm01db04                 mounted on /acfs,STA
                                                                 BLE
                   ONLINE  ONLINE       dm01db05                 mounted on /acfs,STA
                                                                 BLE
                   ONLINE  ONLINE       dm01db06                 mounted on /acfs,STA
                                                                 BLE
                   ONLINE  ONLINE       dm01db07                 mounted on /acfs,STA
                                                                 BLE
                   ONLINE  ONLINE       dm01db08                 mounted on /acfs,STA
                                                                 BLE
    ——————————————————————————–

    V – Test ACFS Cluster file system


    dm01db01-+ASM1 {/home/oracle}:cd /acfs


    dm01db01-+ASM1 {/acfs}:ls -ltr
    total 64
    drwx—— 2 root root 65536 Jan 25 07:11 lost+found


    dm01db01-+ASM1 {/acfs}:mkdir test


    dm01db01-+ASM1 {/acfs}:cd test/


    dm01db01-+ASM1 {/acfs/test}:vi myfile.txt


    dm01db01-+ASM1 {/acfs/test}:cat myfile.txt
    this is a test


    dm01db01-+ASM1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘cat /acfs/test/myfile.txt’
    dm01db01: this is a test
    dm01db02: this is a test
    dm01db03: this is a test
    dm01db04: this is a test
    dm01db05: this is a test
    dm01db06: this is a test
    dm01db07: this is a test
    dm01db08: this is a test

    dm01db01-+ASM1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘df -h /acfs’
    dm01db01: Filesystem            Size  Used Avail Use% Mounted on
    dm01db01: /dev/asm/acfsvol-228
    dm01db01: 2.0T  4.3G  2.0T   1% /acfs
    dm01db02: Filesystem            Size  Used Avail Use% Mounted on
    dm01db02: /dev/asm/acfsvol-228
    dm01db02: 2.0T  4.3G  2.0T   1% /acfs
    dm01db03: Filesystem            Size  Used Avail Use% Mounted on
    dm01db03: /dev/asm/acfsvol-228
    dm01db03: 2.0T  4.3G  2.0T   1% /acfs
    dm01db04: Filesystem            Size  Used Avail Use% Mounted on
    dm01db04: /dev/asm/acfsvol-228
    dm01db04: 2.0T  4.3G  2.0T   1% /acfs
    dm01db05: Filesystem            Size  Used Avail Use% Mounted on
    dm01db05: /dev/asm/acfsvol-228
    dm01db05: 2.0T  4.3G  2.0T   1% /acfs
    dm01db06: Filesystem            Size  Used Avail Use% Mounted on
    dm01db06: /dev/asm/acfsvol-228
    dm01db06: 2.0T  4.3G  2.0T   1% /acfs
    dm01db07: Filesystem            Size  Used Avail Use% Mounted on
    dm01db07: /dev/asm/acfsvol-228
    dm01db07: 2.0T  4.3G  2.0T   1% /acfs
    dm01db08: Filesystem            Size  Used Avail Use% Mounted on
    dm01db08: /dev/asm/acfsvol-228
    dm01db08: 2.0T  4.3G  2.0T   1% /acfs


    dm01db01-+ASM1 {/home/oracle}:dcli -g ~/dbs_group -l oracle ‘ls -ld /acfs’
    dm01db01: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs
    dm01db02: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs
    dm01db03: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs
    dm01db04: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs
    dm01db05: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs
    dm01db06: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs
    dm01db07: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs
    dm01db08: drwxrwxr-x 4 oracle oinstall 4096 Jan 25 07:11 /acfs




    Conclusion
    In this article we have learned basic concepts about ACFS and it works on Exadata. We also seen a practicle demonstration on how to Load ACFS/ADVM module on all the nodes, Create ASM diskgroup for ACFS, Create ASM Volume and Create ASM Cluster File system.