Overview
You purchased a new Storage cell for expansion or removed Storage from an
existing Cluster and want to add it to the new a Cluster due to space demand.
In this article I will demonstrate how to add a storage cell to an
existing Exadata Database Machine.
Here we are
adding new storage cell with 600 GB disks to existing Exadata database machine using
600 GB disks.
- Exadata Database Machine X2-2 Full Rack
- Exadata Storage Software version 12.1.2.2.0
- Oracle Grid/Database Version 11.2.0.4
Pre
Checks:
- Create
a new file named “new_cell_group” containing only the hostnames for
the new cells.
new_cell_group
new_cell_group
- It is assumed that the
file “cell_group” only contains the original cells, not the new ones.
If you have already modified “cell_group” to contain the new
cells, just comment it out or remove it from the file.
- Here
we use the root user to run the cellcli commands; ensure that user equivalency setup
for root user between Storage cells and Compute nodes.
~]# dcli -g dbs_group -l root ‘uptime’
09:55:05 up 425 days, 23:16, 1
user, load average: 3.26, 3.37, 3.39
09:55:05 up 117 days, 18:20, 0
users, load average: 1.19, 1.42, 1.56
09:55:05 up 85 days, 10:07, 0
users, load average: 6.25, 6.20, 6.22
09:55:05 up 519 days, 11:33, 0
users, load average: 1.53, 1.48, 1.47
09:55:05 up 519 days, 12:45, 0
users, load average: 1.36, 1.35, 1.47
09:55:05 up 515 days, 21:40, 0
users, load average: 1.47, 1.36, 1.36
09:55:05 up 519 days, 12:03, 0
users, load average: 1.44, 1.64, 1.71
09:55:05 up 519 days, 11:29, 0
users, load average: 1.78, 1.90, 1.78
~]# dcli -g cell_group -l root ‘uptime’
09:55:15 up 466 days, 20:25, 0
users, load average: 1.44, 1.35, 1.26
09:55:15 up 519 days, 14:48, 0
users, load average: 1.49, 1.44, 1.51
09:55:15 up 519 days, 14:47, 0
users, load average: 1.01, 0.96, 1.10
09:55:15 up 519 days, 14:47, 0
users, load average: 1.40, 1.32, 1.24
09:55:15 up 519 days, 14:46, 0
users, load average: 1.10, 1.25, 1.30
09:55:15 up 519 days, 14:46, 0
users, load average: 0.99, 1.14, 1.18
09:55:15 up 466 days, 19:37, 0
users, load average: 1.52, 1.24, 1.17
09:55:15 up 519 days, 14:46, 0
users, load average: 1.00, 1.40, 1.56
09:55:15 up 519 days, 14:46, 0
users, load average: 1.09, 1.22, 1.22
09:55:15 up 519 days, 14:46, 0
users, load average: 1.24, 1.27, 1.22
09:55:15 up 519 days, 14:46, 0
users, load average: 1.11, 1.14, 1.14
09:55:15 up 519 days, 14:46, 0
users, load average: 1.13, 1.27, 1.16
09:55:15 up 519 days, 14:46, 0
users, load average: 1.20, 1.12, 1.15
- First verify if any cell disks
already exist:
new_cell_group -l root “cellcli -e list celldisk attributes
name,status,freespace where disktype=harddisk”
0
0
0
0
0
0
0
0
0
0
0
0
- If no cell disks are created.
There will be no output.
new_cell_group -l root “cellcli -e list celldisk attributes
name,status,freespace where disktype=harddisk”
- If
this command returns a list of cell disks, then you should verify if any grid
disks exist:
new_cell_group -l root “cellcli -e list griddisk where
disktype=harddisk”
- If no grid disks are created.
There will be no output.
new_cell_group -l root “cellcli -e list griddisk where
disktype=harddisk”
- If any grid disks are
present, then you must ensure they are not in use by looking in V$ASM_DISK and
verifying the mount status is “closed” and the header status is
“former” or “candidate”:
oraenv
with value /u01/app/oracle
/ as sysasm
Production on Mon Oct 17 10:01:15 2016
Oracle. All rights reserved.
Edition Release 11.2.0.4.0 – 64bit Production
Clusters and Automatic Storage Management options
mode_status,header_status, mount_status from v$asm_disk where header_status
<> ‘MEMBER’ order by path;
- If it is OK to drop the grid
disks, you can use the following command to drop set of grid disks (by prefix
like, RECO):
new_cell_group -l root “cellcli -e drop griddisk all harddisk
prefix=RECO”
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
new_cell_group -l root “cellcli -e drop griddisk all harddisk
prefix=DATA”
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
new_cell_group -l root “cellcli -e drop griddisk all harddisk prefix=DBFS_DG”
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
- Verify Grid disks are dropped
new_cell_group -l root “cellcli -e list griddisk where
disktype=harddisk”
- Drop cell disks
new_cell_group -l root “cellcli -e drop celldisk all harddisk force”
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
successfully dropped
- If
no cell disks exist, create cell disks on all hard disks (output will be
similar to the following):
new_cell_group -l root “cellcli -e create celldisk all harddisk”
successfully created
successfully created
successfully created
successfully created
successfully created
successfully created
successfully created
successfully created
successfully created
successfully created
successfully created
successfully created
- Verify that celldisks were
created (output should be similar to the following)
[root@dm01db01 ~]# dcli -g
new_cell_group -l root “cellcli -e list celldisk attributes
name,status,freespace where disktype=harddisk”
dm01cel05: CD_00_dm01cel05 normal
528.6875G
dm01cel05: CD_01_dm01cel05 normal
528.6875G
dm01cel05: CD_02_dm01cel05 normal
557.8125G
dm01cel05: CD_03_dm01cel05 normal
557.8125G
dm01cel05: CD_04_dm01cel05 normal
557.8125G
dm01cel05: CD_05_dm01cel05 normal
557.8125G
dm01cel05: CD_06_dm01cel05 normal
557.8125G
dm01cel05: CD_07_dm01cel05 normal
557.8125G
dm01cel05: CD_08_dm01cel05 normal
557.8125G
dm01cel05: CD_09_dm01cel05 normal
557.8125G
dm01cel05: CD_10_dm01cel05 normal
557.8125G
dm01cel05: CD_11_dm01cel05 normal
557.8125G
Create
DATA and RECO grid disks at the same offsets found on the existing disks (outer
tracks)
- Verify the current sizes and
offsets for existing grid disks (on the existing cells):
~]# dcli -g cell_group -l root “cellcli -e list griddisk attributes
name,size | grep ‘CD_03’ | grep -i DBFS_DG”
DBFS_DG_CD_03_dm01cel01 29.125G
DBFS_DG_CD_03_dm01cel02 29.125G
DBFS_DG_CD_03_dm01cel03 29.125G
DBFS_DG_CD_03_dm01cel04 29.125G
DBFS_DG_CD_03_dm01cel06 29.125G
DBFS_DG_CD_03_dm01cel07 29.125G
DBFS_DG_CD_03_dm01cel08 29.125G
DBFS_DG_CD_03_dm01cel09 29.125G
DBFS_DG_CD_03_dm01cel10 29.125G
DBFS_DG_CD_03_dm01cel11 29.125G
DBFS_DG_CD_03_dm01cel12 29.125G
DBFS_DG_CD_03_dm01cel13 29.125G
DBFS_DG_CD_03_dm01cel14 29.125G
~]# dcli -g cell_group -l root “cellcli -e list griddisk attributes
name,size,offset | grep ‘CD_03’ | grep -i data”
DATA_CD_03_dm01cel01 260G 32M
DATA_CD_03_dm01cel02 260G 32M
DATA_CD_03_dm01cel03 260G 32M
DATA_CD_03_dm01cel04 260G 32M
DATA_CD_03_dm01cel06 260G 32M
DATA_CD_03_dm01cel07 260G 32M
DATA_CD_03_dm01cel08 260G 32M
DATA_CD_03_dm01cel09 260G 32M
DATA_CD_03_dm01cel10 260G 32M
DATA_CD_03_dm01cel11 260G 32M
DATA_CD_03_dm01cel12 260G 32M
DATA_CD_03_dm01cel13 260G 32M
DATA_CD_03_dm01cel14 260G 32M
~]# dcli -g cell_group -l root “cellcli -e list griddisk attributes
name,size,offset | grep ‘CD_03’ | grep -i reco”
RECO_CD_03_dm01cel01
268.6875G 260.046875G
RECO_CD_03_dm01cel02
268.6875G 260.046875G
RECO_CD_03_dm01cel03
268.6875G 260.046875G
RECO_CD_03_dm01cel04
268.6875G 260.046875G
RECO_CD_03_dm01cel06
268.6875G 260.046875G
RECO_CD_03_dm01cel07
268.6875G 260.046875G
RECO_CD_03_dm01cel08 268.6875G 260.046875G
RECO_CD_03_dm01cel09
268.6875G 260.046875G
RECO_CD_03_dm01cel10
268.6875G 260.046875G
RECO_CD_03_dm01cel11
268.6875G 260.046875G
RECO_CD_03_dm01cel12
268.6875G 260.046875G
RECO_CD_03_dm01cel13
268.6875G 260.046875G
RECO_CD_03_dm01cel14
268.6875G 260.046875G
- Create similar grid disks on the
new cells with the same size; be sure to create them in order of offset
position, from smallest to largest offset.
[root@dm01db01 ~]# dcli -g
new_cell_group -l root “cellcli -e CREATE GRIDDISK ALL HARDDISK
PREFIX=’DATA’, size=260G”
dm01cel05: GridDisk DATA_CD_00_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_01_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_02_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_03_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_04_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_05_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_06_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_07_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_08_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_09_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_10_dm01cel05
successfully created
dm01cel05: GridDisk DATA_CD_11_dm01cel05
successfully created
[root@dm01db01 ~]# dcli -g
new_cell_group -l root “cellcli -e CREATE GRIDDISK ALL HARDDISK
PREFIX=’RECO’, size=268.6875G”
dm01cel05: GridDisk RECO_CD_00_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_01_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_02_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_03_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_04_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_05_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_06_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_07_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_08_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_09_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_10_dm01cel05
successfully created
dm01cel05: GridDisk RECO_CD_11_dm01cel05
successfully created
- Validate that grid disks were
created properly on the new cells (size and offset matches the original cells)
[root@dm01db01 ~]# dcli -g
new_cell_group -l root “cellcli -e list griddisk attributes
name,size,offset | grep -i data”
dm01cel05: DATA_CD_00_dm01cel05 260G 32M
dm01cel05: DATA_CD_01_dm01cel05 260G 32M
dm01cel05: DATA_CD_02_dm01cel05 260G 32M
dm01cel05: DATA_CD_03_dm01cel05 260G 32M
dm01cel05: DATA_CD_04_dm01cel05 260G 32M
dm01cel05: DATA_CD_05_dm01cel05 260G 32M
dm01cel05: DATA_CD_06_dm01cel05 260G 32M
dm01cel05: DATA_CD_07_dm01cel05 260G 32M
dm01cel05: DATA_CD_08_dm01cel05 260G 32M
dm01cel05: DATA_CD_09_dm01cel05 260G 32M
dm01cel05: DATA_CD_10_dm01cel05 260G 32M
dm01cel05: DATA_CD_11_dm01cel05 260G 32M
[root@dm01db01 ~]# dcli -g
new_cell_group -l root “cellcli -e list griddisk attributes
name,size,offset | grep -i reco”
dm01cel05: RECO_CD_00_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_01_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_02_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_03_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_04_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_05_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_06_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_07_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_08_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_09_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_10_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_11_dm01cel05 268.6875G 260.046875G
Create
the new DBFS_DG grid disks only on disks 3 through 12 of the new storage
servers
- List
the size of the DBFS_DG grid disks on the original cells (they should all be
the same size) – we’ll call this size, “dbfs_dg_size”:
[root@dm01db01 ~]# dcli -c
dm01cel01 -l root “cellcli -e list griddisk attributes name,size | grep ‘DBFS_DG’”
dm01cel01: DBFS_DG_CD_02_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_03_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_04_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_05_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_06_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_07_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_08_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_09_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_10_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_11_dm01cel01 29.125G
- The DBFS_DG grid disks will be
automatically placed just after the RECO grid disks on the innermost tracks.
NOTE: This
command will “fail” on the first and second disks of each cell, but
that is expected since there is no more free space on those disks due to the
system partitions. The “error” message is similar to:
“Cell disks were skipped because they had no freespace for grid disks:
CD_00_dm01cel05, CD_01_dm01cel05.”
[root@dm01db01 ~]# dcli -g
new_cell_group -l root “cellcli -e CREATE GRIDDISK ALL HARDDISK PREFIX=’DBFS_DG’,
size=29.125G”
dm01cel05: Cell disks were
skipped because they had no freespace for grid disks: CD_00_dm01cel05, CD_01_dm01cel05.
dm01cel05: GridDisk DBFS_DG_CD_02_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_03_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_04_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_05_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_06_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_07_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_08_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_09_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_10_dm01cel05
successfully created
dm01cel05: GridDisk DBFS_DG_CD_11_dm01cel05
successfully created
Validate
that Grid Disks on all Cells are the Same Size
- All Cells at once:
[root@dm01db01 ~]# dcli -g
cell_group -l root “cellcli -e list griddisk attributes name,size,offset |
grep -i data”
dm01cel01: DATA_CD_00_dm01cel01 260G 32M
dm01cel01: DATA_CD_01_dm01cel01 260G 32M
dm01cel01: DATA_CD_02_dm01cel01 260G 32M
dm01cel01: DATA_CD_03_dm01cel01 260G 32M
dm01cel01: DATA_CD_04_dm01cel01 260G 32M
dm01cel01: DATA_CD_05_dm01cel01 260G 32M
dm01cel01: DATA_CD_06_dm01cel01 260G 32M
dm01cel01: DATA_CD_07_dm01cel01 260G 32M
dm01cel01: DATA_CD_08_dm01cel01 260G 32M
dm01cel01: DATA_CD_09_dm01cel01 260G 32M
dm01cel01: DATA_CD_10_dm01cel01 260G 32M
dm01cel01: DATA_CD_11_dm01cel01 260G 32M
…..
dm01cel14: DATA_CD_00_dm01cel14 260G 32M
dm01cel14: DATA_CD_01_dm01cel14 260G 32M
dm01cel14: DATA_CD_02_dm01cel14 260G 32M
dm01cel14: DATA_CD_03_dm01cel14 260G 32M
dm01cel14: DATA_CD_04_dm01cel14 260G 32M
dm01cel14: DATA_CD_05_dm01cel14 260G 32M
dm01cel14: DATA_CD_06_dm01cel14 260G 32M
dm01cel14: DATA_CD_07_dm01cel14 260G 32M
dm01cel14: DATA_CD_08_dm01cel14 260G 32M
dm01cel14: DATA_CD_09_dm01cel14 260G 32M
dm01cel14: DATA_CD_10_dm01cel14 260G 32M
dm01cel14: DATA_CD_11_dm01cel14 260G 32M
[root@dm01db01 ~]# dcli -g
cell_group -l root “cellcli -e list griddisk attributes name,size,offset |
grep -i reco”
dm01cel01: RECO_CD_00_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_01_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_02_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_03_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_04_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_05_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_06_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_07_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_08_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_09_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_10_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_11_dm01cel01 268.6875G 260.046875G
……
dm01cel13: RECO_CD_11_dm01cel13 268.6875G 260.046875G
dm01cel14: RECO_CD_00_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_01_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_02_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_03_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_04_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_05_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_06_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_07_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_08_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_09_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_10_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_11_dm01cel14 268.6875G 260.046875G
[root@dm01db01 ~]# dcli -g
cell_group -l root “cellcli -e list griddisk attributes name,size,offset |
grep -i DBFS_DG”
dm01cel01: DBFS_DG_CD_02_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_03_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_04_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_05_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_06_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_07_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_08_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_09_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_10_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_11_dm01cel01 29.125G 528.734375G
….
dm01cel14: DBFS_DG_CD_02_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_03_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_04_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_05_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_06_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_07_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_08_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_09_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_10_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_11_dm01cel14 29.125G 528.734375G
- One cell at a time:
[root@dm01db01 ~]# dcli -c dm01cel01
-l root “cellcli -e list griddisk attributes name,size,offset | grep -i
data”
dm01cel01: DATA_CD_00_dm01cel01 260G 32M
dm01cel01: DATA_CD_01_dm01cel01 260G 32M
dm01cel01: DATA_CD_02_dm01cel01 260G 32M
dm01cel01: DATA_CD_03_dm01cel01 260G 32M
dm01cel01: DATA_CD_04_dm01cel01 260G 32M
dm01cel01: DATA_CD_05_dm01cel01 260G 32M
dm01cel01: DATA_CD_06_dm01cel01 260G 32M
dm01cel01: DATA_CD_07_dm01cel01 260G 32M
dm01cel01: DATA_CD_08_dm01cel01 260G 32M
dm01cel01: DATA_CD_09_dm01cel01 260G 32M
dm01cel01: DATA_CD_10_dm01cel01 260G 32M
dm01cel01: DATA_CD_11_dm01cel01 260G 32M
[root@dm01db01 ~]# dcli -c dm01cel01
-l root “cellcli -e list griddisk attributes name,size,offset | grep -i
reco”
dm01cel01: RECO_CD_00_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_01_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_02_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_03_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_04_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_05_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_06_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_07_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_08_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_09_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_10_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_11_dm01cel01 268.6875G 260.046875G
[root@dm01db01 ~]# dcli -c dm01cel01
-l root “cellcli -e list griddisk attributes name,size,offset | grep -i DBFS_DG”
dm01cel01: DBFS_DG_CD_02_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_03_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_04_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_05_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_06_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_07_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_08_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_09_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_10_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_11_dm01cel01 29.125G 528.734375G
Add
the DBFS_DG grid disks to the existing DBFS_DG disk group
- Update the cellip.ora file
[root@dm01db01 ~]# vi
/etc/oracle/cell/network-config/cellip.ora
[root@dm01db01 network-config]#
cat /etc/oracle/cell/network-config/cellip.ora
cell=”192.168.2.9″
cell=”192.168.2.10″
cell=”192.168.2.11″
cell=”192.168.2.12″
cell=”192.168.2.13″
cell=”192.168.2.14″
cell=”192.168.2.15″
cell=”192.168.2.16″
cell=”192.168.2.17″
cell=”192.168.2.18″
cell=”192.168.2.19″
cell=”192.168.2.20″
cell=”192.168.2.21″
cell=”192.168.2.22″
[root@dm01db01 ~]# cd
/etc/oracle/cell/network-config/
[root@dm01db01 network-config]#
dcli -g ~/dbs_group -l root -d /etc/oracle/cell/network-config -f cellip.ora
[root@dm01db01 network-config]#
dcli -g ~/dbs_group -l root ls -l /etc/oracle/cell/network-config/cellip.ora
dm01db01: -rwxr-xr-x 1 oracle
oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db02: -rwx—— 1 oracle
oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db03: -rwx—— 1 oracle
oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db04: -rwx—— 1 oracle
oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db05: -rwx—— 1 oracle
oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db06: -rwx—— 1 oracle
oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db07: -rwx—— 1 oracle
oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db08: -rwx—— 1 oracle oinstall
279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db01-orcldb1 {/home/oracle}:.
oraenv
ORACLE_SID = [orcldb1] ? +ASM1
The Oracle base remains unchanged
with value /u01/app/oracle
dm01db01-+ASM1
{/home/oracle}:sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0
Production on Mon Oct 17 11:18:34 2016
Copyright (c) 1982, 2013,
Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise
Edition Release 11.2.0.4.0 – 64bit Production
With the Real Application
Clusters and Automatic Storage Management options
SQL> alter diskgroup DBFS_DG
add disk ‘o/*/DBFS_DG*dm01cel05’
rebalance power 32 NOWAIT;
Diskgroup altered.
Add
the DATA and RECO grid disks to the existing DATA and RECO disk groups
- From
one DB node add the new DATA disks to the DATA disk group (replace the pattern
below ” ‘o/*/DATA_DM01*new_cel01’, ‘o/*/DATA_DM01*new_cel02’,
‘o/*/DATA_DM01*new_cel03′” with list of actual new cells):
SQL>
alter diskgroup DATA add disk ‘o/*/DATA*dm01cel05’ rebalance power 32 NOWAIT;
Diskgroup
altered.
- From
another DB node add the new RECO disks to the RECO disk group (replace the
pattern below, ”
‘o/*/RECO_DM01*new_cel01′,’o/*/RECO_DM01*new_cel02′,’o/*/RECO_DM01*new_cel03’
” with list of actual new cells):
SQL> alter diskgroup RECO add
disk ‘o/*/RECO*dm01cel05’ rebalance power 32 NOWAIT;
Diskgroup altered.
- Monitor the progress of the
rebalance activities using this query (you can proceed with the rest of the
steps while this is continuing):
SQL> select * from
gv$asm_operation;
INST_ID GROUP_NUMBER OPERA STAT
POWER ACTUAL SOFAR
EST_WORK EST_RATE EST_MINUTES
ERROR_CODE
———- ———— —–
—- ———- ———- ———- ———- ———- ———–
——————————————–
7 1 REBAL WAIT 32
3 1 REBAL WAIT 32
2 1 REBAL WAIT 32
1 1 REBAL WAIT 32
6 1 REBAL WAIT 32
8 1 REBAL WAIT 32
4 1 REBAL WAIT 32
5 1 REBAL RUN 32 32
41666 45014 17263 0
8 rows selected.
SQL> select * from
gv$asm_operation;
no rows selected
Validate
that DATA, RECO, and DBFS_DG contain disks from new cells
Before:
dm01db01-orcldb1 {/home/oracle}:.
oraenv
ORACLE_SID = [orcldb1] ? +ASM1
The Oracle base remains unchanged
with value /u01/app/oracle
dm01db01-+ASM1
{/home/oracle}:sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0
Production on Mon Oct 17 10:43:16 2016
Copyright (c) 1982, 2013,
Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise
Edition Release 11.2.0.4.0 – 64bit Production
With the Real Application Clusters
and Automatic Storage Management options
SQL> set lines 200
SQL> set pages 200
SQL> select
d.failgroup,dg.name, sum(d.total_mb) cell_total_mb, count(1) cell_disk_count
from v$asm_disk d, v$asm_diskgroup dg where d.group_number = dg.group_number
group by d.failgroup, dg.name order by d.failgroup, dg.name;
FAILGROUP NAME CELL_TOTAL_MB
CELL_DISK_COUNT
——————————
—————————— ————- —————
DM01CEL01 DATA 3194880 12
DM01CEL01 RECO 3301632 12
DM01CEL01 DBFS_DG 298240 10
DM01CEL02 DATA 2928640 12
DM01CEL02 RECO 3026496 12
DM01CEL02 DBFS_DG 268416 10
DM01CEL03 DATA 3194880 12
DM01CEL03 RECO 3301632 12
DM01CEL03 DBFS_DG 298240 10
DM01CEL04 DATA 3194880 12
DM01CEL04 RECO 3301632 12
DM01CEL04 DBFS_DG 298240 10
DM01CEL06 DATA 3194880 12
DM01CEL06 RECO 3301632 12
DM01CEL06 DBFS_DG 298240 10
DM01CEL07 DATA 2928640 11
DM01CEL07 RECO 3026496 11
DM01CEL07 DBFS_DG 268416 10
DM01CEL08 DATA 3194880 12
DM01CEL08 RECO 3301632 12
DM01CEL08 DBFS_DG 298240 10
DM01CEL09 DATA 2396160 12
DM01CEL09 RECO 2476224 12
DM01CEL09 DBFS_DG 238592 10
DM01CEL10 DATA 3194880 12
DM01CEL10 RECO 3301632 12
DM01CEL10 DBFS_DG 298240 10
DM01CEL11 DATA 2928640 12
DM01CEL11 RECO 3026496 12
DM01CEL11 DBFS_DG 298240 10
DM01CEL12 DATA 3194880 12
DM01CEL12 RECO 3301632 12
DM01CEL12 DBFS_DG 298240 10
DM01CEL13 DATA 3194880 12
DM01CEL13 RECO 3301632 12
DM01CEL13 DBFS_DG 298240 10
DM01CEL14 DATA 3194880 12
DM01CEL14 RECO 3301632 12
DM01CEL14 DBFS_DG 298240 10
39 rows selected.
After
SQL> set lines 200
SQL> set pages 200
SQL> select
d.failgroup,dg.name, sum(d.total_mb) cell_total_mb, count(1) cell_disk_count
from v$asm_disk d, v$asm_diskgroup dg where d.group_number = dg.group_number
group by d.failgroup, dg.name order by d.failgroup, dg.name;
FAILGROUP NAME CELL_TOTAL_MB
CELL_DISK_COUNT
——————————
—————————— ————- —————
DM01CEL01 DATA 3194880 12
DM01CEL01 RECO 3301632 12
DM01CEL01 DBFS_DG 298240 10
DM01CEL02 DATA 2928640 12
DM01CEL02 RECO 3026496 12
DM01CEL02 DBFS_DG 268416 10
DM01CEL03 DATA 3194880 12
DM01CEL03 RECO 3301632 12
DM01CEL03 DBFS_DG 298240 10
DM01CEL04 DATA 2928640 12
DM01CEL04 RECO 3026496 12
DM01CEL04 DBFS_DG 268416 10
DM01CEL05 DATA 3194880 12
DM01CEL05 RECO 3301632 12
DM01CEL05 DBFS_DG 298240 10
DM01CEL06 DATA 3194880 12
DM01CEL06 RECO 3301632 12
DM01CEL06 DBFS_DG 298240 10
DM01CEL07 DATA 2928640 12
DM01CEL07 RECO 3026496 12
DM01CEL07 DBFS_DG 268416 10
DM01CEL08 DATA 3194880 12
DM01CEL08 RECO 3301632 12
DM01CEL08 DBFS_DG 298240 10
DM01CEL09 DATA 2396160 12
DM01CEL09 RECO 2476224 12
DM01CEL09 DBFS_DG 238592 10
DM01CEL10 DATA 3194880 12
DM01CEL10 RECO 3301632 12
DM01CEL10 DBFS_DG 298240 10
DM01CEL11 DATA 2928640 12
DM01CEL11 RECO 3026496 12
DM01CEL11 DBFS_DG 298240 10
DM01CEL12 DATA 3194880 12
DM01CEL12 RECO 3301632 12
DM01CEL12 DBFS_DG 298240 10
DM01CEL13 DATA 3194880 12
DM01CEL13 RECO 3301632 12
DM01CEL13 DBFS_DG 298240 10
DM01CEL14 DATA 3194880 12
DM01CEL14 RECO 3301632 12
DM01CEL14 DBFS_DG 298240 10
42 rows selected.
disks in any disk group:
name,total_mb,free_mb,offline_disks from v$asm_diskgroup;
———- ———- ————-
how to add a storage cell to an existing Exadata Database machine of same disk
size.