Oracle has provided the verify_roce_cables.py to verify if the rack is cabled correctly for Transport Layer systems based on a RDMA over Converged Ethernet (RoCE) Network Layer network on Exadata Database Machine X8M-2/X8M-8 and later hardware.
Here is the step by step procedure to verify RoCE cabling on Oracle Exadata X8M Database Machine.
Exadata Infrastrure details:
Exadata Model : Exadata X8M-2 half rack
Exadata Software version: 19.3.3.0.0
Step 1: Download 30323660 for your Exadata smart software. In my case I have downloaded it for 19.3.3
Step 2: Copy the patch to staging area and unzip
[root@dm01db01 ~]# cd /u01/app/oracle/software
[root@dm01db01 software]# ls -l
total 8
-rw-r–r– 1 root root 8143 Feb 10 08:12 p30323660_193300_Linux-x86-64.zip
[root@dm01db01 software]# unzip p30323660_193300_Linux-x86-64.zip
Archive: p30323660_193300_Linux-x86-64.zip
inflating: setup_roce_switch_passwordless.sh
inflating: verify_roce_cables.py
inflating: README.txt
[root@dm01db01 software]# ls -ltr
total 40
-r-xr-xr-x 1 root root 2895 Oct 1 17:38 setup_roce_switch_passwordless.sh
-rwxr-xr-x 1 root root 22637 Jan 28 07:51 verify_roce_cables.py
-rw-r–r– 1 root root 673 Jan 28 08:45 README.txt
-rw-r–r– 1 root root 8143 Feb 10 08:12 p30323660_193300_Linux-x86-64.zip
Step 3: Read the readme.html file carefully
Step 4: setup password-less ssh to RoCE switches
[root@dm01db01 software]# ./setup_roce_switch_passwordless.sh dm01sw-rocea01
enter switch ‘dm01sw-rocea01’ ‘admin’ password:
checking if ‘dm01sw-rocea01’ is reachable… [OK]
setting up SSH equivalency for ‘admin’ on ‘dm01sw-rocea01’… [OK]
[root@dm01db01 software]# ./setup_roce_switch_passwordless.sh dm01sw-roceb01
enter switch ‘dm01sw-roceb01’ ‘admin’ password:
checking if ‘dm01sw-roceb01’ is reachable… [OK]
setting up SSH equivalency for ‘admin’ on ‘dm01sw-roceb01’… [OK]
“RECOMMENDED READ: HERE’S EVERYTHING YOU NEED TO KNOW ABOUT ORACLE AUTONOMOUS DATABASE CLOUD”
Step 5: setup passwordless ssh to all compute nodes and storage cells
[root@dm01db01 software]# cd /opt/oracle.SupportTools/
[root@dm01db01 oracle.SupportTools]# ls -ltr setup_ssh_eq.sh
-r-xr-x— 1 root root 1518 Dec 11 22:53 setup_ssh_eq.sh
[root@dm01db01 oracle.SupportTools]# ./setup_ssh_eq.sh ~/all_group root welcome1
[root@dm01db01 oracle.SupportTools]# dcli -g ~/all_group -l root uptime
dm01cel01: 09:11:46 up 5 days, 21:30, 0 users, load average: 2.44, 2.31, 2.05
dm01cel02: 09:11:46 up 5 days, 21:30, 0 users, load average: 2.31, 1.74, 1.97
dm01cel03: 09:11:46 up 5 days, 21:30, 0 users, load average: 2.97, 2.67, 2.60
dm01cel04: 09:11:46 up 5 days, 21:30, 0 users, load average: 2.78, 1.95, 1.71
dm01cel05: 09:11:46 up 5 days, 21:30, 0 users, load average: 3.83, 2.54, 2.05
dm01cel06: 09:11:46 up 5 days, 21:30, 0 users, load average: 0.82, 1.30, 1.68
dm01cel07: 09:11:46 up 5 days, 21:30, 0 users, load average: 1.55, 1.67, 1.71
dm01db01: 09:11:46 up 5 days, 21:37, 1 user, load average: 0.63, 0.48, 0.39
dm01db02: 09:11:46 up 5 days, 21:36, 0 users, load average: 0.23, 0.39, 0.34
dm01db03: 09:11:46 up 5 days, 21:36, 0 users, load average: 0.41, 0.32, 0.34
dm01db04: 09:11:46 up 5 days, 21:36, 0 users, load average: 0.19, 0.32, 0.29
Step 6: Create the file with compute nodes and storage cells in the formate given below
Lines in in nodes_list should be of format
adm<num>: <db1-hostname>
or
celadm<num>: <cell1-hostname>
[root@dm01db01 ~]# cat nodes_list
adm01: dm01db01
adm02: dm01db02
adm03: dm01db03
adm04: dm01db04
celadm01: dm01cel01
celadm02: dm01cel02
celadm03: dm01cel03
celadm04: dm01cel04
celadm05: dm01cel05
celadm06: dm01cel06
celadm07: dm01cel07
[root@dm01db01 ~]# cat roce_list
dm01sw-rocea01
dm01sw-roceb01
Step 7: Execute the RoCE Verify tool
[root@dm01db01 software]# cd /u01/app/oracle/software
[root@dm01db01 software]# ./verify_roce_cables.py -n ~/nodes_list -s ~/roce_list
Enter dm01sw-rocea01 admin password
User Access Verification
Enter dm01sw-roceb01 admin password
User Access Verification
Enter dm01sw-rocea01 admin password
User Access Verification
Enter dm01sw-roceb01 admin password
User Access Verification
SWITCH PORT (EXPECTED PEER) LEAF-1 (dm01sw-rocea01) : CABLE OK? LEAF-2 (dm01sw-roceb01) : CABLE OK?
———– ————— ——————————– : ——– ——————————– : ———
Eth1/4 (ISL peer switch) : dm01sw-roceb01 Ethernet1/4 : OK dm01sw-rocea01 Ethernet1/4 : OK
Eth1/5 (ISL peer switch) : dm01sw-roceb01 Ethernet1/5 : OK dm01sw-rocea01 Ethernet1/5 : OK
Eth1/6 (ISL peer switch) : dm01sw-roceb01 Ethernet1/6 : OK dm01sw-rocea01 Ethernet1/6 : OK
Eth1/7 (ISL peer switch) : dm01sw-roceb01 Ethernet1/7 : OK dm01sw-rocea01 Ethernet1/7 : OK
Eth1/8 () : – : – – : –
Eth1/9 () : – : – – : –
Eth1/10 () : – : – – : –
Eth1/11 () : – : – – : –
Eth1/12 () : – : – – : –
Eth1/13 () : – : – – : –
Eth1/14 () : – : – – : –
Eth1/15 () : – : – – : –
Eth1/16 () : – : – – : –
Eth1/17 () : – : – – : –
Eth1/18 () : – : – – : –
Eth1/19 (adm04) : dm01db04 port-1 : OK dm01db04 port-2 : OK
Eth1/20 (adm03) : dm01db03 port-1 : OK dm01db03 port-2 : OK
Eth1/21 (adm02) : dm01db02 port-1 : OK dm01db02 port-2 : OK
Eth1/22 (adm01) : dm01db01 port-1 : OK dm01db01 port-2 : OK
Eth1/23 (celadm07) : dm01cel07 port-1 : OK dm01cel07 port-2 : OK
Eth1/24 (celadm06) : dm01cel06 port-1 : OK dm01cel06 port-2 : OK
Eth1/25 (celadm05) : dm01cel05 port-1 : OK dm01cel05 port-2 : OK
Eth1/26 (celadm04) : dm01cel04 port-1 : OK dm01cel04 port-2 : OK
Eth1/27 (celadm03) : dm01cel03 port-1 : OK dm01cel03 port-2 : OK
Eth1/28 (celadm02) : dm01cel02 port-1 : OK dm01cel02 port-2 : OK
Eth1/29 (celadm01) : dm01cel01 port-1 : OK dm01cel01 port-2 : OK
Eth1/30 (ISL peer switch) : dm01sw-roceb01 Ethernet1/30 : OK dm01sw-rocea01 Ethernet1/30 : OK
Eth1/31 (ISL peer switch) : dm01sw-roceb01 Ethernet1/31 : OK dm01sw-rocea01 Ethernet1/31 : OK
Eth1/32 (ISL peer switch) : dm01sw-roceb01 Ethernet1/32 : OK dm01sw-rocea01 Ethernet1/32 : OK
Eth1/33 (ISL peer switch) : dm01sw-roceb01 Ethernet1/33 : OK dm01sw-rocea01 Ethernet1/33 : OK
From the above output we can see the cabling is OK which means it is good. There are no errors/failure reported for cabling.
The following states are applicable for “CABLE OK”
OK : Cabling looks good
SWAPPED : Cables have been crossed on same host
FAIL : Cabling check failed due to cabling error, discovery issues, etc.
– : If any of the table entry has a ‘-‘, it can be treated as a don’t care or not applicable
Conclusion
In this article we have learned how to verify the cabling for Transport Layer systems based on a RDMA over Converged Ethernet (RoCE) Network Layer network on Exadata Database Machine X8M-2/X8M-8 and later hardware.
Have more queries related Exadata Database Machines? Get in touch with Netsoftmate’s Oracle Engineered Systems experts to get the best solution on how to implement and maintain your Oracle Exadata Database Machines. Sign-up for a 30 mins free consultation by clicking the image below –
