Uncategorized

7 Easy Steps to Verify RoCE Cabling on Oracle Exadata X8M

Oracle has provided the verify_roce_cables.py to verify if the rack is cabled correctly for Transport Layer systems based on a RDMA over Converged Ethernet (RoCE) Network Layer network on Exadata Database Machine X8M-2/X8M-8 and later hardware.

 

Here is the step by step procedure to verify RoCE cabling on Oracle Exadata X8M Database Machine.

 

Exadata Infrastrure details:

 

Exadata Model : Exadata X8M-2 half rack

Exadata Software version: 19.3.3.0.0

 

Step 1: Download 30323660 for your Exadata smart software. In my case I have downloaded it for 19.3.3

 

Step 2: Copy the patch to staging area and unzip

 

[root@dm01db01 ~]# cd /u01/app/oracle/software

 

[root@dm01db01 software]# ls -l

total 8

-rw-r–r– 1 root root 8143 Feb 10 08:12 p30323660_193300_Linux-x86-64.zip

 

[root@dm01db01 software]# unzip p30323660_193300_Linux-x86-64.zip

Archive:  p30323660_193300_Linux-x86-64.zip

  inflating: setup_roce_switch_passwordless.sh

  inflating: verify_roce_cables.py

  inflating: README.txt

 

[root@dm01db01 software]# ls -ltr

total 40

-r-xr-xr-x 1 root root  2895 Oct  1 17:38 setup_roce_switch_passwordless.sh

-rwxr-xr-x 1 root root 22637 Jan 28 07:51 verify_roce_cables.py

-rw-r–r– 1 root root   673 Jan 28 08:45 README.txt

-rw-r–r– 1 root root  8143 Feb 10 08:12 p30323660_193300_Linux-x86-64.zip

 

 

Step 3: Read the readme.html file carefully

 

 

Step 4: setup password-less ssh to RoCE switches

 

[root@dm01db01 software]# ./setup_roce_switch_passwordless.sh dm01sw-rocea01

enter switch ‘dm01sw-rocea01’ ‘admin’ password:

checking if ‘dm01sw-rocea01’ is reachable… [OK]

setting up SSH equivalency for ‘admin’ on ‘dm01sw-rocea01’… [OK]

 

[root@dm01db01 software]# ./setup_roce_switch_passwordless.sh dm01sw-roceb01

enter switch ‘dm01sw-roceb01’ ‘admin’ password:

checking if ‘dm01sw-roceb01’ is reachable… [OK]

setting up SSH equivalency for ‘admin’ on ‘dm01sw-roceb01’… [OK]

 


“RECOMMENDED READ: HERE’S EVERYTHING YOU NEED TO KNOW ABOUT ORACLE AUTONOMOUS DATABASE CLOUD”


Step 5: setup passwordless ssh to all compute nodes and storage cells

 

[root@dm01db01 software]# cd /opt/oracle.SupportTools/

 

[root@dm01db01 oracle.SupportTools]# ls -ltr setup_ssh_eq.sh

-r-xr-x— 1 root root 1518 Dec 11 22:53 setup_ssh_eq.sh

 

[root@dm01db01 oracle.SupportTools]# ./setup_ssh_eq.sh ~/all_group root welcome1

 

 

[root@dm01db01 oracle.SupportTools]# dcli -g ~/all_group -l root uptime

dm01cel01: 09:11:46 up 5 days, 21:30,  0 users,  load average: 2.44, 2.31, 2.05

dm01cel02: 09:11:46 up 5 days, 21:30,  0 users,  load average: 2.31, 1.74, 1.97

dm01cel03: 09:11:46 up 5 days, 21:30,  0 users,  load average: 2.97, 2.67, 2.60

dm01cel04: 09:11:46 up 5 days, 21:30,  0 users,  load average: 2.78, 1.95, 1.71

dm01cel05: 09:11:46 up 5 days, 21:30,  0 users,  load average: 3.83, 2.54, 2.05

dm01cel06: 09:11:46 up 5 days, 21:30,  0 users,  load average: 0.82, 1.30, 1.68

dm01cel07: 09:11:46 up 5 days, 21:30,  0 users,  load average: 1.55, 1.67, 1.71

dm01db01: 09:11:46 up 5 days, 21:37,  1 user,  load average: 0.63, 0.48, 0.39

dm01db02: 09:11:46 up 5 days, 21:36,  0 users,  load average: 0.23, 0.39, 0.34

dm01db03: 09:11:46 up 5 days, 21:36,  0 users,  load average: 0.41, 0.32, 0.34

dm01db04: 09:11:46 up 5 days, 21:36,  0 users,  load average: 0.19, 0.32, 0.29

 

 

Step 6: Create the file with compute nodes and storage cells in the formate given below

 

Lines in in nodes_list should be of format

adm<num>: <db1-hostname>

or

celadm<num>: <cell1-hostname>

 

 

[root@dm01db01 ~]# cat nodes_list

adm01: dm01db01

adm02: dm01db02

adm03: dm01db03

adm04: dm01db04

celadm01: dm01cel01

celadm02: dm01cel02

celadm03: dm01cel03

celadm04: dm01cel04

celadm05: dm01cel05

celadm06: dm01cel06

celadm07: dm01cel07

 

[root@dm01db01 ~]# cat roce_list

dm01sw-rocea01

dm01sw-roceb01

 

 

Step 7: Execute the RoCE Verify tool

 

[root@dm01db01 software]# cd /u01/app/oracle/software

 

[root@dm01db01 software]# ./verify_roce_cables.py -n ~/nodes_list -s ~/roce_list

Enter dm01sw-rocea01 admin password

User Access Verification

Enter dm01sw-roceb01 admin password

User Access Verification

Enter dm01sw-rocea01 admin password

User Access Verification

Enter dm01sw-roceb01 admin password

User Access Verification

SWITCH PORT (EXPECTED PEER)                      LEAF-1 (dm01sw-rocea01) : CABLE OK?                    LEAF-2 (dm01sw-roceb01) : CABLE OK?

———– —————             ——————————– : ——–            ——————————– : ———

    Eth1/4 (ISL peer switch)   :               dm01sw-roceb01 Ethernet1/4 : OK                       dm01sw-rocea01 Ethernet1/4 : OK

    Eth1/5 (ISL peer switch)   :               dm01sw-roceb01 Ethernet1/5 : OK                         dm01sw-rocea01 Ethernet1/5 : OK

    Eth1/6 (ISL peer switch)   :               dm01sw-roceb01 Ethernet1/6 : OK                         dm01sw-rocea01 Ethernet1/6 : OK

    Eth1/7 (ISL peer switch)   :               dm01sw-roceb01 Ethernet1/7 : OK                         dm01sw-rocea01 Ethernet1/7 : OK

    Eth1/8 ()                  :                                       – : –                                                  – : –

    Eth1/9 ()                  :                                       – : –                                                  – : –

   Eth1/10 ()                  :                                       – : –                                                  – : –

   Eth1/11 ()                  :                                       – : –                                                  – : –

   Eth1/12 ()                  :                                       – : –                                                  – : –

   Eth1/13 ()                  :                                       – : –                                                  – : –

   Eth1/14 ()                  :                                       – : –                                                  – : –

   Eth1/15 ()                  :                                       – : –                                                  – : –

   Eth1/16 ()                  :                                       – : –                                                  – : –

   Eth1/17 ()                  :                                       – : –                                                  – : –

   Eth1/18 ()                  :                                       – : –                                                  – : –

   Eth1/19 (adm04)             :                         dm01db04 port-1 : OK                                   dm01db04 port-2 : OK

   Eth1/20 (adm03)             :                         dm01db03 port-1 : OK                                   dm01db03 port-2 : OK

   Eth1/21 (adm02)             :                         dm01db02 port-1 : OK                                   dm01db02 port-2 : OK

   Eth1/22 (adm01)             :                         dm01db01 port-1 : OK                                   dm01db01 port-2 : OK

   Eth1/23 (celadm07)          :                        dm01cel07 port-1 : OK                                  dm01cel07 port-2 : OK

   Eth1/24 (celadm06)          :                        dm01cel06 port-1 : OK                                  dm01cel06 port-2 : OK

   Eth1/25 (celadm05)          :                        dm01cel05 port-1 : OK                                  dm01cel05 port-2 : OK

   Eth1/26 (celadm04)          :                        dm01cel04 port-1 : OK                                  dm01cel04 port-2 : OK

   Eth1/27 (celadm03)          :                        dm01cel03 port-1 : OK                                  dm01cel03 port-2 : OK

   Eth1/28 (celadm02)          :                        dm01cel02 port-1 : OK                                  dm01cel02 port-2 : OK

   Eth1/29 (celadm01)          :                        dm01cel01 port-1 : OK                                  dm01cel01 port-2 : OK

   Eth1/30 (ISL peer switch)   :             dm01sw-roceb01 Ethernet1/30 : OK                         dm01sw-rocea01 Ethernet1/30 : OK

   Eth1/31 (ISL peer switch)   :              dm01sw-roceb01 Ethernet1/31 : OK                        dm01sw-rocea01 Ethernet1/31 : OK

   Eth1/32 (ISL peer switch)   :              dm01sw-roceb01 Ethernet1/32 : OK                        dm01sw-rocea01 Ethernet1/32 : OK

   Eth1/33 (ISL peer switch)   :              dm01sw-roceb01 Ethernet1/33 : OK                        dm01sw-rocea01 Ethernet1/33 : OK

 

 

From the above output we can see the cabling is OK  which means it is good. There are no errors/failure reported for cabling.

 

The following states are applicable for “CABLE OK”

 

OK : Cabling looks good

SWAPPED : Cables have been crossed on same host

FAIL : Cabling check failed due to cabling error, discovery issues, etc.

 – : If any of the table entry has a ‘-‘, it can be treated as a don’t care or not applicable

 

 

Conclusion

 

In this article we have learned how to verify the cabling for Transport Layer systems based on a RDMA over Converged Ethernet (RoCE) Network Layer network on Exadata Database Machine X8M-2/X8M-8 and later hardware.


Have more queries related Exadata Database Machines?  Get in touch with Netsoftmate’s Oracle Engineered Systems experts to get the best solution on how to implement and maintain your Oracle Exadata Database Machines. Sign-up for a 30 mins free consultation by clicking the image below –


Expert Support for Oracle Exadata | Netsoftmate

Author


Avatar