Netsoftmate

Tag: reset

  • Exadata – Display and Clear Fault using Fault Manager (faultmgmt)

    Exadata Database Machine consists of a storage grid, compute grid, and network grid. Each grid, or hardware layer, is built with multiple high-performing, industry-standard Oracle servers to provide hardware and system fault tolerance. The hardware components are subjected to failure. Most common failure on Exadata is Hard Disk failure on Storage Cells. With the latest generation of Exadata the hardware failures are very minimal and less troublesome. 


    The Exadata Storage Cells and Compute nodes consists of several hardware components, such as:

    • Hard disk
    • Flash disk 
    • Physical Memory
    • Processor  
    • IB ports
    • Mother Board
    • Batteries
    • Power Supply
    • and So on





    In this article we will demonstrate how to view the hardware fault and clear it using ILOM fault manager (faultmgmt).




    Steps to display and clear hardware fault using faultmgmt:




    Step 1: Login to compute node ILOM where the fault occurred


    [root@dm01db01 ~]# ssh dm01db02-ilom
    Password:


    Oracle(R) Integrated Lights Out Manager


    Version 4.0.0.24 r121523


    Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.


    Warning: HTTPS certificate is set to factory default.


    Hostname: dm01db02-ilom


    Step 2: Check if the fault manager is supported. If you get the output like below then fault manager is supported.

    -> show /SP/faultmgmt/shell


     /SP/faultmgmt/shell
        Targets:


        Properties:


        Commands:
            cd
            show
            start




    Step 3: Start the fault manager shell


    -> start /SP/faultmgmt/shell
    Are you sure you want to start /SP/faultmgmt/shell (y/n)? y


    Step 4: Execute the following command to display the fault. Here we can see that there is no issue with hardware but he ILOM file system is 100% full.


    faultmgmtsp> fmadm faulty
    ——————- ———————————— ————– ——–
    Time                UUID                                 msgid          Severity
    ——————- ———————————— ————– ——–
    2018-06-17/15:55:32 2a854ad2-4a31-e829-e26c-c84ba212d7f2 ILOM-8000-JV   Major


    Problem Status           : open
    Diag Engine              : fdd 1.0
    System
       Manufacturer          : Oracle Corporation
       Name                  : Exadata X5-2
       Part_Number           : Exadata X5-2
       Serial_Number         : AK00XXXXXX


    System Component
       Manufacturer          : Oracle Corporation
       Name                  : ORACLE SERVER X5-2
       Part_Number           : 7090664
       Serial_Number         : 15XXXXXXXX
       Firmware_Manufacturer : Oracle Corporation
       Firmware_Version      : (ILOM)4.0.0.24
       Firmware_Release      : (ILOM)2017.09.23


    —————————————-
    Suspect 1 of 1
       Problem class  : defect.ilom.fs.full
       Certainty      : 100%
       Affects        : /SYS/SP
       Status         : faulted


       FRU
          Status            : faulty
          Location          : /SYS/SP
          Manufacturer      : Oracle Corporation
          Name              : SP
          Part_Number       : PILOT3
          Chassis
             Manufacturer   : Oracle Corporation
             Name           : ORACLE SERVER X5-2
             Part_Number    : 7090664
             Serial_Number  : 1547NM10CX


    Description : An ILOM filesystem has exceeded the filesystem capacity
                  limit.


    Response    : The chassis wide service-required LED will be illuminated.


    Impact      : ILOM commands may fail, especially those which make
                  configuration changes.


    Action      : Please refer to the associated reference document at
                  http://support.oracle.com/msg/ILOM-8000-JV for the latest
                  service procedures and policies regarding this diagnosis.


    Step 5: Execute the below command to clear the fault


    faultmgmtsp> fmadm acquit UUID –> Get the UUID from the from output of the above command.


    faultmgmtsp> fmadm acquit 2a854ad2-4a31-e829-e26c-c84ba212d7f2


    Step 6: Verify that the fault is cleared


    faultmgmtsp> fmadm faulty
    No faults found


    Step 7: Exit from the fault manager


    faultmgmtsp> exit


    Step 8: Reset the ILOM service processor


    -> reset /SP
    Are you sure you want to reset /SP (y/n)? y
    Performing reset on /SP


    Step 9: Exit from the ILOM


    -> exit
    Connection to dm01db02-ilom closed.


    Step 10: Connec to ILOM and verify the ILOM SP is restarted


    [root@dm01db01 ~]# ssh dm01db02-ilom
    Password:


    Oracle(R) Integrated Lights Out Manager


    Version 4.0.0.24 r121523


    Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.


    Warning: HTTPS certificate is set to factory default.


    Hostname: dm01db02-ilom




    -> show -d properties /SP/clock uptime


     /SP/clock
        Properties:
            uptime = 0 days, 00:08:02




    Conclusion


    In this article we have learned how to display and clear a fault using fault manager (faultmgmt). The Fault Management Shell is the preferred method for displaying the details of a diagnosed fault. faultmgmt support for command shell varies depending ILOM release level and server product model.