Netsoftmate

Tag: Exadata diagnostic utilities

  • Exadata Diagnostic Utilities

    Overview
    As we know Exadata database
    machine is a combination of Hardware and Software. Over a period of time these
    hardware and software can failure or give performance issues. In my experience
    I have seen hardware failures during the Exadata install and as well immediate
    after completion of Installation. But the newer generations of Exadata machines
    are more stable and you might find fewer hardware failures.

    When you work with Support on
    hardware, software or performance issues they would request you to run the
    following Diagnostic utilities and uploaded the diagnostic data.
    The example of hardware, software
    or performance issues are as follows:
    • Hardware
      failure:       Hard disk, Flash disk, mother board, processor, DIMM and so on
    • Software
      issues:       Operating system, firmware, Oracle software and so on
    • Performance
      issues:       Operating system and database

    In this article I will
    demonstrator how to execute these utilities with a live example.

    Diagnostic Utilities at a Glance 
    Utility Name
    Description
    SOSREPORT
    collects
    detailed information about the hardware and configuration of Oracle Linux
    server
    SUNDIAG
    The
    utility is used for gathering hardware related information
    ILOM
    SNAPSHOT
    The
    utility is used for gathering hardware related information
    EXAWATCHER
    It
    collects the system data and reporting utilities. This information is mostly
    used for troubleshooting OS or performance issue.



    Now let’s take a look at these
    utilities in little more detail

    SOSREPORT UTILITY

    SOSREPORT
    utility collects detailed information about the hardware and configuration of
    Oracle Linux server.
    Steps to run SOSREPORT:
    • Log in to
      the compute node or storage cell as root user account for which you are running
      SOSREPORT (example: dm01db01) 

    [root@dm01db01
    ~]# id
    uid=0(root)
    gid=0(root) groups=0(root), 1(bin), 2(daemon), 3(sys), 4(adm), 6(disk),
    10(wheel)
    • You will
      find the sosreport utility under
      /usr/sbin location. You also use the Linux command “locate” to search for the
      utility.

    [root@dm01db01 ~]# locate
    sosreport
    /usr/sbin/sosreport
    • Execute the
      sosreport utility at the shell as follows

    [root@dm01db01 ~]# /usr/sbin/sosreport
    On the
    execution of this utility it will ask you for the input.
    a.     
    Press ENTER
    to continue, or CTRL-C to quit.
    Press ENTER
    on your keyboard
    b.     
    Please enter
    your first initial and last name [dm01db01]:
    Press ENTER
    to accept the default or enter a value of your choice
                           
    c.      
    Please enter
    the case number that you are generating this report for:
    Enter the SR
    number
    At this time
    it will take a while (approximately 5-6 minutes) and generate a compressed
    archive file in /tmp directory.
    • Use WinScp or
      similar utility to copy the output file to your desktop
    • Upload the output file to
      Oracle Support for review.

    Sample SOSREPORT Run:
    [root@dm01db01
    ~]# locate sosreport
    /usr/sbin/sosreport
    [root@dm01db01
    ~]# /usr/sbin/sosreport
    sosreport
    (version 2.2)
       This command will collect diagnostic and
    configuration
    information
    from this Oracle Linux system and installed
    applications.
      An archive containing the collected
    information will be
    generated
    in /tmp and may be provided to a Oracle USA
    support
    representative.
      Any information provided to Oracle USA will
    be treated
    in
    accordance with the published support policies at:
       https://linux.oracle.com/
      The generated archive may contain data
    considered
    sensitive
    and its content should be reviewed by the
    originating
    organization before being passed to any third
    party.
      No changes will be made to system
    configuration.
    Press
    ENTER to continue, or CTRL-C to quit.
    Please
    enter your first initial and last name [dm01db01]:
    Please
    enter the case number that you are generating this report for [None]:  3-1386095xxxx
      Running plugins. Please wait …
      Completed [66/66] …
    Creating
    compressed archive…
    Your
    sosreport has been generated and saved in:
      /tmp/sosreport-dm01db01.3-1386095-20161230023010-9f83.tar.xz
    The
    md5sum is: b1ccc01a773cbd36d463ba07b57c9f83
    Please
    send this file to your support representative.
    [root@dm01db01
    ~]#



    SUNDIAG UTILITY
    The utility is used for gathering
    hardware related information. Oracle Support uses this diagnostic data to
    assess the hardware failure.
    Steps to run SUNDIAG report:
    Follow the steps listed below to
    run the sundiag.sh utility.
    •  Log in to
      the compute node or storage cell as root user account for which you are running       SUNDIAG (example: dm01db01)
    • You will
      find the sundiag utility under /opt/oracle.SupportTools
      location. You also use the Linux           command “locate” to search for the utility.

    [root@dm01cel01 ~]# locate sundiag
    /opt/oracle.SupportTools/sundiag.sh
    • Run the
      sundiag.sh utility

     [root@dm01cel01 ~]#
    /opt/oracle.SupportTools/sundiag.sh
    • Use WinScp
      or similar utility to copy the output file to your deskto
    • Upload the
      output file to Oracle Support for review.

    Sample sundiag Run:
    [root@dm01cel01
    ~]# locate sundiag
    /opt/oracle.SupportTools/sundiag.sh
     [root@dm01cel01 ~]#
    /opt/oracle.SupportTools/sundiag.sh
    Oracle
    Exadata Database Machine – Diagnostics Collection Tool
    Last
    alert date is beyond 7 days. Skipping OSW/Metrics collection
    Gathering
    Linux information
    Skipping collection of
    OSWatcher/ExaWatcher logs, Cell Metrics and Traces
    Skipping ILOM collection. Use the
    ilom or snapshot options, or login to ILOM
    over the network and run Snapshot
    separately if necessary.
    /var/log/exadatatmp/sundiag_dm01cel01_1605NM70AD_2016_12_30_02_30
    Gathering
    Cell information
    ==============================================================================
    Done.
    The report files are bzip2 compressed in
    /var/log/exadatatmp/sundiag_dm01cel01_1605NM70AD_2016_12_30_02_30.tar.bz2
    ==============================================================================
    If you read the output carefully,
    sundiag utility doesn’t collect the ILOM data and Exawatcher data. That is the
    reason we need to run separate utilities to gather these data.

    ILOM SNAPSHOT
    This data is required to
    troubleshoot a hardware issue by Oracle Support.
    Follow the steps listed below to
    run the snapshot utility in GUI.
    Using GUI Interface:
    • For ILOM 2.x and 3.0
    • Open a web browser (use something
      other than Internet Explorer) and enter the following address

    http://dm01db01-ilom.domain.com 
    Note:  You may see complaints about security –
    ignore or override – click I understand the risks/Add exception/Confirm
    Security Exception
    • Enter root as User Name and its
      password and click on Log In. This will take you to the
      Home screen.

    • Select Maintenance
      -> Snapshot.  (ILOM 2.x and 3.0)

     The Service Snapshot
    Utility page appears.
    ILOM 2.x and 3.0 will
    look similar to this:
    • From the above Screen, Select
      Data Set “Normal”, Select Transfer Method as “Browser” and Click “Run”.

    Normal – Specifies
    that ILOM, operating system, and hardware information is collected.
    The download file will be saved
    according to your browser settings.
    Important Note:  Do
    not enable this option:     ‘Collect Only Log Files from Data Set‘. 
    Doing so will limit the snapshot to a much smaller sub-section of log files.
    •  In the dialog box, specify the directory to which to save the file and
      the file name.

    Click OK.
    The file is saved to the specified directory.
    • For ILOM version 3.1

    If the ILOM version is 3.1 which is the latest version shipped with
    X3/X4 Exadata. There a little difference in the design.
    • Open
      a web browser (use something other than Internet Explorer) and enter the
      following address

    http://dm01db08-ilom.domain.com
    Note:  You may see complaints about security –
    ignore or override – click I understand the risks/Add exception/Confirm
    Security Exception.
    • Enter root as User Name and its
      password and click on Log In. This will take you to the Home Screen.

    • Select ILOM
      Administration -> Maintenance -> Snapshot (ILOM 3.1)

    The Service Snapshot
    Utility page appears.
    ILOM 3.1 will look
    similar to this:
    • Select Data Set “Normal”, Select
      Transfer Method as “Browser” and Click “Run”.

    Normal – Specifies
    that ILOM, operating system, and hardware information is collected.
    The download file will be saved
    according to your browser settings.
    Important Note:  Do
    not enable this option:     ‘Collect Only Log Files from Data Set‘. 
    Doing so will limit the snapshot to a much smaller sub-section of log files.
    • In the dialog box, specify the directory to which to save the file and
      the file name.

    Click OK.
    The file is saved to the specified directory.


    • Using CLI

    Follow the steps listed below to
    run the snapshot utility in command line.

    • Log in to the ILOM CLI interface

    [root@dm01db01 ~]# ssh dm01db01-ilom
    Password:
    • You will see a similar output

    Oracle(R)
    Integrated Lights Out Manager
    Version
    3.0.16.15.j r101695
    Copyright
    (c) 2015, Oracle and/or its affiliates. All rights reserved.
    ->
    • After the ‘->’ prompt, type
      the command in below:

    -> set /SP/diag/snapshot
    dataset=normal
    Set ‘dataset’ to ‘normal’
    • Type the following command:

    -> set /SP/diag/snapshot
    dump_uri=sftp://root:welcome@10.10.10.51/tmp
    Set ‘dump_uri’ to ‘sftp://root:welcome@10.10.10.51/tmp’
    • Next cd to the snapshot directory
      and view the status:

    -> cd /SP/diag/snapshot
    /SP/diag/snapshot
    -> show
     /SP/diag/snapshot
        Targets:
        Properties:
            dataset = normal
            dump_uri = (Cannot show property)
            encrypt_output = false
            result = Running
        Commands:
            cd
            set
            show
    ->
    Wait for the snapshot process to
    complete. It may take several minutes.
    Continue to check until the
    status is shows ‘Snapshot Complete’
    Do not use, access, view, copy or
    move the snapshot file until it has completed.
    -> show
     /SP/diag/snapshot
        Targets:
        Properties:
            dataset = normal
            dump_uri = (Cannot show property)
            encrypt_output = false
            result = Collecting data into sftp://root:*****@10.10.10.51/tmp/dm01db01-ilom_10.10.23.56_2016-12-29T08-44-09.zip
    Snapshot Complete.
    Done.
        Commands:
            cd
            set
            show
    •  You can now exit the CLI
      interface and find your snapshot in the directory you specified.

    -> exit
    Connection to dm01db01-ilom
    closed.
    • The file name will look similar
      to this example:

     dm01db01-ilom_10.10.10.56_2016-12-29T08-44-09.zip
    Do not rename the snapshot file.

    Exawatcher
    The /opt/oracle.ExWatcher
    directory contains the Oracle ExaWatcher system data gathering and reporting
    utilities. This information is mostly used for troubleshooting OS or performance
    issue.
    Steps for ExaWatcher collection:
    •  Navigate to the Exawatcher directory and execute the GetExawatcherResults.sh script.

    [root@dm01db01 ~]#  cd /opt/oracle.ExaWatcher/

    [root@dm01db01
    oracle.ExaWatcher]# ls -ltr GetExaWatcherResults.sh
    -rwx—— 1 root root 21012 Oct
    21  2015 GetExaWatcherResults.sh

    [root@dm01db01 oracle.ExaWatcher]#
    ./GetExaWatcherResults.sh -h
    Usage:
     
    ./GetExaWatcherResults.sh {–from $FromTime [–to $ToTime] | –at
    $AtTime [–range $Hours]}
                                                  
    [–archivedir $ArchiveDir]
                                                   [–scp $UserName@SrvName]
                                                  
    [–filter $SamplerName]
                                                  
    [–resultdir $ResultDir]
    • To collect from/to a certain
      date and time:

    # ./GetExaWatcherResults.sh
    –from 07/31/2015_00:00:00 –to 07/31/2015_23:00:00
                                                       mm/dd/yyyy hh:mm:si
    Default output location:
    /opt/oracle.ExaWatcher/archive/ExtractedResults
    [root@dm01cel07
    ExtractedResults]# cd /opt/oracle.ExaWatcher/archive/ExtractedResults
    • To collect for a time range.
      In this case, we are collecting for 4 hrs before and after 1300:

    # ./GetExaWatcherResults.sh –at
    08/05/2015_13:00:00 –range 4
                                                     mm/dd/yyyy hh:mm:si

    The default archive directory
    is /opt/oracle.ExaWatcher/archive/ExtractedResults; however, you can change
    this using [-d|–archivedir] flag:
    Example of changed default
    archive location to /tmp/ExaWatcherArchive:
    # ./GetExaWatcherResults.sh
    –from 01/25/2014_13:00:00 –to 01/25/2014_14:00:00 –archivedir
    /tmp/ExaWatcherArchive
    Conclusion
    In this article we have seen
    different Exadata diagnostic utilities and how to execute them to collect the
    diagnostic data. These utilities are used on a daily basis to assess the
    hardware, software and performance issues.