Configure OEM12c to perform checkups on EXADATA (EXACHK)

Warning: This one may be longer than normal

How many times have you ran an exachk, produced the text file, and then had to either read the file on the compute node or copy it to your local machine?  Well there is another way to view the report of the EXACHK; it can even be ran on a timely basis so you have current EXACHK information on a regular basis.  What I’m talking about is using Oracle Enterprise Manager 12c for scheduling and producing the exachks. 

If you look at the Oracle Enterprise Manager 12c (OEM) documentation, you will eventually find information on the plug-in for “Oracle Engineered System Healthchecks”.  I’ll make it easy for you, here it is.  This plug-in allows OEM to process the XML output from the Exachk tool, which is part of the OEM by default.  This approach can be used to automate the assessment of Exadata V2, X2-2, X2-8, SPARC Supercluster, Exalogic, Exalytics, and Availability machine system for known configuration problems and best practices.

So why would you want to use OEM to review Exachk items?  The OEM interface allows you to view the output from an interface that is easy to view.  It allows you to setup metrics against the report to keep track of the health of your engineered system.  Lastly, the report can be scheduled to run on a predetermined timeframe so you don’t forget to run it.

Prerequisites

As with anything Oracle there are prerequisites that need to be meet.  Fro the healthcheck plug-in there are four simple ones.

  • Review Oracle Exadata Best Practices and Oracle Database Machine Monitoring Best Practices (757552.1/1110675.1)
  • Verify and enable the Exachk Tool.  The plug-in supports Exachk 2.2.2 and later
  • Operating systems and platforms (docs has the list, here)
  • Oracle Enterprise Manager 12c (12.1.0.3)

Plug-In

Once the prerequisites have been confirmed, then you will need to deploy the plug-in.  Before the plug-in can be deployed, in most environments, it needs to be applied to the software library.  To download or apply the plug-in to the software library, this is done from the Setup –> Extensibility –> Self Update menu.  Figure 1, shows that the plug-in has already been downloaded and applied to the software library.

Figure 1: Plug-in applied to software library

image

Add Healthcheck Targets

Ever wonder what the “Add Targets Declaratively by Specifying Target Monitoring Properties” when adding target manually is used for?  Well, this is one of those times where you get to use it.  When adding Healthcheck targets,  you will use the Setup –> Add Targets –> Add Targets Manually menu items.  When you reach the Add Targets Manually page,  select the “Add Targets Declaratively by Specifying Target Monitoring Properties” option; then select “Oracle Engineered System Healthchecks” from the drop down menu, then add the monitoring agent for one of the compute nodes in the Exadata machine.  Lastly click the Add Manually button.  Once all this is completed, you should end up on the “Add Oracle Engineered System Healthchecks” page (Figure 2).

Figure 2: Add Oracle Engineered System Healthchecks page

image

A few things to notice on this page.  First the Target Name, this can be any name you want to give the healthcheck target.  It will be used to look the target up in the All Targets page.  Other than All Targets, there is no direct way to get to the healthcheck targets (I’ve looked, still searching). 

In the properties dialog area, notice the Max Interval Allowed.  This is the number of days between consecutive Exachk runs.  You can make this longer or shorter; depends on you environment.  31 is the default and recommended by Oracle. 

The Exachk Results Directory is where OEM will read the Exachk information from.  In most of the environments, I usually use /home/oracle.  Fill in what you need.

Prerequisites for Exachk Tool

I know, I know, thought you took care of the prerequisites earlier before starting.  Well, this is another set of prerequisites.  This time, you need to make changes to the .profile (.bash_profile) for the Oracle use.    There are two entries that you need to add to the .profile.  Table 1 below provides the details and values that should be set with these parameters.

Table 1: Environment parameters for profiles

Parameter Value Why
RAT_COPY_EM_XML_FILES

1

Setting this environment variable will enable copying of results between all nodes in the cluster
RAT_OUTPUT <exachk output directory> Location where results will be copied to for all nodes in the cluster.

Run Exachk Tool

Here comes the fun part; you get to run the Exachk tool.  When I say run, I mean configure it to run in the back ground on the Exadata Machine.   What there is a way to keep Exachk running in the background?…Yep! 

The way this is done is by using the daemon (-d start) option to keep it running (Listing 1).  Although the Exachk process will start to run as a daemon, you will still be prompted for the information required.  Once you have provided all the information (password and such), the daemon will remember this until the daemon is stopped.

Listing 1: Start Exachk as daemon

./exachk –d start

Now with Exachk running in the background as a process; how do you get Exachk to run?  Well, you initially have to run Exachk.  When you do, everything you entered at the prompts previously will be used. 

To run Exachk in silent mode, use the –daemon option.  For the purpose of OEM, it is best to gather all the information the report can obtain.  From the command prompt run Exachk (Listing 2).

Listing 2: Run Exachk silently

./exachk –daemon –a

Metrics Collected

Now that the Exachk is done running, you want to see the results in OEM.  In order to do this, you need to tell the agent to upload the results.  Over time, the results will be uploaded; however, on the initial setup, you will want to run the upload process manually.  Listing 3 provides the command used to force the upload.

Listing 3: Force agent to upload Exachk results

./emctl control agent runCollection <target_name>:oracle_exadata_hc ExadataResults

Checking to see what is in OEM

Now that everything has been configured, Exachk ran and the management agent uploaded it, where can you see the results of this configuration?&n
bsp; The easiest way to see is from All Targets –> Engineered Systems –> Oracle Engineered System Healthchecks.  Then you need to click on the target name.

Once you are on the healthchecks page, you will notice a lot of information.  There are 3 main areas: Summary, Incidents and Problems, and Results Summary.  This is all the information that the Exachk report generated and the management agent uploaded.  Figure 3 is a small snapshot of the overall screen.

Figure 3: partial view of healthchecks page

image

Summary

Once again, Oracle Enterprise Manager 12c, has proven to be a tool that can do more than is expected.  Although, us command line junkies who like to run reports and various other items from the command line may look down on this; the Healthcheck plug-in is a cool way to review Exachk information.  Also a slick way to remember to run it each month….because it is scheduled.

Enjoy!

twitter: @dbasolved

blog: http://dbasolved.com

Please follow and like:
Comments
  • Hello! I’ve been reading your site for a while now and finally got the
    bravery to go ahead and give you a shout out from Lubbock Tx!
    Just wanted to tell you keep up the excellent job!

  • Hi Bobby,
    There is a 12.1.0.2 version of “Oracle Exadata Healthchecks” (which works in EM 12.1.0.2 if that’s not obvious) so the 12.1.0.3 prerequisite isn’t actually the case. Unfortunately I don’t have a 12.1.0.3 system to access so I can’t compare them.
    Martin

    • Martin,
      I’ve been looking though the documentation. I didn’t see it in 12.1.0.2; I may have overlooked it. Good to know though that there is a 12.1.0.2 version as well.
      Bobby

  • Bobby,
    I meant to post this nearly 2 months ago when I actually came across the problem, but I wanted to add to this great post with my experience.
    When I tried to start the exachk daemon, I kept receiving the error message below:

    [oracle@exababydb01 exachk]$ ./exachk -d start
    CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11.2.0.3/grid?[y/n][y]
    Checking ssh user equivalency settings on all nodes in cluster
    Default hostname is not mapped to management network rather its mapped to client network.
    Management network hostname = exadm1mgt1
    Default/client network hostnanme = exababydb01
    exachk is exiting.
    run exachk with -clusternodes command line argument and specify comma separated management network hostnames.
    eg ./exachk -clusternodes exababymgt1,
    

    After a few tries, I figured out what it wanted me to specifically add the “clusternodes” parameter before the “-d start”. I presume this was because of a bad database machine configuration file.

    [oracle@exababydb01 exachk]$ ./exachk -clusternodes exababydb01,exababydb02,exababydb03,exababydb04 -d start
    CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11.2.0.3/grid?[y/n][y]
    Checking ssh user equivalency settings on all nodes in cluster
    Node exababydb02 is configured for ssh user equivalency for oracle user
    Node exababydb03 is configured for ssh user equivalency for oracle user
    Node exababydb04 is configured for ssh user equivalency for oracle user
    ...
    exachk daemon is started with PID : 108401
    

    And presto that was it!
    Thanks for writing up this great post!!
    Cheers,
    Maaz

Comments are closed.

Enquire now

Give us a call or fill in the form below and we will contact you. We endeavor to answer all inquiries within 24 hours on business days.