Database Machine (EXADATA) Schematic problem in OEM12c

When adding an Exadata to Oracle Enterprise Manager 12c (OEM), it is pretty easy yet at times you may run into a problem or two.  Many of these problems can be minimized by using the Oracle Enterprise Manager 12c: Exadata Discovery Cookbook (cookbook).  The cookbook can be found here, so you don’t have to search to far for it.

As with adding any target in OEM, you add the Exadata the same way (Setup –> Add Target –> Add Targets Manually).  I’m not going to go into details here on adding the Exadata target because it is covered in the cookbook and it is also a basic function of OEM.   Just keep in mind, that adding an Exadata, you will need the databasemachine.xml or the catalog.xml file.  With the Exadata’s I’ve been adding, I have been using the databasemachine.xml.

What I do want to go over, is when you add an Exadata and the schematic of the Exadata box does not appear in OEM.  In figure 1, you see that I have two Exadata boxes listed on the Oracle Exadata Database Machines page. 

Figure 1: Oracle Exadata Database Machines

One of these Exadata, was added to OEM without a problem; the other the schematic didn’t come up.  Just to make this a bit easier, the top database machine is the one that is missing the schematic.  Figure 2, shows what I’m referring to about missing the schematic.  What shows up is only the legend, but no graphic for the machine itself.

Figure 2: Missing schematic

So, the question has to be asked, Why did this happen? Everything was configured correctly!  Well, I was so puzzled by this, I had to ask Oracle for a bit of help.  What we (myself and Oracle) identified was that the databasemachine.xml file had a duplicate entry in it. 

Fixing the databasemachine.xml file is explained in a bit.  First, you need to remove the database machine that is having the problem from OEM.  This can be done from the Database Machine menu (Figure 3). 

Figure 3: Database Machine menu

After clicking the Remove Target… option, you are taken to a page that provides options for removing the Exadat from OEM (Figure 4).   To remove the whole Exadata, the default radio button is fine.  Just make sure you select the option to Unsubscribe SNMP Subscriptions. Once you check the box, make sure all the passwords are correct.   Then click the Yes button in the right-hand corner.

Figure 4: Remove Database Machine

When the Exadata is removed, OEM will drop you back into the All Targets page.  At this point, the Exadata has been removed from OEM.

Now, you are back to a clean slate for adding the Exadata to OEM.  Before you rush to re-add the machine, we need to take a look at the databasemachine.xml file.  What you are looking for is any possible duplicates in the file.

Note: The EMOMS.log file on the OEM server helps in resolving this issue.

In reviewing the EMOMS.log, you will want to search for any interactions between OEM on the name of your Exadata.  In searching the EMOMS.log you may find an Oracle error, in my case the error was ORA-00001.  OEM couldn’t add a record to the EM_DBM_RACK_COMPONENTS table due to a PK error (Listing 1).

Listing 1: PK Error in EMOMS.log

ERROR model.DbmDiscovery logp.251 – ORA-00001: unique constraint (SYSMAN.EM_DBM_RACK_COMPONENTS_PK) violated

Knowing that there is a primary key error in the EM_DBM_RACK_COMPONETS table, this should lead you to think that the problem is in the databasemachine.xml file.   The databasemachine.xml file can be edited with any text editor; the file is an XML file.  A simple search for each of the values in the <ADMINNAME>.  Listing 2, shows you what I found in the databasemachine.xml file I’m working with.

Listing 2: Duplicates in databasemachine.xml

   <ITEM ID=”1″>

        <TYPE>ib</TYPE>

        <ADMINNAME>sw-ib1</ADMINNAME>

         <ADMINIP>10.*.*.*</ADMINIP>

         <ULOCATION>1</ULOCATION>

         <UHEIGHT>1</UHEIGHT>

    </ITEM>

    <ITEM ID=”13″>

        <TYPE>ib</TYPE>

        <ADMINNAME>sw-ib1</ADMINNAME>

        <ADMINIP>10.*.*.*</ADMINIP>

         <ULOCATION>20</ULOCATION>

         <UHEIGHT>1</UHEIGHT>

    </ITEM>

One of these entries in the databasemachine.xml file has to go; however, do not remove the whole <ITEM_ID></ITEM_ID>.  Just remove the name from the <ADMINNAME></ADMINNAME> in the second entry.  In this example, it would be <ITEM_ID=13> is the duplicate and I removed the name from the <ADMINNAME> tags.

After editing the databasemachine.xml file,  copy it to all the nodes within the Exadata.  This way you make sure that all the machines have the same file.  To do this,  use SCP (Listing 3).

Listing 3: SCP to copy to other compute nodes

scp ./databasemachine.xml root@db02:/opt/oracle.SupportTools/onecommand

Now, with the databasemachine.xml file in place, you can try and re-add the Exadata to OEM.  Again, keep in mind the easiest way to add the Exadata is to follow the cookbook (here).  That way you make sure all the points are covered.

Once you rediscover the Exadata in OEM, you should be able to pull the Database Machine Schematic up correctly (Figure 5). 

Figure 5: The correct view of Database Machine Schematic

Now, that the Exadata has been imported into OEM successfully, you can now add all the associated targets that are running on the Exadata (ie, database, ASM, listener, etc..). 

Enjoy!

twitter: @dbasolved

blog: http://dbasolved.com

Please follow and like: