Use DCLI to check #em12c agents on Exadata

For all the benefits of Oracle Enterprise Manager 12c (OEM), there is one draw back when using OEM with Real Application Clusters or Exadata…. checking the status of the EM Agent on each node. When a node is bounced, some times the EM Agent does not always restart. When this happens on a single instance box or a RAC; normally you would have to check each node individually and restart the agent. With Exadata, Oracle has made this check easier by providing the Distributed Command Line Interface (DCLI). Using DCLI, an administrator can run commands across all the nodes of the Exadata.

Note: The DCLI is a shell script that allows commands to be ran on multiple machines with a single invocation.

The example I’m going to use in this post is an example of checking the Enterprise Management Agents after rebooting the computer (db) nodes of an Exadata.

After rebooting each of the nodes, I needed to validate that the EM Agents were successfully restarted. All the agents were installed on each of the nodes in the same directory structure (/u01/app/oracle/product/agent12c), which makes using DCLI a breeze. Now that I know where the agents are installed, I need to create a file that lists all the compute (db) nodes in the Exadata; this file name is dbs_group. The last thing I need is a login for each of the nodes. On all Exadata systems the Oracle user can be used to login to each of the nodes.

Now that I have all the information I need; I need two command line switches that will be used with the DCLI command. These switches are –l (login) and –g (group).

The command that will be used to check all the EM Agents on the Exadata is:

dcli -l oracle -g ~/dbs_group /u01/app/oracle/product/agent12c/core/12.1.0.3.0/bin/emctl status agent

Once I run the DCLI command, I get output for all the EM Agents across all the nodes of the Exadata.

[e###db01:oracle:F####1] /home/oracle
> dcli -l oracle -g ~/dbs_group /u01/app/oracle/product/agent12c/core/12.1.0.3.0/bin/emctl status agent
e###db01: Oracle Enterprise Manager Cloud Control 12c Release 3
e###db01: Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
e###db01: —————————————————————
e###db01: Agent Version     : 12.1.0.3.0
e###db01: OMS Version       : 12.1.0.3.0
e###db01: Protocol Version : 12.1.0.1.0
e###db01: Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
e###db01: Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.3.0
e###db01: Agent Process ID : 6638
e###db01: Parent Process ID : 6307
e###db01: Agent URL         : https://e###db01.acme.com:3873/emd/main/
e###db01: Repository URL    : https://c#####01.acme.com:4904/empbs/upload
e###db01: Started at        : 2014-03-16 02:29:10
e###db01: Started by user   : oracle
e###db01: Last Reload       : (none)
e###db01: Last successful upload                       : 2014-03-17 10:15:41
e###db01: Last attempted upload                        : 2014-03-17 10:15:41
e###db01: Total Megabytes of XML files uploaded so far : 3.84
e###db01: Number of XML files pending upload           : 0
e###db01: Size of XML files pending upload(MB)         : 0
e###db01: Available disk space on upload filesystem    : 29.98%
e###db01: Collection Status                            : Collections enabled
e###db01: Heartbeat Status                             : Ok
e###db01: Last attempted heartbeat to OMS              : 2014-03-17 10:16:56
e###db01: Last successful heartbeat to OMS             : 2014-03-17 10:16:56
e###db01: Next scheduled heartbeat to OMS              : 2014-03-17 10:17:56
e###db01:
e###db01: —————————————————————
e###db01: Agent is Running and Ready
e###db02: Oracle Enterprise Manager Cloud Control 12c Release 3
e###db02: Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
e###db02: —————————————————————
e###db02: Agent Version     : 12.1.0.3.0
e###db02: OMS Version       : 12.1.0.3.0
e###db02: Protocol Version : 12.1.0.1.0
e###db02: Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
e###db02: Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.3.0
e###db02: Agent Process ID : 28588
e###db02: Parent Process ID : 28478
e###db02: Agent URL         : https://e###db02.acme.com:3873/emd/main/
e###db02: Repository URL    : https://c#####01.acme.com:4904/empbs/upload
e###db02: Started at        : 2014-03-16 02:24:59
e###db02: Started by user   : oracle
e###db02: Last Reload       : (none)
e###db02: Last successful upload                       : 2014-03-17 10:15:40
e###db02: Last attempted upload                        : 2014-03-17 10:15:40
e###db02: Total Megabytes of XML files uploaded so far : 3.18
e###db02: Number of XML files pending upload           : 0
e###db02: Size of XML files pending upload(MB)         : 0
e###db02: Available disk space on upload filesystem    : 38.92%
e###db02: Collection Status                            : Collections enabled
e###db02: Heartbeat Status                          &
nbsp; : Ok
e###db02: Last attempted heartbeat to OMS              : 2014-03-17 10:16:59
e###db02: Last successful heartbeat to OMS             : 2014-03-17 10:16:59
e###db02: Next scheduled heartbeat to OMS              : 2014-03-17 10:17:59
e###db02:
e###db02: —————————————————————
e###db02: Agent is Running and Ready
e###db03: Oracle Enterprise Manager Cloud Control 12c Release 3
e###db03: Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
e###db03: —————————————————————
e###db03: Agent Version     : 12.1.0.3.0
e###db03: OMS Version       : 12.1.0.3.0
e###db03: Protocol Version : 12.1.0.1.0
e###db03: Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
e###db03: Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.3.0
e###db03: Agent Process ID : 5834
e###db03: Parent Process ID : 5733
e###db03: Agent URL         : https://e###db03.acme.com:3873/emd/main/
e###db03: Repository URL    : https://c#####01.acme.com:4904/empbs/upload
e###db03: Started at        : 2014-03-16 02:42:35
e###db03: Started by user   : oracle
e###db03: Last Reload       : (none)
e###db03: Last successful upload                       : 2014-03-17 10:12:39
e###db03: Last attempted upload                        : 2014-03-17 10:12:39
e###db03: Total Megabytes of XML files uploaded so far : 8.54
e###db03: Number of XML files pending upload           : 0
e###db03: Size of XML files pending upload(MB)         : 0
e###db03: Available disk space on upload filesystem    : 39.97%
e###db03: Collection Status                            : Collections enabled
e###db03: Heartbeat Status                             : Ok
e###db03: Last attempted heartbeat to OMS              : 2014-03-17 10:16:38
e###db03: Last successful heartbeat to OMS             : 2014-03-17 10:16:38
e###db03: Next scheduled heartbeat to OMS              : 2014-03-17 10:17:38
e###db03:
e###db03: —————————————————————
e###db03: Agent is Running and Ready
e###db04: Oracle Enterprise Manager Cloud Control 12c Release 3
e###db04: Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
e###db04: —————————————————————
e###db04: Agent Version     : 12.1.0.3.0
e###db04: OMS Version       : 12.1.0.3.0
e###db04: Protocol Version : 12.1.0.1.0
e###db04: Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
e###db04: Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.3.0
e###db04: Agent Process ID : 5853
e###db04: Parent Process ID : 5734
e###db04: Agent URL         : https://e###db04.acme.com:3873/emd/main/
e###db04: Repository URL    : https://c#####01.acme.com:4904/empbs/upload
e###db04: Started at        : 2014-03-16 02:53:08
e###db04: Started by user   : oracle
e###db04: Last Reload       : (none)
e###db04: Last successful upload                  : 2014-03-17 10:16:33
e###db04: Last attempted upload                   : 2014-03-17 10:16:33
e###db04: Total Megabytes of XML files uploaded so far : 2.81
e###db04: Number of XML files pending upload           : 0
e###db04: Size of XML files pending upload(MB)         : 0
e###db04: Available disk space on upload filesystem    : 45.75%
e###db04: Collection Status                         : Collections enabled
e###db04: Heartbeat Status                             : Ok
e###db04: Last attempted heartbeat to OMS           : 2014-03-17 10:16:50
e###db04: Last successful heartbeat to OMS          : 2014-03-17 10:16:50
e###db04: Next scheduled heartbeat to OMS           : 2014-03-17 10:17:50
e###db04:
e###db04: —————————————————————
e###db04: Agent is Running and Ready

From the output, I can clearly see that all the EM Agents are monitoring the Exadata and are in running status. I really like the DCLI command option, makes life simple especially when needing to check on multiple agents within an environment. Makes me wonder if I can put DCLI on a traditional RAC system?

Enjoy!

twitter: @dbasolved

blog: http://dbasolved.com

Please follow and like: