Hang Time – Disconnect a session

The other day I was installing Oracle Enterprise Manager 12c Cloud Control and ran into a few problems.  I had to scrub the install and start from scratch due to a naming issue in the /etc/hosts file (story for another time).  In order to start over, I had to remove the SYSMAN and associate schemas from the repository database.  When I went to drop the schema, the database said the schema was still connected and active.  Besides being frustrated at this point, I had to identify and kill the sessions that were still active for the SYSMAN schema.  

Any time when needing to find active sessions within the database, we can use the V$SESSION view.  The only thing we need to grab from this view is the SID and SERIAL#.  The SQL I used to get this is below:

select sid, serial#, username
from v$session
where username like ‘SYSMAN%’
and status = ‘ACTIVE’;

This will pull all the sessions that are tied SYSMAN.  Being that this was an Oracle Enterprise Manager 12c repository, there are a few different SYSMAN schemas we need to account for.

Once we have all the SID and SERIAL#; we need to disconnect the session from the database.  Yes, we can do an ‘alter system kill session‘, but this would abandon the transaction for the session, if any.  To cleanly disconnect the session, we can use the ‘alter system disconnect session‘ command.  This command has been around since 8i, and does a good job of disconnecting a session cleanly.  The syntax for the command is listed below:

alter system disconnect session ‘sid,serial#’ immediate;

or

alter system disconnect session ‘sid,serial#’ post_transaction;

Now that the sessions that were hanging me up have been disconnected, I can clean up the OEM repository and prepare to restart the installer.

 

OEM Job moves – Not easy!

Note: Scripts contained within work for 11g OEM repository databases.

How many times has Oracle requested or suggested that you should be running the latest version of Oracle Enterprise Manager?  How many times has your organization just wanted to upgrade versions to be on the latest of Oracle Enterprise Manager?  The answer to these questions are really the same…. way to often or not often enough (depends on your view of the world)!

For the most part, many of the things in Oracle Enterprise Manager (OEM) can be exported and imported between environments (check compatibilities prior migrations).  There are only a few functionalities of OEM that cannot be exported.  This would be the OEM job library and associated jobs!  Now the question is, if I have to migrate from my current version of OEM to a newer version, how do I do this without loosing my jobs?  The standard answer from Oracle would be to upgrade your OEM environment; which is the correct response.  However, when looking at the upgrade paths for OEM, there is a chance of losing visibility of the enterprise during the upgrade.  For many of the customers I work with this is not a good thing or an option.  What other solutions are there if I just want to install a new OEM environment and then move my jobs?

At this point, I have posed a few questions dealing with moving jobs between OEM environments.  Since export/import is off the table, what options do we have to move jobs to a new OEM environment?  After discussing this with a few friends and looking through a lot of MOS notes, I can honestly say the best way is to manually move the jobs (until Oracle decides implement export/import for jobs).

I have come to find that there are two ways that we can migrate jobs between OEMs.  The first is documented at DBAKevlar’s blog (www.dbakevlar.com).  After talking with DBAKevlar, her approach works but is not/will not be supported by Oracle.  The second way is documented here. If you have a lot of jobs, this may become a time consuming project for you or your department.

The first things we need to identify are the tables that relate to OEM jobs.  The below script will help you in identifying tables used for jobs.

select count(*) from dba_tables
where owner = ‘SYSMAN’
and table_name like upper(‘%mgmt%job%’)
and table_name not like upper(‘%bcn%’)
and table_name not like upper(‘%gensvc%’);
 
 
Next I wanted to find how many of these tables had data associated with them.  If the table had no data, I didn’t want them listed.  This limited the pool of tables I had to look at.

declare
v_owner varchar2(30);
v_table varchar2(30);
countval pls_integer;
tabcount pls_integer;
v_limit integer := 0;
cursor tab_count_cur is  
      select owner, table_name from dba_tables
   
     where owner = ‘SYSMAN’
       
     and table_name like upper(‘%mgmt%job%’)
       
    and table_name not like upper(‘%bcn%’)
   
    and table_name not like upper(‘%gensvc%’)
   
    order by table_name asc;
begin
for i in tab_count_cur loop
execute immediate ‘select count(1) from ‘ || i.owner||’.'||i.table_name into countval;      
if countval > v_limit then
     
      dbms_output.put_line(i.owner||’.'||i.table_name ||’ has a count of ‘||countval);
     
end if;
end loop;
end;
/

After identifying the tables for jobs, it looks to be a lot of tables to deal with.  After consulted the Oracle documentation; Oracle provides views that reference these tables for easier identification of jobs.   Next, I wanted to find out what views had the data I needed.   I was looking to see if there was any with identical counts (identical counts typically indicate these items can go together).  The below script can be used to get the views that has information associated with jobs.

declare
countval pls_integer;
v_limit integer := 0;
cursor tab_count_cur is
                  select owner, view_name, read_only
                  from dba_views
                 where owner = ‘SYSMAN’
                 and view_name like upper(‘%mgmt%job%’)
                 order by view_name asc;
begin
for i in tab_count_cur loop
execute immediate ‘select count(1) from ‘ || i.owner||’.'||i.view_name into countval;
    if countval > v_limit then
              dbms_output.put_line(i.owner||’.'||i.view_name ||’ has a count of ‘||countval);
    end if; end loop;
end;
/

Notice, the list of views is much smaller than the list of tables.  Making identifying jobs a bit easier. Lets start getting the information we need to move jobs.

In order to move jobs, we need to identify the general information for the jobs.  This information can be found by joining the MGMT$JOBS and MGMT$JOB_TARGETS.  Depending on the number of jobs you have, this will either be a short or long list of jobs.

select t1.job_id, substr(nvl(t2.target_name, ‘OEM Targetless’),0,20), t1.job_name, t1.job_type, t1.job_owner, substr(t1.job_description,0,30) description,  t1.target_type, t1.timezone_type, t1.timezone_region, t1.schedule_type, t1.is_library, t1.start_time
from sysman.mgmt$jobs t1, sysman.mgmt$job_targets t2
where t1.job_id = t2.job_id
and t1.job_name = t2.job_name
and t1.target_type is not null;

In this case, we are looking for backup jobs.  The job type identified above is “Backup”.  In order to find the internal id for backup jobs, we will need to query the MGMT_JOB_TYPE_INFO table.The key thing to look for here is the JOB_TYPE from the MGMT$JOBS view.  This will tell you the type of job it is.  Also, you will need this to find the internal id for the job in the next SQL statement.

select job_type_id, minor_version2, last_modified_date
from mgmt_job_type_info
where job_type = ‘Backup’;

Now that we have the internal id for the backup type jobs, we can find the parameters that are required for this job type.  This query will not give you the current values of the parameters just the name of the parameters associated with the internal id.

select parameter_name
from mgmt_job_type_param_dsply_info
where job_type_id in (
                  select job_type_id
                  from mgmt_job_type_info
                  where job_type = ‘Backup’
)
order by parameter_name asc;

Almost done.  With the job_name, internal id, and the parameters; we can now identify the values of the parameters for the jobs we want to migrate.  This will give us the information needed to ensure that we have all the correct values for the migration.  The important thing to remember at this point is that PARAMETER_TYPE will tell us where we need to look for the information.  The below SQL will help you in identifying what needs to be gathered.

select job_id, parameter_name,
           case parameter_type
                    when 0 then ‘Vector’
                    when 1 then ‘Scalar’
                    when 2 then ‘Large’
             end as parameter_type,
            scalar_value, vector_value, large_value
from mgmt_job_parameter
where job_id in ( select t1.job_id from sysman.mgmt$jobs t1 where t1.target_type is not null )
order by job_id asc;

Depending on the parameters and values of the parameters, you will have to look at a few different areas.  For the purpose of this post, we are focusing on the parameter type of large.   The PARAM_ID located in the LARGE_VALUE column will point us to the SQL/RMAN statement that was placed in the job when it was created.  The below query will pull the PARAM_VALUE, based on the LARGE_VALUE from previously SQL statement, from the MGMT_JOB_LARGE_PARAMS table.

select param_value
from mgmt_job_large_params
where param_id = ‘<values from LARGE_VALUE>’;

Once you have all the information, which has been outlined in this document, you should be able to reproduce your jobs from in the new OEM environment.

Moving jobs from one OEM environment to another is not as easy as I would have hoped.  I truly hope that Oracle listens to the end users and eventually get around to producing an export/import process for jobs.  Is an XML export/import process that difficult for jobs?

Collaborate 13 – my take

Last week I attended Collaborate 13 in Denver, Colorado.  It was a fun week; fun and busy, for the most part.  I finally had the chance to do a few things and meet people that I’ve wanted to meet.  The conference started on Sunday officially; however, knowing that the Rockies (baseball fan here…) were in town I took the opportunity to see them play the Saturday before.  It was a good game; what was even more fun was the company I had for the game.  The company that came to the game was a mix of locals and foreign friends of the Oracle community.  Like I said the game was fun but the after game drinks and dinner set the tempo for the week at Collaborate 13!

Sunday, it was time to get down to business and see what was going on and where.  When I arrived at the convention center it was not as packed as I had seen at Oracle Open World last September-October 2012.  After hanging around the convention center a bit and attending an all-day Sunday session, more people started to arrive.  At the end of the day, all sessions I attended were great and well worth the attendance.

Monday, I was so keyed up about my presentation that I rehearsed all morning before going to the conference center.  When my presentation time finally came, I was ready to present.  My presentation on security compliance went well.  After I completed my session, I had a few people come up and ask more questions about security compliance with OEM12c.  This was refreshing seeing others really trying to put what I had just presented to use.  Many in the audience were planning on taking it back to their offices and trying to implement what they had just learned.    Overall, I thought I did decent job on the presentation; looking forward to good reviews from the evaluations.

Tuesday, I had the privilege to participate on the Oracle Database Appliance (ODA) Panel.  We discussed with customers what the ODA was and what options and concerns there may be when migrating to ODA.  After the panel discussion I ran into a friend from a testing event with Oracle.  He asked if I was willing to do a third event on Thursday. After accepting, this only gave us a day to prepare.  Tuesday night, I was invited to a dinner with some members of the ODTUG board of directors.  The dinner was fun and provided a great opportunity to network and meet new people.  Can’t wait until June to see many of them again at KScope13 in New Orleans. 

Wednesday, I was excited about seeing some sessions in the morning since I would be prepping for the Thursday morning presentation.   Needless to say, I got to see the sessions I wanted to see that morning.  After walking through our dry-run of the presentation, it was time to prepare for the Collaborate party that night.  The party was fun and lead to many different interesting discussions around technology.  At the close of the party, I believe everyone had a unique view of what the conversation was about. 

Thursday, I had to be at the conference center before 7:30 am to prepare for our 8:30 am presentation (I had hoped to sleep in a bit).  Prior to presentation time, the topic had been tweeted about later Tuesday night and all day on Wednesday to try and get as many people as possible to show.  This topic was on the next-generation of the Oracle database and what everyone could expect to see (permission was granted from Oracle on Tuesday for this presentation).  The session went very well and everyone was excited to see what new features may be included when finally released.  After I was done with my session, I attended a co-workers presentation on compression with the Exadata and ODA.  For his first presentation, he did well and even had a surprise audience member during his session.  After his session it was about lunchtime; we sat around the lobby area for a bit talking with other Oracle specialists before heading to lunch. 

 If I had to summarize last week at Collaborate 13 on a personal level, I would say I gained a lot of knowledge, presented a lot of knowledge, and gained a whole new network of friends within the Oracle community.  If you have not attended a conference or been a speaker at a conference I would highly recommend that you look into doing so.  There is so much to gain from doing one or the other or both.

Install OEM Agents silently in any environment

 

Outlined in this post are the steps that I have used on many installs of Oracle Enterprise Manager 12c agents in windows environments.  I have found that this is the simplest way to install agents on a windows box. Yes this will take some time to do one-by-one; however, the process is much simpler than installing Cygwin.  Additionally, the process can be repeated very quickly from system-to-system by moving only the RSP and ZIP files needed for the install.

Silent Install of Oracle EMAgent:

1. On the OMS host move to the OMS_HOME/bin and login using the EMCLI command

cd /oracle/app/oracle/product/12.1/middleware/oms/bin

./emcli login –username=sysman –password=<sysman password>

2. Sync the repository using EMCLI

./emcli sync

Message returned should be “synchronized successfully”

3. List support platforms that are in the repository.

./emcli get_supported_platforms

4. Download the agent image to a temporary location on the OMS host.

./emctl get_agentimage –destination/mnt/nasbackup/oracle/12.1 –platform=”Linux x86-64” –version=12.1.0.2.0

5. Transfer the file 12.1.0.2.0_AgentCore_226.zip to the Repository host.

6. Unzip the 12.1.0.2.0_AgentCore_226.zip file to the agent_linux-x86-64

unzip ./12.1.0.2.0_AgentCore_226.zip -d ./agent_linux-x86-64

7. Edit the agent.rsp file for the specifics to needed during the install.  The file that should be used is the “agent_<name>.rsp” file.

The only thing that should change in the response file (agent_<name>.rsp) is the hostname where the agent is going to be installed.

8. Execute the deployment.

./agentDeploy.sh AGENT_BASE_DIR=/oracle/app/oracle/product/12.1/agent RESPONSE_FILE=<full path>/agent_<name>.rsp

9. Execute root.sh

  1. Exit Oracle (back to the user you logged in as, example: ssc+bia_bcurtis)
  2. Execute the root.sh script

$ /oracle/app/oracle/product/12.1/agent/core/12.1.0.2.0/root.sh

10. At this point, you will have to manually add non-host targets through the OEM console.

Agent running yet OEM shows unreachable/metric collection error…what gives?

One of the most confusing (and frustrating) things with Oracle Enterprise Manager is figuring out why agents are not uploading from time-to-time.  This issue was worse in previous versions; in OEM12c the uploading issues have been somewhat corrected and do not seem to be that big of an issue.  What I have found to be more of an issue with OEM12c is when the agent says it is unreachable within OEM12c.

An unreachable agent could be almost anything, a firewall blocking the required upload port, invalid DNS entries, and hostname configuration issues to network related issues.  In most of these cases, except for agent configuration issues, the DBA doesn’t have access to resolve these issues and require the assistance of different departments within the IT.  Sometimes these external resources are not available to help troubleshoot and resolve network issues; example, if you are trying to configure OEM12c at home as I have.

The problem I recently ran into while adding an agent to a local server without DNS; was that the agent would install and run; yet OMS couldn’t sync it due to metric collection errors.  The status of the agent after the install was up and running yet OEM still wouldn’t recognize it.  This prompted me to uninstall the agent and push the agent again.  From the OEM12c management server, I was able to push the agent and add the target successfully though the Add Host Targets wizard.  Although the host target is added successfully, the status in OEM12c still showed “Unreachable” due to Metric Collection Error.   Why is this target unreachable, I just added the agent and didn’t have any problems with the push form OEM12c?

I was perplexed to say the least; however, in searching MOS I came across a note that helped resolve this issue.  The note number for reference is: 1440682.1.  This note outlines similar symptoms to what I was having and provided a workable solution.  What I found interesting in this note was the fact that this issue is an unpublished bug.  The note also gives examples of what messages may be received from OEM via notification messages.

The incorrect message that may be received is (via notification emails):

Message=Agent is Unreachable (REASON = unable to connect to the agent at https://hostname.domain:3872/emd/main/ [Connection timed out]). Host is unreachable (REASON = Unknown Error pinging the host of URL https://hostname.domain:3872/emd/main/.1)

As outlined in the MOS note, the workaround for this issue is to check to see if the ping property is set.  If so, then it needs to be disabled to allow for the target host status to be changed.  The steps below will assist in resolving this issue:

On the OMS, check the property via emctl

./emctl get property –name oracle.sysman.core.omsAgentComm.ping.pingCommand

If it returns this status:

Emdrep.ping.pingCommand=%EM_PING_COMMAND%

This is the reason for the invalid agent status and metric collection error in OEM12c.

Disable the ping command:

./emctl delete property –name “oracle.sysman.core.omsAgentComm.ping.pingCommand”

With this property removed, the OMS will ping targets using an alternative successful method (getPingCmdForOS).

Now stop and restart the OMS:

./emctl stop oms

./emctl start oms

Lastly, ensure that the agent is started and that the status in OMS is saying up.  (may take a few minutes due to agent uploads)

Let me know what you thing about this resolution to this interesting problem.

Upcoming conferences…some work still to do!

Just around the corner are two conferences which I will be presenting at this year.  I would say I’m excited about these conferences; however, I’m more nervous than anything else.  Sometimes it seems like there is not enough hours in a day to get everything done.  I’ll be ready to speak on the days I’m assigned though!

The first conference for 2013 will be at IOUG Collaborate 13 in Denver, CO in April.  I’ll be speaking on how to setup and use Oracle Enterprise Manager 12c for Security Compliance.  The compliance framework of OEM12c often gets overlooked and many people don’t understand how it works, which is a disappointment.  The compliance framework within OEM12c is very flexible, helpful, and provides valuable feedback on the health of an enterprise infrastructure.    If you are interested in finding out more about compliance within OEM12c and will be in the Denver, CO, stop by and check out session 319. 

The second conference I’ll be at in 2013 will be ODTUG KScope 13 in New Orleans, LA in June.  This conference is going to be fun.  The topic I’ll be speaking on is related to the next generation of Oracle Database.  I cannot go into great detail here; however, with permission from Oracle I’ll be able to present some good information in June.  This is going to be a good session to attend if you are interested in what is coming next.  As of yesterday, my abstract and title has been submitted; as soon as I have a date and time of my session, I’ll post it here. 

Overall, I feel humble to be talking about these topics at such events.  I hope that everyone gets some valuable information from the presentations that are presented at these conferences.  If there is anything specific you would like to see in my presentations feel free to leave a message.

Screwball Host Names

Note: Hostnames in images have been blurred out on purpose.

There is always one thing you can count on when working with Oracle products; you will always learn something new.  Today, while I was working with OEM12c, I manually added a host with no issues.  I provided the host name, port number, directory for the agent install and passwords required.  When the agent install was complete, I checked out the target just to make sure everything was fine.  What I found out was that the host name target did not include the domain name.

This got me to thinking, why did it do this?  In previous versions, it was difficult to change the name of a host target. In OEM 12c, it appears not that easy as well.  So, how did I solve this problem?

Image

After digging around for a bit, I figured out that the host name when adding manually; the domain name (acme.com) needs to be provided with the hostname.  Before this can be fixed, the existing target needs to be removed.  I make my life easy, I’m opting for removing the target via removing the agent.

Enter EMCLI!

The easiest way to remove the target is to do the following:

  1. Stop the agent on the host that we want to change.

cd $AGENT_HOME/core/12.1.0.2.0/bin/emctl stop agent

2. Go to the OMS home and login to the OMS with emcli

$ cd $OMS_HOME/bin/emcli login –username=sysman –password=<password>

$ emcli sync

4. Remove the agent from the repository with monitored targets.

       $ emcli delete_target –name=”<agent/host>:3872” –type=”oracle_emd” –delete_monitored_targets

Removing the agent this way, will force all the monitored targets to be removed as well.  Once the agent and associated targets are removed emcli will indicate successful.

      Target “<agent/host>:3872:oracle_emd” deleted successfully

Back on the target server, remove the directory for the agent and clean up the inventory.xml file if you want to use the same directory structure for the new agent.

$ rm –rf ./12.1

Now, let’s re-add the agent and associated host target.

Just go through the normal manual process for adding a target host.  When providing the hostname, you’ll need to add the domain name as part of the host.

Image

Once the agent has been redeployed to the host target; it will come up in OEM listed with the domain name.

Image

I know that sometimes it is not idea to add an agent, remove the agent and then add it again.  If you are like me, I like to have clean environment to look at and understand.  Let me know what your thoughts are on this issue.