Blog Feed

“Failed to find the OPatch patch directory” while applying patch 25105555 ( 13.2.0.0.161231) from OEM console

While applying patch 25105555 ( 13.2.0.0.161231) to an agent using the “provisioning and Patching” console from OEM 13.2 and choosing the “Upgrade OPatch” option, got this error:
———- Error Message ———-
Error: 010
Failed to find the OPatch patch directory “/tmp/p6880880_600000000057471_2000_0/oracle/OPatch”.
If you chose “No” for the “StagePatches” option, ensure that you manually place the patch at the required location and retry the operation.
———– End Message ———–
Tue Aug 27 15:41:48 2017 – Patching aborted.
You have to follow the solution section from below MOS note:
EM 13c: Applying EM-AGENT Bundle Patch 25105555 ( 13.2.0.0.161231) from Enterprise
Manager 13.2 Cloud Control Fails with Message: Error: 010 Failed to find the OPatch patch directory (Doc ID 2229452.1)
It depends on whether you have an online or offline OMS configuration.
After doing this I was able to apply patch 25105555 to the agent.
Hope this helps.

Alfredo

RMAN restore slow due to ASMB process from ASM to filesystem

I was restoring a database running on ASM to an instance running on filesystem.
The step for cataloging files was extremely slow and after taking a look at the alertlog file, I noticed tons of errors related to ASMB process.
Well, turns out this is an unpublished bug explained in more detail on the below MOS note.
12c RMAN Operations from ASM To Non-ASM Slow Performance (Doc ID 2081537.1)
After applying the suggested patch (19503821) the performance went back to normal and the restore progressed as expected.
Thanks,

Alfredo

2017 April PSU – OEM sending tablespace related alerts every collection schedule

I noticed that several alerts related to tablespace full were sent from OEM every collection schedule. At first sight I thought this may be related to OEM itself but after spending some significant time in MOS found a bug note stating that the 2017 April PSU patch was responsible.



The issue is that dba_tablespace_usage_metrics is not accounting for autoextend datafiles. So even if you add a new datafile, your used space will continue be reported as it was before.
You need to apply patch 26198757 on top of April PSU.
Hope this help you to avoid all this noise and happy patching.
Thanks,

Alfredo

Oracle Management Cloud – Setup Entities in Log Analytics

In order to gather Weblogic’s Admin Server and Managed Server log files, you need first to install and configure the OMC Cloud Agent.  
Then you need to verify that the Cloud Agent is up and running.  Refer to my previous post on how to manage the Cloud Agent.
The next step is to setup the Log Analytics entities. In order to do this you have to make use of a JSON template. Oracle provides examples of these templates in the OMC documentation:
In this example we are going to add a Weblogic Admin Server and Managed server as Cloud Agent entities.
Unzip the example zip file and modify the omc_weblogic_domain_credless.json file.
The data added to this file really depends on your environment. This is how it looks for my environment:
{
   “entities”:[
      {
         “name”:”base_domain”,
         “type”:”omc_weblogic_domain”,
         “displayName”:”Weblogic_on_”,
         “timezoneRegion”:”PST”,
         “properties”:{
            “port”:{
               “displayName”:”Port”,
               “value”:”7001″
            },
            “domain_home”:{
               “displayName”:”Domain Home”,
               “value”:”/u01/app/oracle/product/ofm12.1.2/user_projects/domains/base_domain”
            },
            “admin_server_host”:{
               “displayName”:”AdminServer”,
               “value”:”.”
            },
            “webLogic_home”:{
               “displayName”:”WebLogic Home”,
               “value”:”/u01/app/oracle/product/ofm12.1.2/wlserver”
            }
         },
         “associations”:[
            {
               “assocType”:”omc_contains”,
               “sourceEntityName”:”base_domain”,
               “sourceEntityType”:”omc_weblogic_domain”,
               “destEntityName”:” __AdminServer”,
               “destEntityType”:”omc_weblogic_j2eeserver”
            }
         ]
      },
      {
         “name”:”_AdminServer”,
         “type”:”omc_weblogic_j2eeserver”,
         “displayName”:”AdminServer_”,
         “timezoneRegion”:”PST”,
         “properties”:{
            “host_name”:{
               “displayName”:”Weblogic Host”,
               “value”:”.”
            },
            “domain_home”:{
               “displayName”:”Domain Home”,
               “value”:”/u01/app/oracle/product/ofm12.1.2/user_projects/domains/base_domain”
            },
            “listen_port”:{
               “displayName”:”Listen Port”,
               “value”:”7001″
            },
            “listen_port_enabled”:{
               “displayName”:”Listen Port Enabled”,
               “value”:”true”
            },
            “ssl_listen_port”:{
               “displayName”:”SSL Listen Port”,
               “value”:”7002″
            },
            “server_names”:{
               “displayName”:”Server Names”,
               “value”:”AdminServer”
            }
         },
         “associations”:[
            {
               “assocType”:”omc_monitored_by”,
               “sourceEntityName”:”_AdminServer”,
               “sourceEntityType”:”omc_weblogic_j2eeserver”,
               “destEntityName”:”base_domain”,
               “destEntityType”:”omc_weblogic_domain”
            }
         ]
      },
      {
         “name”:”_ManagedServer_1″,
         “type”:”omc_weblogic_j2eeserver”,
         “displayName”:”ManagedServer_1_”,
         “timezoneRegion”:”PST”,
         “properties”:{
            “host_name”:{
               “displayName”:”Weblogic Host”,
               “value”:”.”
            },
            “domain_home”:{
               “displayName”:”Domain Home”,
               “value”:”/u01/app/oracle/product/ofm12.1.2/user_projects/domains/base_domain”
            },
            “listen_port”:{
               “displayName”:”Listen Port”,
               “value”:”7003″
            },
            “listen_port_enabled”:{
               “displayName”:”Listen Port Enabled”,
               “value”:”true”
            },
            “ssl_listen_port”:{
               “displayName”:”SSL Listen Port”,
               “value”:”7005″
            },
            “server_names”:{
               “displayName”:”Server Names”,
               “value”:”ManagedServer_1″
            }
         },
         “associations”:[
            {
               “assocType”:”omc_monitored_by”,
               “sourceEntityName”:”_ManagedServer_1″,
               “sourceEntityType”:”omc_weblogic_j2eeserver”,
               “destEntityName”:”base_domain”,
               “destEntityType”:”omc_weblogic_domain”
            }
         ]
      }
   ]
}
Save this file and load the entities to the Cloud Agent as follows:
$ ./omcli add_entity agent /home/oracle/omc_weblogic.json
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Operation Succeeded: Accepted 3 of 3 entities for processing.
As you can see, 3 entities were loaded to the agent; the base_domain, Admin Server and Managed Server.
Now verify the new entities in the Log Analytics configuration page.
–        Click on the “Navigation” icon.
–        Under “Cloud Services” click on “Log Analytics”
Click on “Configuration” located in the right side of the screen



In the “Entities” section. Click on the icon with the number (ie. 4)

You will see a list of the new entities just added.
Go back to “Log Explorer” by clicking on





In the search box type ‘Log Source’ = ‘FMW WLS Server Logs’


Click on “Run”


Check all the log information displayed for the WLS Server

This blog post is part of a series of blog posts related to the Oracle Management Cloud:
·                     What is the Oracle Management Cloud?
·                     Types of Oracle APM Agents
·                     Oracle Management Cloud – Cloud Agent Install
·                     Logging into Oracle APM Cloud Service for the first time
·                     Overview of Oracle APM Cloud Service features


Thanks,
Alfredo

Oracle Management Cloud – Managing the Oracle Cloud Agent

This post is about some useful commands to administer the OMC’s Cloud agent. The very first step is to install the OMC Cloud agent. You may refer to my previous post on how to install it here.
The omcli command is located at /agent_inst/bin.
I’m skipping these 3 basic commands start, stop and status.
Here we go!
      List the targets currently managed by the Cloud agent
$ ./omcli config agent listtargets
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
[hostname.localdomain:1302, Lama]
[hostname.localdomain, omc_host_linux]
      Get the TimeZone of the server where the Cloud agent is running
$ ./omcli config agent getTZ
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
US/Eastern
      Get a detailed output of the Cloud agent scheduler
$ ./omcli status agent scheduler
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Number of currently running scheduled items :: 0
Number of currently ready scheduled items :: 0
Number of scheduled items :: 57
Schedule Attempts Delay Percentage :: 0.000%
Running entries::
Ready entries::
Scheduled entries::
2017-03-31 14:32:58.968 : HeapMonitorTask
2017-03-31 22:34:14.667 : PurgeTargetEvents
2017-03-31 22:35:47.885 : PurgeADR
2017-04-01 13:34:20.810 : EventStatusPurge
—————————————————————
Agent is Running and Ready
      Get a summary of the Cloud agent’s scheduler
$ ./omcli status agent scheduler -summary
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Number of currently running scheduled items :: 0
Number of currently ready scheduled items :: 0
Number of scheduled items :: 57
Schedule Attempts Delay Percentage :: 0.000%
—————————————————————
Agent is Running and Ready
      Dump the internal state of the agent [option component name]
$ ./omcli dumpstate agent
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Generated file: /u01/omc/infra/agent_inst/sysman/emd/dumps/SystemDump_1490985620656.xml
      Dump internal information of all components of the Cloud agent
$ ./omcli dumpstate agent -dump_full_state
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Generated file: /u01/omc/infra/agent_inst/sysman/emd/dumps/SystemDump_1490985775823.xml
      Add entities to the Cloud agent using JSON template
$ ./omcli add_entity agent /home/oracle/omc_weblogic.json
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Operation Succeeded: Accepted 2 of 2 entities for processing.
      Delete entities from the Cloud agent using JSON template
$ ./omcli delete_entity agent /home/oracle/omc_weblogic.json
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Request for removing the entities was accepted for processing.
This is just a small taste of all the options built into the Oracle’s Cloud agent.  For more information about all the options refer to Oracle’s documentation.
Thanks,

Alfredo

Oracle Management Cloud – Install the Cloud Agent

This is my first experience with the Oracle Cloud, in particular with Oracle Management Cloud (OMC). In this first post I will guide you through the installation of the Cloud Agent in a Linux host.
First you have to login to OMC and click on the navigation icon below.


It’ going to open the navigation screen that contains 4 sections; Home, Cloud Services, Explorers and Administration.


In the Administration section click on Agents.
In this page we are looking into 2 important items for us. Registration Keys and Download.


The way I see the Registration Keys is like the Agent Registration password in OEM 13c. You need a valid Registration Key to deploy your Cloud Agent. So, click on that link and verify that you have a Registration Key; if not, you can create one.   
Then proceed to download the agent by clicking on the Download link.
In order to install the Cloud Agent in the Linux host there are some pre-reqs to be met.
First verify that you have acl and libaio.i686 packages installed.
# rpm -q acl
acl-2.2.49-6.el6.x86_64
# rpm -qa|grep libaio
libaio-devel-0.3.107-10.el6.x86_64
libaio-0.3.107-10.el6.x86_64
libaio-0.3.107-10.el6.i686
Then move the AgentInstall.zip previously downloaded into an staging directory and unzip it.
$ mkdir –p /home/oracle/omc
$ mv AgentInstall.zip /home/oracle/omc
$ cd /home/oracle/omc
$ unzip AgentInstall.zip
$ $ ls
AgentInstall.bat  AgentInstall.sh  AgentInstall.zip  README
Create a folder to install the Cloud Agent.
$ mkdir –p /u01/omc/infra
Verify that the server has the FQN configured (name.domain_name).
$ hostname
ol6-121-plug.localdomain
Finally install the Cloud Agent.
$ ./AgentInstall.sh AGENT_TYPE=cloud_agent AGENT_REGISTRATION_KEY=R******************1  AGENT_BASE_DIR=/u01/omc/infra AGENT_PORT=1302 ORACLE_HOSTNAME=ol6-121-plug.localdomain
WARNING: The current cURL version 7.19.7 does not support TLS1.2 protocol; the downloader script will attempt to continue but may not work. Please install 7.49.1 or later versions of cURL for error-free execution.
Downloading lama agent software …
Generating emaas.properties …
Extracting Agent Software …
Installing the Agent …
Registering the Agent …
Downloading Certificates …
Configuring the Agent …
Cleanup temporary files …
Then you have to run the root script required after the installation.
# /u01/omc/infra/core/1.15.0/root.sh
/etc exist
/u01/omc/infra/core/1.15.0
Check the status of the Cloud Agent.
$ ./omcli status agent
Oracle Management Cloud Agent 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Version                : 1.15.0
State Home             : /u01/omc/infra/agent_inst
Log Directory          : /u01/omc/infra/agent_inst/sysman/log
Binaries Location      : /u01/omc/infra/core/1.15.0
Process ID             : 9828
Parent Process ID      : 9790
URL                    : https://ol6-121-plug.localdomain:1302/emd/main/
Started at             : 2017-03-20 20:49:15
Started by user        : oracle
Operating System       : Linux version 3.8.13-26.2.2.el6uek.x86_64 (amd64)
Data Collector enabled : false
Sender Status          : FUNCTIONAL
Gateway Upload Status  : FUNCTIONAL
Last successful upload : 2017-03-20 21:04:39
Last attempted upload  : 2017-03-20 21:04:38
Pending Files (MB)     : 0.01
Pending Files          : 14
Backoff Expiration     : (none)
—————————————————————
Agent is Running and Ready
Now if you go to the OMC portal, the agent will show up automatically.



Want to learn more about Oracle Management Cloud? There’s a free hands on lab “Cloud Experience” at Collaborate 2017 for all attendees on Wednesday 4/5/17.
Thanks,

Alfredo

OMS responded illegally [ERROR- Failed to Update Target Type Metadata] in 13c R2

I want to discuss about this OMS error coming from the agent’s heartbeat status. The agent shows with an unreachable status but is actually up and running.
$: emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 13.2.0.0.0
OMS Version            : (unknown)
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 2,210
Size of XML files pending upload(MB)         : 2.89
Available disk space on upload filesystem    : 56.39%
Collection Status                            : Collections enabled
Heartbeat Status       : OMS responded illegally [ERROR- Failed to Update Target Type Metadata]
Last attempted heartbeat to OMS              : 2017-01-24 21:09:21
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2017-01-24 21:09:51
—————————————————————
Agent is Running and Ready
In this case the agent is not able to upload the metadata because is different in the OMS.
This is mainly due to a patch in the agent side that is missing in the OMS. By mistake I applied the latest bundle patch to the agent before applying it to the OMS.
Just another proof that any patches have to be installed to the OMS first then the agents.
Thanks,

Alfredo

Upgrade OPatch for OEM 13c R2 Agents

Opatch utility has to be updated often as Oracle delivers regular updates to it. I was used to version 12 and it only required an overwrite of the OPatch directory. Well, not anymore with version 13c R2.
I will walk you through an OPatch upgrade from 13.8 to 13.9 version.
First export your agent ORACLE_HOME:
$: export ORACLE_HOME=/u01/oracle/oemagent/agent_13.2.0.0.0/
Verify the current OPatch version:
$: $ORACLE_HOME/OPatch/opatch version
OPatch Version: 13.8.0.0.0
OPatch succeeded.
Download and unzip the 13.9 version in a temporary directory:
$: unzip p6880880_139000_Generic.zip
Archive:  p6880880_139000_Generic.zip
   creating: 6880880/
  inflating: 6880880/README.txt
  inflating: 6880880/opatch_generic.jar
  inflating: 6880880/version.txt
As you can see, now we have some zip files and a java file. Execute java to upgrade OPatch as follows:
$: $ORACLE_HOME/jdk/bin/java -jar opatch_generic.jar -silent oracle_home=$ORACLE_HOME
Launcher log file is /tmp/OraInstall2017-01-19_08-47-21PM/launcher2017-01-19_08-47-21PM.log.
Extracting the installer . . . . Done
Checking if CPU speed is above 300 MHz.   Actual 3000.000 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 16383 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Checking temp space: must be greater than 300 MB.   Actual 18842 MB    Passed
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2017-01-19_08-47-21PM
Installation Summary
Disk Space : Required 6 MB, Available 5,229 MB
Feature Sets to Install:
        Next Generation Install Core 13.9.1.0.0
        OPatch 13.9.1.0.0
        OPatch Auto OPlan 13.9.1.0.0
Session log file is /tmp/OraInstall2017-01-19_08-47-21PM/install2017-01-19_08-47-21PM.log
Loading products list. Please wait.
 1%
 40%
Loading products. Please wait.
 43%
 …
 97%
 99%
Updating Libraries
Starting Installations
 1%
 …
 91%
 92%
Install pending
Installation in progress
 Component : oracle.swd.opatch 13.9.1.0.0
Copying files for ‘oracle.swd.opatch 13.9.1.0.0 ‘
 Component : oracle.glcm.osys.core 13.9.1.0.0
Copying files for ‘oracle.glcm.osys.core 13.9.1.0.0 ‘
 Component : oracle.glcm.oplan.core 13.9.1.0.0
Copying files for ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘
 Component : oracle.glcm.opatch.common.api 13.9.1.0.0
Copying files for ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘
 Component : oracle.glcm.opatchauto.core 13.9.1.0.0
Copying files for ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘
Install successful
Post feature install pending
Post Feature installing
 Feature Set : apache_commons_cli_lib
 Feature Set : oracle.glcm.opatchauto.core.actions.classpath
Post Feature installing ‘apache_commons_cli_lib’
 Feature Set : oracle.glcm.opatchauto.core.binary.classpath
 Feature Set : oracle.glcm.osys.core.classpath
Post Feature installing ‘oracle.glcm.opatchauto.core.binary.classpath’
Post Feature installing ‘oracle.glcm.osys.core.classpath’
 Feature Set : apache_commons_compress_lib
 Feature Set : oracle.glcm.opatchauto.core.wallet.classpath
Post Feature installing ‘oracle.glcm.opatchauto.core.actions.classpath’
Post Feature installing ‘apache_commons_compress_lib’
 Feature Set : oracle.glcm.opatchauto.core.classpath
Post Feature installing ‘oracle.glcm.opatchauto.core.wallet.classpath’
Post Feature installing ‘oracle.glcm.opatchauto.core.classpath’
 Feature Set : oracle.glcm.opatch.common.api.classpath
 Feature Set : oracle.glcm.oplan.core.classpath
Post Feature installing ‘oracle.glcm.opatch.common.api.classpath’
Post Feature installing ‘oracle.glcm.oplan.core.classpath’
Post feature install complete
String substitutions pending
String substituting
 Component : oracle.swd.opatch 13.9.1.0.0
String substituting ‘oracle.swd.opatch 13.9.1.0.0 ‘
 Component : oracle.glcm.osys.core 13.9.1.0.0
String substituting ‘oracle.glcm.osys.core 13.9.1.0.0 ‘
 Component : oracle.glcm.oplan.core 13.9.1.0.0
String substituting ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘
 Component : oracle.glcm.opatch.common.api 13.9.1.0.0
String substituting ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘
 Component : oracle.glcm.opatchauto.core 13.9.1.0.0
String substituting ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘
String substitutions complete
Link pending
Linking in progress
 Component : oracle.swd.opatch 13.9.1.0.0
Linking ‘oracle.swd.opatch 13.9.1.0.0 ‘
 Component : oracle.glcm.osys.core 13.9.1.0.0
Linking ‘oracle.glcm.osys.core 13.9.1.0.0 ‘
 Component : oracle.glcm.oplan.core 13.9.1.0.0
Linking ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘
 Component : oracle.glcm.opatch.common.api 13.9.1.0.0
Linking ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘
 Component : oracle.glcm.opatchauto.core 13.9.1.0.0
Linking ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘
Linking in progress
Link successful
Setup pending
Setup in progress
 Component : oracle.swd.opatch 13.9.1.0.0
Setting up ‘oracle.swd.opatch 13.9.1.0.0 ‘
 Component : oracle.glcm.osys.core 13.9.1.0.0
Setting up ‘oracle.glcm.osys.core 13.9.1.0.0 ‘
 Component : oracle.glcm.oplan.core 13.9.1.0.0
Setting up ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘
 Component : oracle.glcm.opatch.common.api 13.9.1.0.0
Setting up ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘
 Component : oracle.glcm.opatchauto.core 13.9.1.0.0
Setting up ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘
Setup successful
Save inventory pending
Saving inventory
 93%
Saving inventory complete
 94%
Configuration complete
Logs successfully copied to /u01/app/oraInventory/logs.
Verify the new OPatch version

$: $ORACLE_HOME/OPatch/opatch version
OPatch Version: 13.9.1.0.0
OPatch succeeded. 
Thanks,

Alfredo

ORA-01503: CREATE CONTROLFILE failed on RMAN duplicate between RAC databases

It was while since I performed an RMAN duplicate from a RAC database to another RAC database. You may find this post useless as there’s a lot of information out there about the error I faced, but all my posts are kind of my own little diary.
So, basically. In order to successfully duplicate one database to another you may need this.

    – TNS setup on the server where the database to be duplicated is hosted. You need to be able to tnsping both, the target and the auxiliary database.

     – Create a parameter file and password file for the auxiliary and startup nomount the first instance.
    – Create a full backup of the target database including archivelogs.
    – Connect to rman to both the target and the auxiliary databases from the auxiliary server:
rman target sys/*****@ auxiliary /
    – Issue the duplicate command:

 duplicate target database to ;
This should successfully finish but I got this error:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 01/03/2017 18:29:58
RMAN-05501: aborting duplication of target database
RMAN-06136: ORACLE error from auxiliary database: ORA-01503: CREATE CONTROLFILE failed
ORA-12720: operation requires database is in EXCLUSIVE mode
I searched this and the culprit is that my parameter file for the auxiliary had this:

cluster_database = true

By MOS note “Rman duplicate fail with RMAN-06136, ORA-01503, ORA-12720, ORA-00494 enqueue [CF] (Doc ID 1335479.1)”. Duplicate from a RAC database to another RAC database is not supported, hence we need to modify the parameter cluster_database to false in the auxiliary database.
After doing this the duplicate finished correctly.
Thanks,

Alfredo