Thursday, November 29, 2018

EBS ADOP Phase Failing Due to ORA-600 Error

There’s an issue in EBS 12.2 and database versions related to Materialized Views. While executing ADOP (prepare, apply, finalize or cutover) phases, the cycle could fail due to an ORA-600 : internal error code, arguments: [kqllod:no stub for dependency parent] error.

After working with Oracle, they provided information about BUG 27883586 related to MView Refresh or Create while they are editioned.

The solution is to apply patch 27883586 to the database which is currently available for August 2017 release and then recreate the MViews. Oracle also released a merge patch on top if this (28820125).

If you need it for a different release, please contact Oracle to provide it.

Happy patching,

Friday, October 26, 2018

OEM 13c Reporting Incorrect Tablespace Space Usage (%) After July 2018 PSU

The topic for today is Tablespace monitoring. OEM 13c started reporting inaccurate values for the Space Usage (%) metric after applying either the April 2018 or the July 2018 Database PSU's. The way this was detected, is that OEM 13c Corrective Actions started firing here and there after the patches.

Oracle confirmed this is tracked on BUG 26198757.
This patch is available on top of both the April 2018 and the October 2018 PSU's but is not available for October 2018 PSU. The issue could also manifest itself as huge wait times on "Sync ASM rebalance" event.

Looks like the issue is corrected after applying the patch on top of July 2018 PSU. Below are some MOS realted to this.

Bug 26198757 - dba_tablespace_usage_metrics.used_percent is incorrect after applying 25397136 (Doc ID 26198757.8)

EM13c Space Monitoring Query (sqlid=69p6my4hpdm3j) On dba_tablespace_usage_metrics Timed Out after Db Upgraded To 12.2 (Doc ID 2375714.1)

EM 13c, Enterprise Manager Cloud Control Tablespace Space Used (%) Metric Incorrectly Triggering (Doc ID 2313520.1)

DBA_TABLESPACE_USAGE_METRICS Returns Incorrect Information After applying (25397136) Bundle patch (Doc ID 2289448.1)

Thanks and happy patching,

Sunday, September 30, 2018

RMAN Recovery Catalog Issues after 12.2 Upgrade

Well, after the upgrade of the DB to 12.2 the next step is to upgrade the RMAN Recovery Catalog to the same version in order to be able to run backups using it. The process is pretty straight forward.
.    Connect to RMAN using target and catalog
.    Execute upgrade catalog

RMAN> upgrade catalog;

This finishes correctly. But then we try to resync the catalog we got below error:

RMAN> resync catalog;

starting full resync of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 06/07/2018 22:15:36
ORA-01403: no data found

This is due to a bug 27013146. The solution is to apply this patch to the newly upgraded Oracle Home and re-run the upgrade catalog again.

More information can be found in MOS Note 2341947.1.

Everything is cool now, right? Well, now the old database backups are failing with below error:

RMAN-03008: error while performing automatic resync of recovery catalog

For that, the solution is outlined in MOS Doc Id 2291791.1.

Happy 12.2 upgrades!


Friday, August 31, 2018

Exadata/Exacloud CellServer health using AWR report

Exadata’s unique sauce relies on the fact that the DB instance can offload work to the CellServers to speed up I/O bound queries. But how do you monitor the health of the CellServers?

A quick and dirty way, especially if you don’t have access to the CellServers is to use AWR in a database that’s currently using those CellServers. Let’s get an AWR report and search for the SMART IO section. It should look like this.

What you want to see here is that % Total MB Requested is evenly distributed across all Cells. The same applies for Storage Index and Flash Cache metrics.

Now this is the interesting part. If you happen to see the Offload % Efficiency dropping for one of the cells, that means that the Cell is sending the complete blocks back to the DB and then the % Passthru metric is going to increase.

You don’t really want the % Passthru to increase as this will have a direct performance is the DBs being served by this Cell.

What are the reasons of this Passthru increasing?

There are 3 reasons for Passthru.

The first one is due to an issue with the Cell itself. The second one is due to issues with DST patches and the third one is related to user queries crashing the Cell.

You should look at the below MOS note in order to gather all the required information and open an SR immediately after you see this behavior.

SRDC - Exadata: Smart Scan Not Working Issues (Doc ID 2310422.1)


Thursday, August 9, 2018

Exalogic Agents being automatically deleted from OEM 13c

Oracle Enterprise Manager (OEM) 13c is capable of monitoring virtual guest hosts running on Exalogic (virtual configuration). OEM will connect (read-only) to the Oracle VM Manager and get the configuration and status of the VM’s running there. Then you can deploy an agent to those VM hosts and you have a complete monitoring solution for Exalogic.

The problem comes when for whatever reason you have to drop those VM’s from Exalogic but you want to keep the OEM configuration in place. For example, you want to drop the VM from Exalogic and then restore it from backup.

In this case you want to keep your monitoring configuration (metrics, credentials, etc.), but you realize that the agent was removed (along with all targets) as soon as you drop the VM from Exalogic.

This setup is being driven by metric like configuration. This synchronization is happening every minute by default.  This is covered on below Oracle’s document:

The solution was to set this synchronization schedule from every 1 minute to every 1 day.


Thursday, May 31, 2018

Manage Your Oracle Cloud Database - SQL Developer

In my previous post (How To Connect To The Oracle Cloud Instance) we reviewed the steps to connect using SQL Developer. Now we will see what DBA tasks we can perform using SQL Developer.

First click on the View menu, then click on the DBA sub menu. This is going to open the DBA pane in the lower left corner.

Click on Connections and add the connection we already setup to the Oracle Cloud Instance. 

Here you can expand the tree to verify what is available from SQL Developer.

Let's click on Tuning and then Real Time SQL Monitoring. The right pane is going to display the SQL Monitoring output. I'm really familiar to this feature in OEM 13c and honestly, looks very similar in SQL Developer.

Now let's click on Instance Viewer under Database Status menu.
This is going to show you the overall container status and statistics on where our PDB is running on.

The storage menu can help you verify the size of your tablespaces and under RMAN Backup/Recovery you can verify that there are actually backups happening every night. 


Wednesday, May 30, 2018

How To Connect To The Oracle Cloud DB Instance?

In my previous post (How To Create An Oracle Cloud Instance) I showed you create your first Oracle Cloud DB instance. Now that the instance is there the question is, how to connect to it?

Let's say you want to connect using SQL Developer to start creating objects and inserting data. First you need to enable the Client Access feature. Yes, it is disabled by default. 

Navigate to the Cloud DB instance dashboard. Click on Manage menu on the left side. 
Click on Admin Password and provide a password for the PDB_ADMIN account.

Then click on Client Access and check the enabled flag, then save.

It will show like this.

Once this is enabled you need to download the Access Credentials in order to connect to your instance.

Click on the Client Credentials button located on the right side of the Client Access one. This is going to prompt for a password (type one that you can remember) and it will download a .zip file containing these credentials and connect descriptors.

Note: You don't have to unzip this file.

Open your SQL Developer software and click on New Connection.

Type a name for your connection. In the username & password you can either type the PDB_ADMIN username and password or the credentials for the newly created schema.

In Connection Type select Cloud PDB and provide the downloaded .zip file along with the password provided (the one that you should remember).  

It should look like this.

Click Save and then Connect.

Now let's test our connection by executing a simple SQL statement.

Using the Cloud DB instance dashboard you can create different schemas as needed.
You just provide the username and password.

At the end you just need to follow the same steps as we did before, just provide the username and password of the new schema.

More information can be found at: