Adventures in RAC: gc buffer busy acquire and release

It seems that I’m getting more and more drawn into the world of performance analysis, and since I sometimes tend to forget things I need to write them down. I almost enjoy the “getting there” more than ultimately solving the problem. You pick up quite a few things on the way.

This environment is Exadata but as with so many things the fact that the database (more...)


Purpose is to demonstrate how to create HA for Bi-Directional Replication Goldengate installed on ACFS with RAC cluster using XAG.

XAG simplifies the process since there are no requirements to create action scripts.

Please review REFERENCE section for versions used in test case and versions requirements.

Goldengate is installed on ACFS for simplicity; otherwise, at a minimum the following directories br, dirchk, dirdat, dirtmp will need to be on shared storage with symbolic links if (more...)

RAC and preventing Active Data Guard: My experiences

When you happen to have a customer that want’s to use Data Guard on Oracle RAC without a license for Active Data Guard then you might want to read this post.

You probably have searched on the internet (just like I did) and already found this nice post from Uwe:

If you haven’t already then go ahead and read it, but come back when you also use RAC.

I had set-up Data Guard between (more...)

Controlling RMAN channels on RAC

This was sent to me for posting  by my friend and ex-colleague Vitaly Kaminsky ….

I have recently worked with the customer where standard RMAN backups of production 2-node cluster ( were getting too big and taking longer than 24 hours to run.

The problem with this particular cluster was the fact that ALLOCATION of RMAN connections to the instances of the cluster was controlled by SCAN and driven by the load-balancing (more...)

Using runInstaller to check Prereqs with responseFile

The SILENT method

$ ./runInstaller -silent -executePrereqs -showProgress -waitforcompletion -force -responseFile /media/sf_Linux/
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 48807 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 8191 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-11-30_06-40-05PM. Please wait ...[grid@rac01:/media/sf_Linux/]
$ cd /u01/app/oraInventory/logs/
$ ls -lrt
total 84
-rw-r--r--. 1  (more...)

Avoid false errors with for VirtualBox

My typical VirtualBox network configuration is: eth0 (NAT), eth1 (Host Only), eth2(Internal) stage -pre crsinst -n rac01,rac02 -r 11gR2 -fixup -fixupdir /tmp – FAILED

Node connectivity passed for subnet "" with node(s) rac02,rac01

PRVF-7617 : Node connectivity between "rac01 :" and "rac02 :" failed
TCP connectivity check failed for subnet ""

Node connectivity passed for subnet "192.168. (more...)

New leafmisscount Option in Flex Cluster 12c

In the Oracle RAC previous version we have worked with the option “misscount” of crsctl, which applies for all the servers because there was only one kind of servers, but in the new version of Oracle (12c) we have a new kind of RAC configuration, the “Flex Cluster”. With Flex Cluster we have two kinds of servers “Hub nodes” and “Leaf Nodes”, the Hub Nodes have the same concept of the normal nodes in the previous versions, however (more...)

The Importance of the In-Memory DUPLICATE Clause for a RAC System

With the INMEMORY clause you can specify 4 sub-clauses:

  • The MEMCOMPRESS clause specifies whether and how compression is used
  • The PRIORITY clause specifies the priority (“order”) in which the segments are loaded when the IMCS is populated
  • The DISTRIBUTE clause specifies how data is distributed across RAC instances
  • The DUPLICATE clause specifies whether and how data is duplicated across RAC instances

The aim of this post is not to describe these attribues in detail. Instead, (more...)

How to Find Blocking Sessions in Enterprise Manager Cloud Control 12c

My friend Franck Pachot noticed that EM12c doesn’t show the blocking sessions across all RAC nodes. Let’s check if we can find them using EM12c. I have connected to first node of our RAC database, created a sample table, inserted a row, committed, and then deleted the row. While the session is open, I opened another terminal, connected to second node and tried to delete the rows in same table so my second session started (more...)

Extending an ACFS filesystem dynamically.

To extend an ACFS cluster filesystem dynamically, we need to use the acfsutil command:

node01:/u01/grid>/sbin/acfsutil size +10G /u02
acfsutil size: ACFS-03008: The volume could not be resized.  The volume expansion limit has been reached.
acfsutil size: ACFS-03216: The ADVM compatibility attribute for the diskgroup was below the required
                           version ( for unlimited volume expansions.

Oh dear, not, so you can only extend volumes dynamically a (more...)

OOW14 Session: RAC’fying Multitenant

Thank you for attending my session RAC'fying Multitenant at Oracle Open World 2014. You can download the slide deck here.

As always I would love to hear from you.

OOW14 Session: Cache Fusion Demystified

Thank you all those who attended my session Cache Fusion Demystified.

As I mentioned, it was impossible to fit in a demo within that compressed 45 minute timeline. I put the details of the demo in a separate file with all the scripts and instructions that allows you to execute the demos at your own site.Download the slide deck as well as the scripts. In addition I have also written a whitepaper to explain (more...)

Grid Infrastructure Disk Space Problem – CHM DB file: crfclust.bdb

The Grid Infrastructure filesystem was reporting that it was a bit full today (release This was tracked down to the “crfclust.bdb” file, which records information about the cluster health for monitoring purposes. It was 26GB. It’s not supposed to get bigger than 1GB so this is probably a bug, but let’s explicitly resolve the size issue right now and search Oracle support later. Worst case, bdb (Berkerley Database) files get (more...)

Rolling Out-of-Place patching para Oracle Grid Infrastructure 12c y 11gR2: clonación en esteroides

Mi artículo más reciente para ToadWorld, dentro de una serie dedicada a aplicar patches con mínima suspensión del servicio. Ahora detallando cómo aplicar patches a Grid Infrastructure usando un Golden Image.

Convert 12cR1 non-RAC database to RAC database using rconfig

I have recently configured Active/Passive Failover cluster using Grid Infrastructure 12cR1 and wrote document on that subject. Now I plan to use this environment and convert "test12" database from Single-Instance to RAC database.

There are several ways to convert non-RAC database to RAC database:
- using Database Configuration Assistant (DBCA)
- Oracle Enterprise Manager
- rconfig command line utility
- manually

In this post I will describe how to perform conversion using rconfig utility.

Before (more...)

Database Shutdown With crsctl

I noticed something for the first time tonight when I was playing around in the Enkitec lab – something that I have been doing wrong for a while.  When working in the lab, I often rely on the crsctl command to shut down the entire cluster stack for me.  It’s really easy to use “crsctl stop cluster -all” followed by “dcli -l root -g ~/dbs_group /u01/app/ stop crs” to get everything down (more...)

Clusterware 12c and Restricted Service Registration for RAC

Topic: This post is about exploring the mechanisms used by Oracle Clusterware to restrict remote service registration, i.e. the 12c new feature "Restricting Service Registration for Oracle RAC Deployments"

Why is this useful? This improvement of 12c clusterware and listeners over the 11.2 version is useful mainly for security purposes, for example as a measure against TNS poisoning attacks (see also CVE-2012-1675), and it is particularly relevant for (more...)

Oracle GI 12.1 error when using NFS

I had quite an interesting case recently where I had to build stretch cluster for a customer using Oracle GI 12.1 and placing quorum voting disk on NFS. There is a document at OTN regarding the stretch clusters and using NFS as a third location for voting disk but it has information for 11.2 only as of the moment. Assuming there is no difference in the NFS parameters I used the Linux parameters (more...)

12c Improvement in Database Service Management

Removing a service via srvctl has not historically resulted in the service being fully removed from the database and it would still be visible in DBA_SERVICES as show below in an database:

Create the service:

$ srvctl add service -d orcl -s demo -r "ORCL1,ORCL2"

Have (more...)

Oracle SOA Suite 11g tuning tips for Oracle RAC database 11.2

During my project work I had to tune the SOA dehydration store on more than one occasion. Through this posting I would like to share the tuning tips collected during these exercises. It is not a step-by-step guide, because with tuning there never is a 'one-size-fits-all' in my humble opinion, but I provide in general a guidance that you can use as a reference for your own situation. I will also refer to related Oracle (more...)