Monitoring Oracle Database using Prometheus

Prometheus is a very popular framework for gathering metrics from a plethora of runtime components, recording, analyzing, visualizing them and making them available for companion technologies such as Grafana. Prometheus harvests information from ‘exporters’ – processes that expose an HTTP endpoint where Prometheus can scrape metrics in a format that it understands.

Recently, I received an email with the following question:

I need help on monitoring Oracle Database using Prometheus. I Googled but not find (more...)

Check Cluster Resources Where Target != State

Current version.

[oracle@racnode-dc2-1 patch]$ cat /etc/oratab
#Backup file is  /u01/app/12.2.0.1/grid/srvm/admin/oratab.bak.racnode-dc2-1 line added by Agent
-MGMTDB:/u01/app/12.2.0.1/grid:N
hawk1:/u01/app/oracle/12.2.0.1/db1:N
+ASM1:/u01/app/12.2.0.1/grid:N          # line added by Agent
[oracle@racnode-dc2-1 patch]$

Kill database instance process.

[oracle@racnode-dc2-1 patch]$ ps -ef|grep pmon
oracle   13542     1  0 16:09 ?        00:00:00 asm_pmon_+ASM1
oracle   27663     1  0 16:39 ?        00:00:00 ora_pmon_hawk1
oracle   29401 18930  0 16:40 pts/0    00:00:00 grep --color=auto pmon
[oracle@racnode-dc2-1 patch]$
[oracle@racnode-dc2-1 patch]$  (more...)

RAC Installation Logs

Note to self for 2 Nodes RAC installation and DB creation logs location.

Oracle Universal Installer logs for GI/DB:

[oracle@racnode-dc1-1 logs]$ pwd; ls -lhrt
/u01/app/oraInventory/logs
total 2.3M
-rw-r----- 1 oracle oinstall    0 Jun  7 16:39 oraInstall2019-06-07_04-39-01PM.err
-rw-r----- 1 oracle oinstall    0 Jun  7 16:43 oraInstall2019-06-07_04-39-01PM.err.racnode-dc1-2
-rw-r----- 1 oracle oinstall  121 Jun  7 16:43 oraInstall2019-06-07_04-39-01PM.out.racnode-dc1-2
-rw-r----- 1 oracle oinstall  11K Jun  7 16:43 AttachHome2019-06-07_04-39-01PM.log.racnode-dc1-2
-rw-r----- 1 oracle oinstall   (more...)

Data Guard Unexpected Lag

facepalmWhen configuring a physical standby database for Oracle using Data Guard, you need to create Standby Redo logs to allow the redo to be applied in (near) real time to the Standby. Without standby redo logs, Oracle will wait for an entire Archive Log to be filled and copied across to the standby before it will apply changes, which could take quite a while.

Which leads me to the problem I encountered a while ago, (more...)

enabling Database Vault on e-business RAC database

Right now I'm in a process to setup Database Vault for an E-Business suite database. This is a 2 node RAC cluster.
The DB is 12.1.0.2 with April 17 BP.

As the DB exists already, I followed How To Enable Database Vault in a 12c database ? (Doc ID 2112167.1).
Everything looks smooth, but unfortunately, at the Configuration of DV
exec dvsys.configure_dv('DVOWNER','DVMANAGER');
failes with
ERROR at line 1: 
ORA-47500: Database Vault (more...)

asmcmd “connected to an idle instance” – not

This is more a note to myself in case I’ll encounter a similar environment. But maybe it helps others – at least my search results weren’t suitable to Windows in the first place. Issue C:\> set ORACLE_HOME=C:\path\to\grid\home C:\> set ORACLE_SID=+ASM1 C:\> asmcmd connected to an idle instance. Environment Windows 2012R2 Oracle Grid Infrastructure 12.1.0.1 2 […]

root.sh fails with CRS-2101:The OLR was formatted using version 3

Got this while trying to install 11.2.0.4 RAC on Redhat Linux 7.2. root.sh fails with a message like

ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2017-11-09 15:43:37.883:
[client(37246)]CRS-2101:The OLR was formatted using version 3.

This is bug 18370031. Need to apply the patch before running root.sh.

I am speaking at Mumbai, OTNYatra 2007! Come and say Hi!

I am speaking at Mumbai leg of OTNYatra 2007. I shall be presenting about high availability in RAC and what are different options and configurations possible in it, including the 12c specific Transacion Guard and Application Continuity. If you are attending, do come at my session at 4pm and say Hello!

Vagrant? Again? …Really?

(Yes. And Ansible. And Oracle…) TL;DR. This is the repo I’ll be talking about. It can use Ansible to provision Oracle (SI/RAC). Like I’ve said before, I use Vagrant quite a lot and I basically have 2 configs that I use every time. One that uses an external ‘hosts.yml’ to define the hosts (ip, ram etc) […]

Exadata Upgrade to 12.2.0.1 – The Missing Step

I decided this week to be a little brave and upgrade one of the Enkitec Exadata racks to 12.2.0.1.  I installed the 12.2.1.0.0 Exadata image a few weeks ago, and have been waiting for a chance to upgrade clusterware to 12.2.  Thankfully, Oracle provides a very good note for this, but I did hit one large snag that should be documented.

The process for upgrading GI to 12.2 (more...)

Creating a RAC cluster using Ansible (part 2)

In my previous 2 blogposts i explained how to setup a Vagrantfile that deploys 2 virtualbox machines that can be used as a basis for a RAC cluster and how you could use Ansible to deploy RAC on those VM’s. In this post i want to dive a bit deeper into how i setup Ansible and why, keep in mind that this just one way of doing this.

The Github repository containing all the files (more...)

Grid Infrastructure 12c installation fails because of 255 in the subnet ID

I was doing another GI 12.1.0.2 cluster installation last month when I got really weird error.

While root.sh was running on the first node I got the following error:

2016/07/01 15:02:10 CLSRSC-343: Successfully started Oracle Clusterware stack
2016/07/01 15:02:23 CLSRSC-180: An error occurred while executing the command '/ocw/grid/bin/oifcfg setif -global eth0/10.118.144.0:public eth1/10.118.255.0:cluster_interconnect' (error code 1)
2016/07/01 15:02:24 CLSRSC-287: FirstNode configuration failed
Died at /ocw/grid/crs/install/crsinstall.pm  (more...)

When should you use Oracle RAC?

Over the past years we've worked on many Oracle projects, some with RAC, some without, some which intended to implement RAC and failed, and some which implemented poorly and ripped it out at the last minute. 

For those new to Oracle, RAC is the Real Application Cluster option for the database which lets you cluster the database servers. The main reason for using for RAC is that it brings resilience to a system, it (more...)

Review: Oracle RAC Performance Tuning

Some time ago, I received a free review copy of Brian Peasland‘s recent book, Oracle RAC Performance Tuning.

First, a note on my RAC background: I spent 7 years on Oracle’s RAC Support team. When customers had an intractable RAC performance issue, I was on the other end of the “HELP!” line until it was resolved.

I made Brian’s acquaintance through the MOS RAC Support forum, where Brian stood out as a frequent (more...)

racattack, meet ansible-oracle!

A while back I was approached by Jeremy Schneider, who is one of the original contributors to the racattack project and he wanted to know if I was interested in integrating ansible-oracle with the RAC Attack automation project, and of course I was! The idea was to provide a completely hands off installation of an […]

Creating a RAC Flex Cluster using ansible-oracle

As of version 1.3 it is possible to create a RAC Flex Cluster using ansible-oracle. From a ansible-oracle configuration perspective there is not a huge difference from a normal ‘standard’ cluster, basically a few new parameters. There are other differences though, specifically in how you have to run the playbook and deal with the inventory configuration. In […]

ansible-oracle, the RAC edition

First off, I’ll be setting up page, where I intend to have a complete list of parameters used in all roles and a description of what they actually do. Up until now I’ve kept it in a file in the github repo, but I think it will be easier to maintain this way. The page is […]

Rolling Out-of-Place patching para Oracle Grid Infrastructure 12c y 11gR2: clonación en esteroides

Mi artículo más reciente para ToadWorld, dentro de una serie dedicada a aplicar patches con mínima suspensión del servicio. Ahora detallando cómo aplicar patches a Grid Infrastructure usando un Golden Image.





12c Improvement in Database Service Management

Removing a service via srvctl has not historically resulted in the service being fully removed from the database and it would still be visible in DBA_SERVICES as show below in an 11.2.0.3 database:

Create the service:

$ srvctl add service -d orcl -s demo -r "ORCL1,ORCL2"

Have (more...)

Oracle SOA Suite 11g tuning tips for Oracle RAC database 11.2

During my project work I had to tune the SOA dehydration store on more than one occasion. Through this posting I would like to share the tuning tips collected during these exercises. It is not a step-by-step guide, because with tuning there never is a 'one-size-fits-all' in my humble opinion, but I provide in general a guidance that you can use as a reference for your own situation. I will also refer to related Oracle (more...)