Failed to create voting files on disk group RECOC1

Long story short, faced this issue while running OneCommand for one Exadata system. The root.sh step (Initialize Cluster Software) was failing with the following error on the screen

Checking file root_dm01dbadm02.in.oracle.com_2017-04-27_18-13-27.log on node dm01dbadm02.somedomain.com
Error: Error running root scripts, please investigate…
Collecting diagnostics…
Errors occurred. Send /u01/onecommand/linux-x64/WorkDir/Diag-170427_181710.zip to Oracle to receive assistance.

Doesn’t make much sense. So let us check the log file of this step

2017-04-27 (more...)

Vagrant? Again? …Really?

(Yes. And Ansible. And Oracle…) TL;DR. This is the repo I’ll be talking about. It can use Ansible to provision Oracle (SI/RAC). Like I’ve said before, I use Vagrant quite a lot and I basically have 2 configs that I use every time. One that uses an external ‘hosts.yml’ to define the hosts (ip, ram etc) […]

Error ORA-01033 After Doing a Switchover in a 12.1 RAC Environment

The other day I did a switchover in a RAC environment , which went pretty smooth , but after doing the switchover in the primary, I kept getting the following error:

select dest_name,status,error from gv$archive_dest_status where dest_id=2;

DEST_NAME
--------------------------------------------------------------------------------
STATUS	  ERROR
--------- -----------------------------------------------------------------
LOG_ARCHIVE_DEST_2
ERROR	  ORA-01033: ORACLE initialization or shutdown in progress

LOG_ARCHIVE_DEST_2
ERROR	  ORA-01033: ORACLE initialization or shutdown in progress

I went and checked the standby, and saw the standby was in recover mode (more...)

Decipher EM Error Message for RAC

I am not sure if there is a way to have EM display the actual commands it uses to check and alert errors.

It would be nice to be able to verify using the same syntax.

Examples of errors I was paged for:

Message=ora.net2.network has 1 instances in OFFLINE State
Key Value=resource_ora.network.type_ora.net2.network
Message=ora.host01_2.vip has 1 instances in OFFLINE State
Key Value=resource_ora.cluster_vip_net2.type_ora.host01_2.vip

Of course, crsctl (more...)

Exadata Upgrade to 12.2.0.1 – The Missing Step

I decided this week to be a little brave and upgrade one of the Enkitec Exadata racks to 12.2.0.1.  I installed the 12.2.1.0.0 Exadata image a few weeks ago, and have been waiting for a chance to upgrade clusterware to 12.2.  Thankfully, Oracle provides a very good note for this, but I did hit one large snag that should be documented.

The process for upgrading GI to 12.2 (more...)

Oracle RAC 12.1 – lsnodes exited with code 9

I was trying to do a 2 node RAC setup on Solaris 11.3 where Oracle Solaris Cluster 4.3 was already configured. Installed was running but the Cluster Node Information screen was appearing like this

error

The install log shows this:

INFO: Checking cluster configuration details

INFO: Found Vendor Clusterware. Fetching Cluster Configuration

INFO: Executing [/tmp/OraInstall2017-03-28_12-50-48PM/ext/bin/lsnodes]

with environment variables {TERM=xterm, LC_COLLATE=, SHLVL=3, JAVA_HOME=, XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt, SSH_CLIENT=172.16.64.55 56370 22, LC_NUMERIC=, LC_MESSAGES=, MAIL=/var/mail/oracle, PWD=/export/software/grid/grid, XTERM_VERSION=XTerm(320), (more...)

racattack-ansible-oracle Up and Running

From a time long ago – https://mdinh.wordpress.com/2016/12/04/toys-for-when-you-i-are-bored/

With help from oravirt, I was able to install RAC VMs.

At this point, only the servers have been created and nothing has been installed; that’s coming up at some point.

Some clarification for setup=standard vagrant provision

setup=standard (shell environment variable)

vagrant provision (executable)

This is where the confusion was at first.

falcon@falconidae MINGW64 /e/racattack-ansible-oracle (master)
$ setup=standard vagrant provision

collabn2 eth1 lanip  :192.168.78. (more...)

Little things worth knowing: Executing RDA on RAC

Result! I have finally been able to gather a complete RDA (Oracle Remote Diagnostic Agent) output on my 2 node RAC system. After consulting the relevant documentation on MOS-which is spread over at least 42 Doc IDs-I found them not to be very helpful to the degree that some of what I read is actually wrong or contradicting. I put together a short note, primarily to myself, to remind me of the process. I hope you find it useful, (more...)

UKOUG 2-Stream London SIG

ukoug_logoWe not have the agenda online for the UKOUG 2-stream London SIG, due to take place on 23rd February. There will be an RDBMS Stream and a RAC, Cloud, Infrastructure and Availability Stream.

Mark Rittman will be giving us a keynote talk, with an emphasis on kettles or eating dinner in the dark or the Hadoop cluster in his garage something.

There are some really cool other speakers too, from Oracle, Leading Consultancies and the (more...)

Creating a RAC cluster using Ansible (part 2)

In my previous 2 blogposts i explained how to setup a Vagrantfile that deploys 2 virtualbox machines that can be used as a basis for a RAC cluster and how you could use Ansible to deploy RAC on those VM’s. In this post i want to dive a bit deeper into how i setup Ansible and why, keep in mind that this just one way of doing this.

The Github repository containing all the files (more...)

Call for Papers – UKOUG 2017 SIGs

All,

The time has come for me to plan the upcoming 2017 UKOUG Special Interest Groups.

I am chairman of the RAC, Cloud, Infrastructure and Availability (RAC-CIA) SIG and I’m after presentations for the 3 joint SIGs we will be putting on with the RDBMS SIG, plus the stand-alone SIG will we will having in the autumn.

A SIG is a single day one-or-two stream conference which we take around the UK to make it (more...)

reinstall tfactl after GI Upgrade 12.1.0.2

Uncategorized
| Oct 19, 2016

tfactl

Recently I finished a Grid Upgrade from 11.2.0.4 to 12.1.0.2 + PSU JUL 2016. So far so good during a check I saw that the old tfactl tool under Software Release 11.2.0.4 where up and running.

That could not be okay.  So I start an Uninstall and Setup for Release 12.1.0.2.

What steps has to be done?

Check the actual tfactl installation


 /u01/app/grid/tfa/bin/tfactl  (more...)

addNode.sh, failed root.sh and IB listener

So this customer has an Exadata quarter rack and they have an IB listener configured on both DB nodes (for DB connections from a multi-racked Exalogic system). We were adding a new DB node to this rack. So just followed the standard procedure of creating users, directories etc on the new node, setting up ssh equivalence and running addNode.sh. All went fine but root.sh failed. Little looking into the logs revealed that it (more...)

Grid Infrastructure 12c installation fails because of 255 in the subnet ID

I was doing another GI 12.1.0.2 cluster installation last month when I got really weird error.

While root.sh was running on the first node I got the following error:

2016/07/01 15:02:10 CLSRSC-343: Successfully started Oracle Clusterware stack
2016/07/01 15:02:23 CLSRSC-180: An error occurred while executing the command '/ocw/grid/bin/oifcfg setif -global eth0/10.118.144.0:public eth1/10.118.255.0:cluster_interconnect' (error code 1)
2016/07/01 15:02:24 CLSRSC-287: FirstNode configuration failed
Died at /ocw/grid/crs/install/crsinstall.pm  (more...)

SCAN Listener Crash After Applying July 2016 PSU

I ran in to a small issue while applying the July 2016 quarterly patch to a couple of Exadata racks last week.  The systems were running GI 12.1.0.2, previously with the January 2016 PSU.  The patches applied successfully, and we were beginning the process of running the post-patch scripts on the databases in the cluster.  This process involves manually starting the database in upgrade mode, and we saw a message in SQL*Plus that the (more...)

When should you use Oracle RAC?

Over the past years we've worked on many Oracle projects, some with RAC, some without, some which intended to implement RAC and failed, and some which implemented poorly and ripped it out at the last minute. 

For those new to Oracle, RAC is the Real Application Cluster option for the database which lets you cluster the database servers. The main reason for using for RAC is that it brings resilience to a system, it (more...)

Configuring VNCR for 11.2.0.4 Oracle RAC

By default on an Oracle RAC installation, the listeners are configured to allow any database to register with them.  There is no security out of the box to determine which databases may register.  While this makes it easy to create new databases without worrying about listener registration, this can cause potential problems in a real environment.

This can be dangerous working with RAC environments where the database registers with both a local and remote listener.  The (more...)

Review: Oracle RAC Performance Tuning

Some time ago, I received a free review copy of Brian Peasland‘s recent book, Oracle RAC Performance Tuning.

First, a note on my RAC background: I spent 7 years on Oracle’s RAC Support team. When customers had an intractable RAC performance issue, I was on the other end of the “HELP!” line until it was resolved.

I made Brian’s acquaintance through the MOS RAC Support forum, where Brian stood out as a frequent (more...)

HAIP and Exadata

If you’ve run an exachk report, y0u may have seen the following message with regard to your databases:

Status Type Message Status On Details
FAIL Database Check Database parameter CLUSTER_INTERCONNECTS is NOT set to the recommended value db01:dbm011, db02:dbm012 View

This check is commonly seen when a database is created on Exadata without using the custom “Exadata” templates included with the database creation assistant.  These customized templates include a multitude of recommended parameter settings found in (more...)

racattack, meet ansible-oracle!

A while back I was approached by Jeremy Schneider, who is one of the original contributors to the racattack project and he wanted to know if I was interested in integrating ansible-oracle with the RAC Attack automation project, and of course I was! The idea was to provide a completely hands off installation of an […]