Change the SID of the Oracle database using DBNEWID

In this post I will describe how to change Oracle SID using utility DBNEWID. As I can see DBNEWID is available from 9i version but I’ve never heard about it. Till now, I have used procedure where I’m manually editing and re-creating control file.

But I think it’s much better (more...)

Exadata 11.2.3.2.1 NFS Issues – Ksplice Support for Exadata?

When the 11.2.3.2.1 release of the Exadata Storage Server software was released, I was a little excited.  There were numerous oneoff patches for the previous release, 11.2.3.2.0, which was the first version to support the Exadata X3, writeback flashcache, run UEK on the X#-2 systems, etc.  With that many large changes introduced in one version, it was likely to see some bugs in the .0 release.  Fortunately, Oracle was quick to fix many of those issues, but it resulted in several separate patches to update the cellsrv software.

I was working with (more...)

The Case of Missing Utility

Recently, I’ve asked to help a client who was having doubts about their backup strategy. The client was considering image backup of database which would be constantly recovered and backed up to tape on the daily basis, by using “backup recovery area” command. However, there was a problem: there is no “restore recovery area” command. The utility to restore the recovery area seems to be missing, thus the name of this article became “The Case of Missing Utility”.

So, let’s see some definitions. This is what the client has envisioned:

RMAN> backup incremental level 1 for recover of copy database;

(more...)

Slides and Offline Recording of my NYOUG Webinar

I have announced my webinar on Oracle ZFS Storage Appliance for Exadata Backup & Recovery in NYOUG DBA webinar series. You can download webinar slides and voice recod from NYOUG web site


Restoring a Database with Rman Controlfile Parameters

There are occasions when you recieve a complete backup of a database from a  place different then yours and have to do a restore. normally a restore is simple . put the files in a directory run rman catalog on them and you are done. but what if you have rman (more...)

RMAN Transportable tablespace bug

I was a long time since I last used Transportable Tablespaces and I decided to refresh my memory and setup a test environment for Streams using TTS feature. I was going to use example script from Oracle Streams Documentation which is creating necessary data files for specified point of time (no need to switch source tablespace into read only mode) and also is running Data Pump creating meta data file. I have done necessary  changes and run it.
[oracle@testdb1 oracle]$ rsp / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Mon Aug 6 12:50:04 2012
Copyright (c)  (more...)

Perform Oracle Block Recovery without having valid backup

My DBA friend Dejan from Baze Podataka database related site wrote post about Oracle Bug 8943287 which reminded me how dangerous bug this is. I was completely unaware about this bug till few months ago.

For this blog post I want to write about “ORA-01578: ORACLE data block corrupted” error and recovery. It is easy to recover from such error if you have RMAN backup - you, of course, have RMAN backup? (if not, create one) Even if you don’t have RMAN backup you can perform block recovery from OS based backups (datafile copies).

But can you perform block recovery (more...)

Failed creating physical standby database

These days I’m implementing Oracle Dataguard for two Oracle databases 10.2 as part of disaster recovery project, one of them is around 1.7TB, not yet production. As part of the DG setup backups have to be available for both primary and standby. I preferred to use ASM and was able to negotiate with the storage admin to run storage replication for the FRA disks during the backup. This way I would have the same structure and files, locally at the disaster site immediately after the backup of the primary database is completed.

Unfortunately two weeks passed and by (more...)

How to move SYSTEM tablespace with minimum downtime

Two weeks ago a colleague DBA asked me what would be the best way to move SYSTEM tablespace with minimum downtime. There are several ways to perform that task and I will note what would be the best way in my opinion.

Messing with SYSTEM tablespace is always tricky so my goal would be minimum risk and minimum downtime.

My testing environment is 11gR1 database on Linux 32bit OS.

After quick search on that subject I’ve noticed that DBA’s are comfortable with moving/renaming other datafiles but SYSTEM or UNDO are making problems. Some tablespaces are essential and cannot be taken (more...)

bzip2 twice?


To check the performance of RMAN backup I recently started to trace it a little bit. As most of the time was not spent in any reading from disk or writing to media manager library event, it was on CPU. It's good to know the CPUs are of any good, but as I still want to know what's going on I tried to dig any deeper. CPU cycles are not just a magic black box where we put in a problem and the answer comes out after some times. At an abstraction layer it's a chain of functions where one (more...)

ORA-16072: a minimum of one standby database destination is required

This is a quick post regarding the error on the subject. This is the second time it happens to me, so I thought I will write a bit about it.

The problem:

I am refreshing one of my UAT environments (happens to be a Full Rack Exadata) using Oracle RMAN duplicate command. Then the following happens (on both occasions).

1.- Duplicate command fails (lack of space for restoring archivelogs, or any other error). This is can be fixed quite easy.

2.- following error while trying to open the database after restore and recover has finished:

SQL> alter database  (more...)

Archive area +RECO has -7440384 free KB remaining (Usable_file_MB is negative)

I must say, this has been a busy weekend.

We have been promoting a release to production and a guaranteed restore point was created on Friday as rollback strategy. On Sunday I was called as we started to receive alerts.

The problem:

Our monitoring system started to send emails and SNMP Traps with the following alerts:

OEM alert for Automatic Storage Management +ASM4_ssssss4: Disk group RECO has used 100% of safely usable free space. (Current Disk Group Used % of Safely Usable value: 100)
OEM alert for Automatic Storage Management +ASM2_ssssss2: Disk group RECO has used 100% of safely usable  (more...)

Moving a datafile Online

A datafile cannot be completely moved online in a user transparent way, but what it can be done is to minimize the time required to perform the operation, and avoid the traditional procedure executed when the database is mounted, which permits more availability for the non affected production tablespaces.

The operation can be performed by creating a datafile copy, either with the traditional commands or by means of recovery manager. In this case the following script was prepared to move a datafile.

run {
copy datafile ''
to '';
sql 'ALTER TABLESPACE OFFLINE IMMEDIATE';
set newname for datafile (more...)

Oracle RMAN command prompt hang???

This afternoon on a Oracle 11.2.0.2 installation, i was trying to run the Oracle RMAN utility. Rman cli was just hanging and not going into RMAN.

[oracle@linux2 ~]$ . oraenv
ORACLE_SID = [oracle] ? 11202
The Oracle base has been set to /opt/oracle/product/rhel4/database/11.2.0.2
[oracle@linux2 ~]$ rman
 

After entering above rman command, nothing happened. It just hung on the command prompt. I did not receive any error at all. Hmm… whats happening? I issued an CTRL-C to cancel it and tried again. Still same result. Okay lets trace it using Linux strace utility.

oracle@linux2 ~]$ strace rman
 (more...)

Last Sunday

Our production databases are backed up with one of two schedules.  Most days our backups are sent offsite and retained for 3 months before being returned.  But on the last Sunday of each month, our backups are archived for a longer period of time.  Therefore the RMAN backups and Veritas catalogs need to be retained for up to 7 years!

Us Linux DBAs love cron for scheduling jobs. These fancy software schedulers with their dependencies and clever logic have never really replaced a really dense, difficult to read crontab file. And that's just the way we like (more...)

Cold Backup RMAN Fun and Games

Here's an RMAN oddity.  If anyone could explain it, I'd be grateful.

We were moving a database to a new server.  Same OS, same version of Oracle (10.2.0.4).  The database is in Archivelog mode, but to keep things easy, we decided to do an offline, consistent backup.

So, we shutdown the database, then mounted it and performed an RMAN backup to a flash_recovery_area on a shared NFS filesystem:

backup device type disk database plus archivelog include current controlfile;
The backup worked fine. So we logged on to the new server, made sure that we could (more...)

Cool but unknown RMAN feature

Unknown to me anyway until just this week.

Some time ago I read a post about RMAN on Oracle-L that detailed what seemed like a very good idea.

The poster's RMAN scripts were written so that the only connection while making backups was a local one using the control file only for the RMAN repository.
rman target sys/manager nocatalog

After the backups were made, a connection was made to the RMAN catalog and a SYNC command was issued.

The reason for this was that if the catalog was unavailable for some reason, the backups would still succeed, which would not (more...)

RMAN Duplicate Gotchas – Part 2: Standby DB Restores and Redo Log Group

In my previous post I discussed identifying which media you would need to use when performing a restore in order to duplicate a database via RMAN.

In this post, I'll discuss a couple of other issues that you may come across; although, admittedly, they are fairly uncommon.

Standby DB Restores

Like a lot of other sites we maintain a physical standby database, managed using Dataguard, ready to use as a failover instance, in case of problems with the main RAC database. This database is the source for all backups, to avoid overloading the production servers. This leads to a small (more...)

RMAN Duplicate Gotchas – Part 1: Identifying the Media

I've recently been doing quite a few RMAN Duplicate Database jobs, in order to create dev and test systems as copies of our production server.

The production database is a 2-node RAC cluster, but each of our test systems are single-instance databases. This causes one or two odd issues which aren't always obvious until you've hit them at least once.

In this post I'll discuss the first tricky issue which I came across: Identifying the Media.

At my current job (as in most IT departments), each day the most recent backup tapes are removed from the tape robot and sent (more...)

Recovering Corrupted Data Blocks

Last Week, I experienced a kernel crash using Oracle 10.2.0.3 on my Oracle Enterprise Linux 4 sandbox server. There was not a single reaction from the server, I had to reboot the box by pressing the reset button.
I happened to have a backup of this sandbox database, which gave me the opportunity to perform a media recovery.

The situation was that the undo Tablespace had a corrupt block.

Here is the alert log content:

Mon Aug 20 08:54:44 2007
SMON: enabling cache recovery
Mon Aug 20 08:54:45 2007
Errors in file /opt/oracle/app/oracle/admin/orcl/udump/orcl_ora_6733.trc:
ORA-01578: ORACLE data (more...)