I have announced my webinar on Oracle ZFS Storage Appliance for Exadata Backup & Recovery in NYOUG DBA webinar series. You can download webinar slides and voice recod from NYOUG web site
For this blog post I want to write about “ORA-01578: ORACLE data block corrupted” error and recovery. It is easy to recover from such error if you have RMAN backup - you, of course, have RMAN backup? (if not, create one) Even if you don’t have RMAN backup you can perform block recovery from OS based backups (datafile copies).
But can you perform block recovery (more...)
These days I’m implementing Oracle Dataguard for two Oracle databases 10.2 as part of disaster recovery project, one of them is around 1.7TB, not yet production. As part of the DG setup backups have to be available for both primary and standby. I preferred to use ASM and was able to negotiate with the storage admin to run storage replication for the FRA disks during the backup. This way I would have the same structure and files, locally at the disaster site immediately after the backup of the primary database is completed.
Unfortunately two weeks passed and by (more...)
Messing with SYSTEM tablespace is always tricky so my goal would be minimum risk and minimum downtime.
My testing environment is 11gR1 database on Linux 32bit OS.
After quick search on that subject I’ve noticed that DBA’s are comfortable with moving/renaming other datafiles but SYSTEM or UNDO are making problems. Some tablespaces are essential and cannot be taken (more...)
To check the performance of RMAN backup I recently started to trace it a little bit. As most of the time was not spent in any reading from disk or writing to media manager library event, it was on CPU. It's good to know the CPUs are of any good, but as I still want to know what's going on I tried to dig any deeper. CPU cycles are not just a magic black box where we put in a problem and the answer comes out after some times. At an abstraction layer it's a chain of functions where one (more...)
This is a quick post regarding the error on the subject. This is the second time it happens to me, so I thought I will write a bit about it.
I am refreshing one of my UAT environments (happens to be a Full Rack Exadata) using Oracle RMAN duplicate command. Then the following happens (on both occasions).
1.- Duplicate command fails (lack of space for restoring archivelogs, or any other error). This is can be fixed quite easy.
2.- following error while trying to open the database after restore and recover has finished:
SQL> alter database (more...)
I must say, this has been a busy weekend.
We have been promoting a release to production and a guaranteed restore point was created on Friday as rollback strategy. On Sunday I was called as we started to receive alerts.
Our monitoring system started to send emails and SNMP Traps with the following alerts:
OEM alert for Automatic Storage Management +ASM4_ssssss4: Disk group RECO has used 100% of safely usable free space. (Current Disk Group Used % of Safely Usable value: 100) OEM alert for Automatic Storage Management +ASM2_ssssss2: Disk group RECO has used 100% of safely usable (more...)
The operation can be performed by creating a datafile copy, either with the traditional commands or by means of recovery manager. In this case the following script was prepared to move a datafile.
copy datafile '
sql 'ALTER TABLESPACE
set newname for datafile (more...)
This afternoon on a Oracle 184.108.40.206 installation, i was trying to run the Oracle RMAN utility. Rman cli was just hanging and not going into RMAN.
[oracle@linux2 ~]$ . oraenv ORACLE_SID = [oracle] ? 11202 The Oracle base has been set to /opt/oracle/product/rhel4/database/220.127.116.11 [oracle@linux2 ~]$ rman
After entering above rman command, nothing happened. It just hung on the command prompt. I did not receive any error at all. Hmm… whats happening? I issued an CTRL-C to cancel it and tried again. Still same result. Okay lets trace it using Linux strace utility.
oracle@linux2 ~]$ strace rman (more...)
Us Linux DBAs love cron for scheduling jobs. These fancy software schedulers with their dependencies and clever logic have never really replaced a really dense, difficult to read crontab file. And that's just the way we like (more...)
We were moving a database to a new server. Same OS, same version of Oracle (10.2.0.4). The database is in Archivelog mode, but to keep things easy, we decided to do an offline, consistent backup.
So, we shutdown the database, then mounted it and performed an RMAN backup to a flash_recovery_area on a shared NFS filesystem:
backup device type disk database plus archivelog include current controlfile;The backup worked fine. So we logged on to the new server, made sure that we could (more...)
Some time ago I read a post about RMAN on Oracle-L that detailed what seemed like a very good idea.
The poster's RMAN scripts were written so that the only connection while making backups was a local one using the control file only for the RMAN repository.
rman target sys/manager nocatalog
After the backups were made, a connection was made to the RMAN catalog and a SYNC command was issued.
The reason for this was that if the catalog was unavailable for some reason, the backups would still succeed, which would not (more...)
In this post, I'll discuss a couple of other issues that you may come across; although, admittedly, they are fairly uncommon.
Standby DB Restores
Like a lot of other sites we maintain a physical standby database, managed using Dataguard, ready to use as a failover instance, in case of problems with the main RAC database. This database is the source for all backups, to avoid overloading the production servers. This leads to a small (more...)
The production database is a 2-node RAC cluster, but each of our test systems are single-instance databases. This causes one or two odd issues which aren't always obvious until you've hit them at least once.
In this post I'll discuss the first tricky issue which I came across: Identifying the Media.
At my current job (as in most IT departments), each day the most recent backup tapes are removed from the tape robot and sent (more...)
Last Week, I experienced a kernel crash using Oracle 10.2.0.3 on my Oracle Enterprise Linux 4 sandbox server. There was not a single reaction from the server, I had to reboot the box by pressing the reset button.
I happened to have a backup of this sandbox database, which gave me the opportunity to perform a media recovery.
The situation was that the undo Tablespace had a corrupt block.
Here is the alert log content:
Mon Aug 20 08:54:44 2007
SMON: enabling cache recovery
Mon Aug 20 08:54:45 2007
Errors in file /opt/oracle/app/oracle/admin/orcl/udump/orcl_ora_6733.trc:
ORA-01578: ORACLE data (more...)