You are in trouble when more than 8 kernel options are enabled in oracle executable

A brief headsup for everyone responsible for maintaining kernel options in the oracle executable.

The environment where I discovered this is a 3 node racattack cluster with Oracle GI & RDBMS 12.1.0.2.0. After enabling Unified auditing in my 12.1.0.2 database I noticed that all of a sudden database startup started failing with an almost  unlimited amount of core dump files being generated.

SQL*Plus: Release 12.1.0.2. (more...)

addNode.sh, failed root.sh and IB listener

So this customer has an Exadata quarter rack and they have an IB listener configured on both DB nodes (for DB connections from a multi-racked Exalogic system). We were adding a new DB node to this rack. So just followed the standard procedure of creating users, directories etc on the new node, setting up ssh equivalence and running addNode.sh. All went fine but root.sh failed. Little looking into the logs revealed that it (more...)

How many memory is truly used by my Oracle instance?

There are many posts about the amount of memory that is taken by the Oracle database executables and the database SGA and PGA. The reason for adding yet another one on this topic is a question I recently gotten, and the complexities which surrounds memory usage on modern systems. The intention for this blogpost is to show a tiny bit about page sharing of linux for private pages, then move on to shared pages, and (more...)

OEDA–Things to keep an eye on

So if you are filling an OEDA for Exadata deployment there are few things you should take care of. Most of the screens are self explanatory but there are some bits where one should focus little more. I am running the Aug version of it and the screenshots below are from that version.

  1. On the Define customer networks screen, the client network is the actual network where your data is going to flow. So typically (more...)

Copy Data Management with Delphix on Exadata

I’ve been busy reading and testing everything I can with Delphix, whenever I get a chance.  I’m incredibly fascinated by copy data management and the idea of doing this with Exadata is nothing new, as Oracle has it’s own version with sparse copy.  The main challenge is that Exadata’s version of this is kind of clunky and really doesn’t have the management user interface that Delphix offers.

reading_well

There (more...)

How to resolve missing dependency on exadata-sun-computenode-minimum

I’ve been really busy last few months – except spending a lot of time on M25 I’ve been doing a lot of Exadata installations and consolidations. I haven’t posted for some time now but the good news is that I got many drafts and presentations ideas.

This is a quick post about an issue I had recently. I had to integrate AD authentication over Kerberos on the compute nodes (blog post to follow) but had (more...)

SCAN Listener Crash After Applying July 2016 PSU

I ran in to a small issue while applying the July 2016 quarterly patch to a couple of Exadata racks last week.  The systems were running GI 12.1.0.2, previously with the January 2016 PSU.  The patches applied successfully, and we were beginning the process of running the post-patch scripts on the databases in the cluster.  This process involves manually starting the database in upgrade mode, and we saw a message in SQL*Plus that the (more...)

Redirecting ILOM Console Via SSH Tunneling With Putty

While preparing for a virtualized Exalytics upgrade, I was testing the various connection methods that would be needed.  Last on the list was to validate that I could connect to the ILOM’s web interface, launch the java console, and mount an ISO file from my laptop to the machine.   This is necessary because the Oracle VM server upgrade for a virtualized Exalytics requires you to boot from a specific ISO file in order to patch (more...)

ORA-15418: Appliance Mode Not Supported For Compatible.asm 12.1.0.0.0

The other day, I was upgrading a compatible.asm parameter to 12.1 on Exadata and I faced this error for the first time :

View the code on Gist.

Indeed, a diskgroup can have this parameter set to TRUE or FALSE :

View the code on Gist.

I then found this note on Metalink : EXADATA : What Is Oracle ASM appliance.mode Attribute (Doc ID 1664261.1) which explains that starting from Exadata (more...)

Notes on vendor lock-in

Vendor lock-in is an important subject. Everybody knows that. But few of us realize just how complicated the subject is, nor how riddled it is with paradoxes. Truth be told, I wasn’t fully aware either. But when I set out to write this post, I found that it just kept growing longer.

1. The most basic form of lock-in is:

  • You do application development for a target set of platform technologies.
  • Your applications can’t run (more...)

How To Configure Exadata Database Machine in Enterprise Manager Cloud Control 13c (OEM13c)

I have followed the steps in Oracle Documentation link: https://docs.oracle.com/cd/E63000_01/EMXIG/ch2_deployment.htm#EMXIG215 to configure Exadata Database Machine in OEM13c. If you want to configure your Exadata in OEM13c you have to follow the above mentioned link. In this post I will share the mandatory steps for configuration, and some of the issues which I faced while configuring the Exadata…

How To Avoid Exadata Cell Node Reboot While Modifying NTP Service

Recently, we had a hardware failure on our NTP server which caused its crash. Due to this, the ntpd daemon on compute nodes and cell nodes went offline. After fixing the hardware issue on NTP server, starting ntpd service using "service ntpd start" on compute nodes was successful but when I started the ntp service…

ORA-56841: Master Diskmon cannot connect to a CELL

Faced this error while querying v$asm_disk after adding new storage cell IPs to cellip.ora on DB nodes of an existing cluster on Exadata. Query ends with ORA-03113 end-of-file on communication channel and ORA-56841 is reported in $ORA_CRS_HOME/log/<hostname>/diskmon/diskmon.log. Reason in my case was that the new cell was using different subnet for IB. It was pingable from the db nodes but querying v$asm_disk wasn’t working. Changing the subnet for IB on new cell to (more...)

Is Your Data Center Ready For Exadata Machines!!

Before opting to procure Exadata machines, the first thing you need to check is the readiness of your data center to hold these machines. Exadata's dimensions are similar to other normal racks i.e a 42U rack. The complete details for X5-2 physical site requirements can be found HERE. This is very important step as, for some data…

Exadata X6-2 Launched

5th April 2016, Oracle announced arrival of Exadata X6-2. Overview of all its capabilities can be found HERE. Check DataSheet for list of hardware components and capacity of each X6-2 rack. Sometimes, Oracle changes hardware specifications for its racks e.g X5-2 was released with 4TB HDD initially, but after few months the datasheet was modified and…

Planning Database Hardware Upgrade !!! Consider Oracle Exadata

Upgrading database hardware in an organization is always a cumbersome process. The most time consuming step is, planing for the upgrade, which mainly includes choosing right hardware for your Oracle databases. After deciding on the hardware type for your databases, rest will be taken care by technical teams involved. I will be discussing, how I…

Oracle Exadata X6 released

Oracle has just announced the next generation of Exadata Database Machine – X6-2 and X6-8.

Here are the changes for Exadata X6-2:
1) X6-2 Database Server: As always the hardware has been updated and the 2-socket database servers are now equip with latest twenty two-core Intel Xeon E5-2699 v4 “Broadwell” processors in comparison to X5 where we had eighteen-core Intel Xeon E5-2699 v3 processors. The memory is still DDR4 and the default configuration comes with (more...)

Collaborate 16: My sessions

I’ll be at Collaborate 16 next month and looking forward to seeing lots of good friends, learning some new things, and sharing a little experience too. For the last of those, I’ll present 3 sessions, er, more like 2.2 sessions:

  • Wed, 13-Apr, 12:45-12:55pm: Oak Table World 10-minute Lightning talk “Tools used for security and compliance on Linux”
  • Wed, 13-Apr, 3-4pm: Oak Table World session “IPv6: What You Need to Know (with Linux & Exadata references)”
  • (more...)

Big Nodes, Concurrent Parallel Execution And High System/Kernel Time

This probably only is relevant for customers that run Oracle on big servers with lots of cores, like some of my clients that make use of the Exadata Xn-8 servers, like a X4-8 with 120 cores / 240 CPUs per node.

They recently came up with a re-write of a core application functionality. Part of this code did start the same code path for different data sets potentially several times concurrently ending up with many (more...)

Storage difference between 2 identical Exa boxes. How and why?

We noticed around 1.6TB storage difference between two Eight (1/8) Exadata boxes while configuring Data Guard. Wondered what went wrong. The Exa box configured for DR was around 1.6TB short compare to the other Exa box. Verified the lun, physical disk and grdidisk status on a cell, which showed active/online status. The tricky part on Exadata is, everything has to be active/online across all cell storage servers. We then figured-out that grid disk status on (more...)