Recently one of my customers got a complaint from their DNS administrators, our Exadata’s are doing 40.000 DNS requests per minute. We like our DNS admins so we had a look into these request and what was causing them. I started with just firing up a tcpdump on one of the bonded client interfaces on a random compute node:
[root@dm01db01 ~]# tcpdump -i bondeth0 -s 0 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bondeth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:41:04.937009 IP dm0101.domain.local.59868 > dnsserver01. (more...)
Well, it's been a really busy past few months, and I hate to admit it, but I've been neglecting this space more than anything. Despite having many different posts in the works, nothing is quite finished yet. I do have a little time to mention a few of my upcoming speaking events, though.
First, I'll be in Denver for the Rocky Mountain Oracle Users Group Training Days 2013, February 11-13. Enkitec has more than a few sessions during the conference, ranging from Exadata to Big Data to APEX. Check out the agenda for a full list. Also, we'll have an (more...)
I’ve worked on 24×7 systems for more than a decade, and I have a real dislike of downtime. For one, it can be a real pain to agree any downtime with the business, and while RAC can and does help when you do work in a rolling fashion, there is still risk.
The promise of online patching has been a long one, and it is only recently that I dipped my toe in the water with them. Unfortunately, they are not a panacea, and in this blog posting I’m going to share some of the downsides.
Of course not all (more...)
I have announced my webinar on Oracle ZFS Storage Appliance for Exadata Backup & Recovery in NYOUG DBA webinar series. You can download webinar slides and voice recod from NYOUG web site
In Friday December 14, 2012 at 12:00 PM -1:00 PM EDT I will be giving a webinar for NYOUG SIG with the following abstract
When it comes to the backup and recovery infrastructure of the Exadata Database Machine, conventional solutions often have only limited performance to keep up with Exadata throughput, whereas Oracle ZFS Storage Appliance can be configured as a very fast, capable, and easy-to-manage backup and recovery solution for any Exadata environment. In this session Husnu Sensoy will describe some of the configuration possibilities of the ZFS Storage Appliance to create a flexible backup and recovery environment for Exadata, (more...)
Hybrid Columnar Compression (HCC) is one of the Exadata features but lately Oracle has been pushing this featurei to other Oracle hardware like the ZFS Storage Appliance and Axiom Pillar Storage series. We recently got a ZFS Storage Appliance (ZFSSA) at VX Company, so we are now able to use HCC on the Oracle Database Appliance (ODA). To use HCC we need to create a tablespace with datafiles on a ZFS Storage Appliance, in order to de so we going to hookup our ODA using directNFS (dNFS). I am not going in the details of explaining dnfs in this blogpost, (more...)
I’ve been working a lot with graphing DB and OS metrics in R. I find it especially useful in Exadata POVs (proof of value) to gather and graph the oswatcher vmstat files for the compute nodes and iostat for the cells. For an example, take a look at this graph (PDF, 168 KB) of what [...]
With the announcement Exadata X3, Oracle has introduced a new feature called “FlashCache Writeback” to allow writes to cell Flash Cache (aka Exadata Smart FlashCache) in WriteBack mode. Earlier with WriteThrough mode, writes were not written to FlashCache, instead they were written directly to cell disks. Exadata software used to decide whether to cache these writes back into FlashCache or not. In WriteBack mode, writes are written to cell FlashCache and acknowledgement is given back to calling process as soon as data is written to flashcache. Exadata Server software de-stages the dirty writes in flashcache to spinning disks in the (more...)
Well, it's finally public, so we're able to openly talk about the new Exadata X3 systems. Looking back on my pre-openworld predictions, I was pretty close on a few things. I was correct on the database servers, which will have Xeon E5-2690 CPUs (8 core, 2.9GHz) with 128GB RAM upgradeable to 256GB. It looks like we won't get active/active Infiniband for a while, since the cards in there are staying the same. On the X3-8, the compute nodes are staying the same, for reasons detailed by Kevin Closson a few weeks ago. I also previously blogged about the (more...)
In my last post, I looked at the effect of the Exadata smart flash logging. Overall, there seemed to be a slight negative effect on median redo log sync times. This chart (slightly different from the last post because of different load and configuration of the system), shows how there’s a “hump” of redo log syncs that take slightly longer when the flash logging is enabled:
But of course, the flash logging feature was designed to improve performance not of the “average” redo log sync, but of the “outliers”.
In my tests, I had 40 concurrent (more...)
Just a quick note about change in the way the compute nodes are patched starting from version 126.96.36.199.1. For earlier versions Oracle provided the minimal pack for patching the compute nodes. Starting with version 188.8.131.52.1 Oracle has discontinued the minimal pack and the updates to compute nodes are done via Unbreakable Linux Network (ULN).
Now there are three ways to update the compute nodes:
1) You have internet access on the Compute nodes. In this case you can download patch 13741363, complete the one time setup and start the update.
I’ll be speaking at the Enkitec Extreme Exadata Expo (E4), August 13-14 in Dallas Texas (you can also attend virtually). They’ve recruited some of the top names from community including keynote speaker Andrew Mendelsohn, Arup Nanda, Cary Millsap, Jonathan Lewis, Karen Morton, Maria Colgan, Kerry Osborne and Tanel Põder. I left a lot of names off the list, many of which you probably [...]
Exadata storage software 184.108.40.206 introduced the Smart flash logging feature. The intent of this is to reduce overall redo log sync times - especially outliers - by allowing the exadata flash storage to serve as a secondary destination for redo log writes. During a redo log sync, Oracle will write to the disk and flash simultaneously and allow the redo log sync operation to complete when the first device completes.
Jason Arneil reports some initial observations here, and Luis Moreno Campos summarized it here.
I’ve reported in the past on using SSD for (more...)
Recently i got a question from an Exadata customer what would happen if you would forget to relink the oracle for IPC RDS use for just one home. Would Oracle be smart enough to fall back on UPC for the complete RAC database? Time to do a little test and find out what will happen here:
First lets see how IPC communication between the nodes is done right now. We can use skgxpinfo to see how it is linked since version 220.127.116.11.0:
[oracle@dm01db01 [dbfs1] ~]$ dcli -g dbs_group -l oracle /u01/app/oracle/product/18.104.22.168/dbhome_1/bin/skgxpinfo