Where did my RAM go?

Uncategorized
| Oct 23, 2019

In my previous posts (e.g. here and here) I showed how to use ps output (e.g. from ExaWatcher) visualization to spot performance problems in Linux. Here I’d like to show that this approach can be taken a little bit further, namely, to find the source of increase in memory usage.

The R code for this is quite straightforward. I also think it shouldn’t be much of a problem to do the same (more...)

Memory fragmentation: the silent performance killer

Uncategorized
| Oct 14, 2019

In my last post, where I analyzed the problems caused by memory fragmentation on a Linux server, I said very little about memory fragmentation itself. I wanted to tell a story, so I had to dial down the technical stuff. But now that the story is told, I figured I should make a more technical post on the subject. I think it might be useful for many Linux DBAs or SAs, since there is not (more...)

How to hang a server with a single ping, and other fun things we learned in a 18c upgrade

Uncategorized
| Oct 3, 2019

Every upgrade is a bit of a lottery. But for a long running, well established system with a lot of legacy code, it can be more of a Russian roulette. You are unlikely to gain much — your system has been running fine, thanks to various tweaks the apps team applied here and there over the years. All these nice and shiny features that arrive with the upgrade — well, sure, there can be a (more...)

Swapping (paging) for DBAs

Uncategorized
| Jul 8, 2019

Classic symptoms of memory pressure (free physical memory running low + swapping) are often more difficult to interpret than they seem, especially on modern enterprise grade servers. In this article, I attempt to bring some clarity to the issue. It is based on Linux, although many observations can be generalized to other Unix-like operating systems.

Basics definitions

Let’s define some basic terms and concepts:

Page is simply a unit of memory allocation, pretty much like (more...)

A picture paints a thousand words

Uncategorized
| Apr 30, 2019

When I first learned about Active Session History, it was a real game changer for me. It’s (kinda) like tracing which is always on, for every single session… well, active session, sure, but who cares about idle ones? For a while I got so obsessed with it that I almost stopped using other tools — fortunately, that was a only short while, because as great as ASH is, you still need other tools. But to (more...)

ASH for OS processes

Uncategorized
| Apr 14, 2019

There is a lot of different tools for analyzing OS process states which can be helpful in resolving non-trivial performance issues. One of the limitations of such tools is that they are mostly active ones — i.e. you have to do some extra work to collect the desired diagnostic information. This is inconvenient when the problem you’re facing is intermittent and manifests itself on an irregular and unpredictable schedule.

However, there are some great (more...)

Finding the root cause of “CPU waits” using stack profiling

Uncategorized
| Apr 8, 2019

Call stack profiling and flame graphs have been a hot topic in Oracle tech blogs last few years, and recently I got a chance to use it to troubleshoot an actual production performance issue. It was quite an interesting journey, with some twists and turns along the way. Let me start by presenting some background for the problem.

The database in question (a 3-node Exadata RAC cluster, 11.2.0.4 database version, 18.1. (more...)

Tools for troubleshooting network performance issues

Uncategorized
| Feb 17, 2019

In my previous article I discussed general questions related to network issues in Data Guard due to packet loss and/or retransmissions. Here I’d like to move to discussing specific tools and methodologies for troubleshooting such issues.

Such tools can be broken down by following criteria:

  • server-side or network-side
  • active or passive
  • level of detail they provide (aggregate statistics or individual packet capture).

I think the first item on the list is more or less self-explanatory: (more...)

Troubleshooting network throughput issues in Oracle Data Guard

Uncategorized
| Dec 8, 2018

Introduction

In this article I describe the basic mechanics of TCP and DataGuard as well as relevant performance metrics on the database, OS and network sides. The idea is to give DBAs some ammunition in addressing DataGuard performance issues. The most important stage of troubleshooting is the correct identification of the nature of the issue, e.g. being able to tell whether the problem has to do with the network as such, or DataGuard, or (more...)

Set-based processing

Uncategorized
| Jul 10, 2017

Last week I participated in Oracle’s Real World Performance event — four days of lectures, quizzes, live demos and hands-on exercises. It was quite interesting, even more so than I expected it to be.

Understandably, a lot of time was spent discussing the perils of row-by-row processing. After all, it was Real World Performance, so it was based on performance problems that the authors of the course faced most often. And many, if not most, (more...)