this post is about investigating Oracle wait events using stack profiles and flame graphs extended with OS-process state and Oracle wait event details.Context: The case of the DB Time > CPU Time + Wait Time
provides wait event and CPU time accounting, a powerful and readily accessible data source for performance troubleshooting. An Oracle session at a given point in time is either on CPU, for example when processing data from cache, (more...)
This blog post is about kernel stack profiling and visualization with flame graphs with examples.
Stack profiling and flame graphs
are very useful techniques and tools for troubleshooting and investigating workloads at the OS-level and understand which code path take most of the execution time. You can find extensive material and examples o flame graphs in Brendan Gregg
's website and blog. A few additional examples of using stack tracing and flame graphs for (more...)
Recently, I have discovered a fancy bug affecting few version of Linux kernel. Without any warnings JVM just hangs in GC pause forever. Root cause is a improper memory access in kernel code. This post by Gil Tene gives a good technical explanation with deep emotional coloring.
While this bug is not JVM specific, there are few other multithreaded processes you can find on typical Linux box.
This recent bug make me remember few other (more...)
It’s getting harder and harder to find time for writing. Anyway, this post stays from a long time in my draft so, I think is time to finish and post it. That one is the last one from the optimizer statistics series. For now, of course :) After describing the horror around the way that […]
Sometimes it just happens. You have a bloated Java application at your hand and it does not perform well. You may have built this application yourself or just got it as it is now. It doesn't matter, thing is - you do not have a slightest idea what is wrong here.
Java ecosystem have abundance of diagnostic tools (thank for interfaces exposed at JVM itself), but they are mostly focused on some specific narrow kinds (more...)
This post is about collecting and visualizing I/O latency histograms for NetApp
filers in C-mode.Motivations:
The drill down of I/O latency is an important technique for troubleshooting and benchmarking storage. Average latency values can hide details
of what is happening on the storage. Think for example of storage systems with flash and spindles, each serving I/O at different latency. Moreover averaging the measured values over time can hide details in case of varying (more...)
event histogram metric, a script and some related comments on collecting and displaying wait event latency histograms for Oracle performance troubleshooting.Why:
Latency histograms (and by extension wait event histograms) provide very useful information when troubleshooting performanc
e for systems exhibiting response time with multi-mode distribution. In such cases average wait values are often not sufficient to understand the behavior of the system under study and histograms
provide a finer level of details. A (more...)
This blog post is the first of a series of 4 related to time series and indexing.
According to wikipedia this is the definition of a time series:
A time series is a sequence of data points, typically consisting of successive measurements made over a time interval.
This blog post shows how we can efficiently store time series in an oracle database and how indexing strategy can have a big impact on insertion performance rate.
This is the second post in my mini-series on leveraging SQL Developer Reports for DBA tasks, today with visualizing Average Active Sessions (AAS). In this article I’ll cover What AAS is and how to interpret it How to build a basic line graph in SQL Developer How to extend the graph with detailed child reports (Time […]
Topic: PyLatencyMap v1.2
and how it can be used to produce heat map visualization of SystemTap histograms.Introduction:
When studying storage performance, the latency drill down is a very important data source. Measuring the average I/O latency is often not enough, latency histograms are proven to more suitable for investigating modern storage systems. This is because for many storage systems the response time has multiple modes: think for example of the common case of storage (more...)
If you run Oracle Standard Edition or haven’t licenced Diagnostics Pack for Enterprise Edition, then you don’t have AWR and ASH Data available. This is when Statspack, the predecessor of AWR, comes in handy to keep a history of database performance metrics. But although Oracle still deliver Statspack with their recent DB releases (yes, even in […]
Recently we had a tablespace space run out and ended up in an application failure. I have questioned my DBA and he just plainly blamed the application team members who loaded large number records without a prior notice. A convincing answer, but you can’t really fool Oracle.
Oracle introduced a new DBA_TABLESPACE_USAGE_METRICS view from 10g onwards to report the space usage with in a tablespace. I created a new tablespace and immediately (more...)
Oracle caches the data blocks in buffer cache in various modes depends on the block usage. As per the Oracle documentation it can CR (Consistent mode – reads), XCUR (Current mode – updates), FREE etc. I understand and other heard saying – whenever a block READs into memory will be in CR mode while if the block is fetching for UPDATE it will be in XCUR mode so that sessions can apply the (more...)
This post is about the latest updates to PerfSheet4
v3.7 (February 2015). PerfSheet4 is a tool aimed at DBAs and Oracle performance analysts. It provides a simplified interface to extract and visualize AWR
time series data using Excel pivot charts.
PerfSheet4 is aimed at querying and displaying time-series data from AWR repository tables. This is very a rich source of information to analyze database workloads and trends in the context of performance analysis or (more...)
If you have a sorted collection of elements, how would you find index of specific value?
"Binary search" is likely to be your answer.
Algorithms theory is teaching us what binary search is most optimal algorithm for this task with log(N) complexity.
Well, hash table can do better, if you need to find key by exact match. In many cases, though, you have reasons to have your collection sorted, not hashed.
On my job, I'm (more...)
Bind peeking is a nice feature in Oracle to have many optimized plans for an SQL for various bind values. DBAs believe that bind peeking happens during a soft parse which will identify an alternate plan. Why do I say that?
Hard Parse: Parsing first time, nothing exists to bind peek
Soft Parse : SQL cursor is existing and executing not the first time. Under the soft parse, bind peeking (more...)
This blog hasn’t seen any updates since over a year now and I do apologize for that. However, the reason for that is simple. Last year I’ve decided to take one a new project. But before I was going to tell everybody about it I wanted to make sure that I also really got the time to do so. In the meantime I got the confidence that the project will succeed and that (more...)
This is a write-up of an issue I recently posted to the OTN discussion forum (https://community.oracle.com/message/12798407). I thought the associated test case was useful in demonstrating the issue, so is captured here for future reference. There were some useful replies to the OTN post, confirming my suspicions.
The test was performed using Oracle Enterprise Edition 18.104.22.168.0 on Linux.
A parallel delete blocks insert into dependent (more...)
Yesterday, I have seen huge waits “enq SQ – contention’” – in every snapshot there were thousands of waits. But the fix was so simple! Here is the root cause of the issue –
When you select from a sequence, the NEXTVAL generated from a the seq$ table if it is not cached. If it is cached, it will be available in a memory structure and no need to generate the value (more...)