this post is about Linux perf and uprobes for tracing and profiling Oracle workloads for advanced troubleshooting.Context
The recent progress and maturity of some of the Linux dynamic tracing tools has raised interest in applying these techniques to Oracle troubleshooting and performance investigations. See Brendan Gregg
's web pages for summary and future developments on dynamic traces for Linux. Some recent work on applying these tools and techniques to Oracle can be found (more...)
Stack trace sampling is very powerful technique for performance troubleshooting. Advantages of stack trace sampling are
- it doesn't require upfront configuration
- cost added by sampling is small and controllable
- it is easy to compare analysis result from different experiments
Unfortunately, tools offered for stack trace analysis by stock profilers are very limited.
Solving performance problem in complex applications (a lot of business logic etc) is one of my regular challenges. Let's assume I have another (more...)
Whenever you use PL/SQL in SQL statements, the Oracle engine needs to switch from doing SQL to doing PL/SQL, and switch back after it is done. Generally, this is called a “context switch”. This is an example of that:
-- A function that uses PL/SQL
create or replace function add_one( value number ) return number is
l_value number(10):= value;
-- A SQL statement that uses the PL/SQL function
select sum(add_one(id)) (more...)
It's probably not that well known that Oracle can actually rollback / re-start the execution of a DML statement should the cursor become invalidated. By rollback / re-start I mean that Oracle actually performs a statement level rollback (so any modification already performed by that statement until that point gets rolled back), performs another optimization phase
of the statement on re-start (due to the invalidation) and begins the execution of the statement from scratch. Note (more...)
There’s been a lot of work in the area of profiling. One of the things I have recently fallen in love with is Brendan Gregg’s flamegraphs. I work mainly on Linux, which means I use perf for generating stack traces. Luca Canali put a lot of effort in generating extended stack profiling methods, including kernel (only) stack traces and CPU state, reading the wait interface via direct SGA reading and kernel stack traces and getting (more...)
"The only thing you can do easily is be wrong, and that's hardly worth the effort."
Norton Juster, The Phantom Tollbooth
Oracle can lie to you. Not like a disreputable uwed-car salesman but more like the ‘little white lie’ sometimes told in order to hide less-than-desirable parts of the truth. And it’s not Oracle, really, it’s the optimizer and it does it by reporting query plans that may not accurtely report the execution path. (more...)
This post provides a short summary and pointers to previous work on Extended Stack Profiling for troubleshooting and performance investigations.Understanding
the workload is an important part of troubleshooting activities. We seek answers to questions like: what is the system doing, where is the time spent
, which code
paths are most used, what are the wait events, etc. Sometimes the relevant diagnostic data is easy to find, other times we need to dig (more...)
A new version 4.22 of the XPLAN_ASH utility is available for download.As usual the latest version can be downloaded here.
This version primarily addresses an issue with 12c - if the HIST mode got used to pull ASH information from AWR in 12c it turned out that Oracle forgot to add the new "DELTA_READ_MEM_BYTES" columns to DBA_HIST_ACTIVE_SESS_HISTORY - although it got officially added to V$ACTIVE_SESSION_HISTORY in 12c. So now I had to implement (more...)
This post is about manually calling and freeing a shared latch. Credits should go to Andrey Nikolaev, who has this covered in his presentation which was presented at UKOUG Tech 15. I am very sorry to see I did miss it.
Essentially, if you follow my Oracle 12 and shared latches part 2 blogpost, which is about shared latches, I showed how to get a shared latch:
SQL> oradebug setmypid
SQL> oradebug (more...)
It is widely known that unindexed foreign keys can be performance issue. Unindexed foreign keys on child tables can cause table locks or performance problems in general.
There are many articles on this subject so I won't go in details.
My plan is to show simple demo case where empty child table with unindexed foreign key column can cause big problems.
Imagine that you have highly active table (supplier) with lots DML operations from many (more...)
Today here’s a shorter post about my experiments with Oracle SQL Developer’s user-defined reports: A report on all long running operations (“LongOps”) with details on session wait events, explain plans and live SQL monitoring. “Wait a minute”, you might say, “there’s already the session report in SQL Developer’s standard reports that shows Session_LongOps”! – and you’re […]
this post is about investigating Oracle wait events using stack profiles and flame graphs extended with OS-process state and Oracle wait event details.Context: The case of the DB Time > CPU Time + Wait Time
provides wait event and CPU time accounting, a powerful and readily accessible data source for performance troubleshooting. An Oracle session at a given point in time is either on CPU, for example when processing data from cache, (more...)
Oracle week is one of the biggest Oracle related events in Israel which has going on yearly for the past 22 years (!).
It’s a 5 days long event with about 2000 participants in total over 80 different day-long seminars.
The seminars are around 8.5 hours (9am to 4:30pm) and are basically a day long course on various Oracle technologies which is nothing like the 45-90 minutes sessions I’m used to in the (more...)
This article is about the Oracle 12c in-memory option, and specifically looks at how the background worker processes do IO to populate the in-memory column store.
Hardware: Apple Macbook with VMWare Fusion 7.1.3.
Operating system: Oracle Linux 6.7, kernel: 3.8.13-118.el6uek.x86_64.
Database version: Oracle 188.8.131.52
Patch: opatch lspatches
19392604;OCW PATCH SET UPDATE : 184.108.40.206.1 (19392604)
19303936;Database Patch Set Update : 12.1. (more...)
While preparing to teach a class this week I have had the time to look into the In Memory (cost) option a bit closer. I noticed a few interesting things along the way and would like to share one of these here.
One of the questions I was asking myself was:
“What happens if I scan segments that are within the IM area, and some are not?”
I was primarily thinking of joins in (more...)
In my last post I demonstrated an example how to use a PL/SQL package to monitor a workload in order to identify correlated columns. Such columns are good candidates for the creation of extended statistics since the optimiser does not assume that information stored in different columns may be related. Before starting my tests I invoked DBMS_STATS.SEED_COL_USAGE, which in turn increased Oracle’s attention level trying to find correlated columns. Eventually these have been identified (more...)
This blog post is about kernel stack profiling and visualization with flame graphs with examples.
Stack profiling and flame graphs
are very useful techniques and tools for troubleshooting and investigating workloads at the OS-level and understand which code path take most of the execution time. You can find extensive material and examples o flame graphs in Brendan Gregg
's website and blog. A few additional examples of using stack tracing and flame graphs for (more...)
When you are migrating to Oracle 12c I hope you might this post useful. I came across this feature when researching what’s new with Oracle 12c (and yes I still find lots of new ones I haven’t noticed before). This one is a bit hidden away in section 220.127.116.11 Automatic Column Group Detection of the 12c New Features Guide. And it’s a lot more complex than I first thought! In this first post (more...)
Recently, I have discovered a fancy bug affecting few version of Linux kernel. Without any warnings JVM just hangs in GC pause forever. Root cause is a improper memory access in kernel code. This post by Gil Tene gives a good technical explanation with deep emotional coloring.
While this bug is not JVM specific, there are few other multithreaded processes you can find on typical Linux box.
This recent bug make me remember few other (more...)