Machine Learning Pipelines for High Energy Physics Using Apache Spark with BigDL and Analytics Zoo

Topic: This post describes a data pipeline for a machine learning task of interest in high energy physics: building a particle classifier to improve event selection at the particle detectors. The pipeline is built using tools from the "Big Data ecosystem", notably Apache Spark, BigDL and Analytics Zoo. The work is accompanied by a set of notebooks with code and sample data.

The Physics use case

High Energy Physics (HEP) is a data-intensive (more...)

Solving DBFS UnMounting Issue

Often, I am quite baffle with Oracle’s implementations and documentations.

RAC GoldenGate DBFS implementation has been a nightmare and here is one example DBFS Nightmare

I am about to show you another.

In general, I find any implementation using ACTION_SCRIPT is good in theory, bad in practice, but I digress.

Getting ready to shutdown CRS for system patching to find out CRS failed to shutdown.

# crsctl stop crs
CRS-2675: Stop of 'dbfs_mount' on 'host02'  (more...)

Power BI report of Azure resources

As our use of Azure is continually expanding so is our requirement for effective reporting and governance of that. However there still seem to be some gaps into what is actually possible (unless i missed something - which is entirely possible as the platform seems to change on a daily basis.....).

For example if i want to directly report on azure resources from Power BI i can't natively do that - there doesn't seem (more...)

Gestern ging’s noch

Brent Ozar hat in seinem Blog eine Liste mit 15 Gründen veröffentlicht, die dazu führen können, dass eine Query im SQL Server plötzlich langsamer wird, als sie das bisher gewesen war - und in den Kommentaren wurden noch ein paar weitere Begründungen aufgeführt. Vor ein paar Jahren hatte Jonathan Lewis eine ähnliche Liste mit 20 Punkten für Oracle erstellt, für die mir Hemant Chitale via Twitter den passenden Link geliefert hat. Zu den aufgeführten Gründen (more...)

WIT and Language

I’ve been thinking about this post for a while, but it took me some time to write it because this is not the kind of thing I usually write about. I really support WIT. During my years in tech I’ve worked with many men and women, and I can’t say that one gender is “better” … Continue reading "WIT and Language"

Build it Yourself — Chatbot API with Keras/TensorFlow Model

Is not that complex to build your own chatbot (or assistant, this word is a new trendy term for chatbot) as you may think. Various chatbot platforms are using classification models to recognize user intent. While obviously, you get a strong heads-up when building a chatbot on top of the existing platform, it never hurts to study the background concepts and try to build it yourself. Why not use a similar model yourself. Chatbot implementation (more...)

Test Remote Asynchronous Request Response services

A few years ago, I described how you can test Asynchronous Request Response services.

The thing with Asynchronous Request Response services is, as I used to describe it, that they're in essence two complementary Request-Only (Fire and Forget) services. That is, the client submits a request to the Asynchronous Request Response service, and at a certain point waits for the response by listening to an endpoint.

To make this work, the responding Asynchronous Request Response (more...)

Machine Learning Pipelines for High Energy Physics Using Apache Spark with BigDL and Analytics Zoo

Topic: This post describes a data pipeline for a machine learning task of interest in high energy physics: building a particle classifi

Take care with automatic indexes

This one came in via an AskTOM question. You need to be careful when disabling constraints that are underpinned by an index, especially when it comes to the storage for that index.

Let’s take a look at a simple example. I’ll explicitly nominate the LARGETS tablespace for my index that will support the primary key.

SQL> create table t as
  2  select 1 x, 1 y from dual;

Table created.

SQL> create unique index t_pk  (more...)

Highlight ‘current’ card in APEX

Earlier today I posted a solution that required a bit of jQuery to execute after refresh of a report.

Avoiding post-processing is always nice, so here's an example that is resolved during render.

I would like to highlight the 'current' card in this classic report:

Cards Classic Report

The SQL that dynamically defines these cards includes the following column, with some decider over which record is deemed the current selection.
,case when :P17_ITEM = 'SOMETHING'  (more...)

APEX Classic Report as Alerts

I love classic reports.
You can make your data look like anything, out-of-the-box, as previously highlighted by Christina and Carsten.

When you create a classic report, you can select different Report templates via the region attributes. I use the Cards and Alerts a fair bit, allowing me to produce data driven content, instead of whatever I type in the application builder.

This has been the general premise of APEX templates since the beginning. We define (more...)

Grab all the DDL

I posted a video a couple of days ago showing a trigger mechanism to customize the capture of DDL that is issued on your database. The aim here is to be more generous with letting developers execute DDL on their Development databases, whilst still having a thorough record of the changes that are occurring. Of course, it goes without saying, which is why I am saying it 🙂 that this is not a replacement for good (more...)

Bloom Filter Efficiency And Cardinality Estimates

I've recently came across an interesting observation I've not seen documented yet, so I'm publishing a simple example here to demonstrate the issue.

In principle it looks like that the efficiency of Bloom Filter operations are dependent on the cardinality estimates. This means that in particular cardinality under-estimates of the optimizer can make a dramatic difference how efficient a corresponding Bloom Filter operation based on such a cardinality estimate will work at runtime. Since Bloom (more...)

Chinar Aliyev’s Blog

Chinar Aliyev has recently started to pick up on several of my blog posts regarding Parallel Execution and the corresponding new features introduced in Oracle 12c.

It is good to see that obviously Oracle has since then improved some of these and added new ones as well.

Here are some links to the corresponding posts:

New automatic Parallel Outer Join Null Handling in 18c

Improvements regarding automatic parallel distribution skew handling in 18c

Chinar has (more...)

Last Week in Stream Data Integration & Stream Analytics – 23.4.2019

This is the 154th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

Parse Calls

When dealing with the library cache / shared pool it’s always worth checking from time to time to see if a new version of Oracle has changed any of the statistics you rely on as indicators of potential problems. Today is also (coincidentally) a day when comments about “parses” and “parse calls” entered my field of vision from two different directions. I’ve tweeted out references to a couple of quirkly little posts I did some (more...)

Using Vagrant and shell scripts to further automate setting up my demo environment from scratch, including ElasticSearch, Fluentd and Kibana (EFK) within Minikube

For training and demo purposes, on my windows laptop, I needed an environment with a guest Operating System, Docker and Minikube available within an Oracle VirtualBox appliance.

So, in a series of articles up till now, I described the following:

Follow up: Upgrading Enterprise Manager 13.2 to 13.3 – missing ‘Update’ Option in GUI

First of all, thanks a lot to Martin Decker for his comment and the description how he could solve that problem. Tried that, but unfortunately, these changes alone did not help. I opened a SR and got a recommendation how to solve the problem, exactly designed for my environment. Beside the tasks Martin described, I had to edit some libraries in the directory tree <OMS_HOME>/inventory/featuresets/. 

I asked Oracle Support if I'm allowed to publish the (more...)

Video : LISTAGG Analytic Function in Oracle Database

Today’s video is a run through the LISTAGG analytic function, with a little included 19c goodness.

If video is not your thing, you can get all the relevant information from these articles.

The star of today’s video is Michael Rainey. 🙂



Video : LISTAGG Analytic Function in Oracle Database was (more...)

Recover table in oracle 12c

Oracle 12.2 came with a new feature , we can simply recover a single table in a database.

In below example we will see how we can restore a table of pluggable database.

Step 1 : Connect to RMAN

[oracle@dbatesting ~]$ rman target /

Recovery Manager: Release - Production on Thu Feb 28 20:38:23 2019

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

connected to target  (more...)