OraNA :: Oracle News Aggregator

As mentioned in the blog entry here, BI Server 11g now supports more functions which were added to support the new UI features. Each of these functions can be used as part of the Logical SQL or can even be used directly in the Answers UI. In this blog entry we shall be looking at how some of these functions can be used.

1. SELECT_PHYSICAL: Though this is not a function, this is a new feature supported by BI Server that can bypass the BMM layer in the SQL queries. So, any SQL that uses SELECT_PHYSICAL will be more closer to the ANSI SQL standard. There are 2 main advantages of this

a. Any external reporting tools that generate SQL automatically can now leverage the functionality of BI Server. For example, in the blog entry here i had shown the difficulty in actually using Logical SQL with Cognos. But introduction of SELECT_PHYSICAL should alleviate this issue thereby making BI Server open for better integration externally.

b. Using SELECT_PHYSICAL one can now make direct joins across data sources without actually having the need to model them separately in the Business Model and Mapping layer.

With the addition of more in-memory operations in the BI Server, this should act as a point of integration for external tools that have the need to leverage the metadata framework of BI EE.

To illustrate how this works, lets look at the repository shown below,

As you see, the above repository contains all the relevant tables in the SH schema. Now, to use the SELECT_PHYSICAL command, lets go to the Issue SQL section of the BI EE and fire the SQL given below

SELECT_PHYSICAL
0 s_0,
CHANNEL_CLASS s_1,
SUM(AMOUNT_SOLD) s_2
FROM
"ORCL - SH".."SH"."CHANNELS"
A,
"ORCL - SH".."SH"."SALES" B
WHERE
A.CHANNEL_ID = B.CHANNEL_ID
GROUP BY CHANNEL_CLASS

Remember, though this is a SELECT_PHYSICAL SQL statement, BI Server will still do the parsing as the objects will have to reside inside the repository.

This option can be enabled at the ODBC/JDBC level as well. That way the DSN can be configured to just accept SELECT_PHYSICAL or SELECT commands.

I will cover more details of how this can be leveraged from external tools like QlikView, Cognos etc in a future blog entry.

2. AGGREGATE AT:

This function adds more multi-level analysis capabilities within the BI Server. This function provides the same functionality as the Level Based Measures. What this does is, it provides an ability to extract the values of a parent level in a query without having to SUM up the values in the lower level. To illustrate, lets look at the simple report below

Now, what we need is an ability to display the value of yearly sales alongside the quarterly sales. To do that, we need to enter the formula as shown below

AGGREGATE("Sales"."AMOUNT_SOLD" AT "Times"."TimesDim - Calendar Year"."Calendar Year")

If we look at the report now, we should see that we can compare the Yearly sales along the Quarterly sales i.e. the AGGREGATE AT function will not respect the filter on that dimension. If you want the filters to be honored as well then the older SUM BY functions can still be used.

If you look at the Physical SQL generated, you would actually notice 2 separate WITH clauses (minimum). One for the Sales figure. And the other for the Yearly Sales figure. The Yearly Sales figure will not have the filter applied.

WITH
SAWITH0 AS (select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3,
     D1.c4 as c4,
     D1.c5 as c5
from
     (select sum(T44515.AMOUNT_SOLD) as c1,
               T44523.CALENDAR_QUARTER_DESC as c2,
               T44523.CALENDAR_QUARTER_ID as c3,
               T44523.CALENDAR_YEAR as c4,
               T44523.CALENDAR_YEAR_ID as c5,
               ROW_NUMBER() OVER
(PARTITION BY T44523.CALENDAR_QUARTER_ID
ORDER BY T44523.CALENDAR_QUARTER_ID ASC) as c6
          from
               TIMES T44523,
               SALES T44515
          where  ( T44515.TIME_ID = T44523.TIME_ID and
(T44523.CALENDAR_QUARTER_NUMBER in (1, 2)) )
          group by T44523.CALENDAR_QUARTER_DESC,
T44523.CALENDAR_QUARTER_ID, T44523.CALENDAR_YEAR,
T44523.CALENDAR_YEAR_ID
     ) D1
where  ( D1.c6 = 1 ) ),
SAWITH1 AS (select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3
from
     (select sum(T44515.AMOUNT_SOLD) as c1,
               T44523.CALENDAR_YEAR as c2,
               T44523.CALENDAR_YEAR_ID as c3,
               ROW_NUMBER() OVER (PARTITION BY
T44523.CALENDAR_YEAR_ID ORDER BY T44523.CALENDAR_YEAR_ID ASC) as c4
          from
               TIMES T44523,
               SALES T44515
          where  ( T44515.TIME_ID = T44523.TIME_ID )
          group by T44523.CALENDAR_YEAR, T44523.CALENDAR_YEAR_ID
     ) D1
where  ( D1.c4 = 1 ) ),
SAWITH2 AS (select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3,
     D1.c4 as c4,
     D1.c5 as c5,
     D1.c6 as c6
from
     (select 0 as c1,
               D1.c2 as c2,
               case  when D2.c2 is not null then D2.c2
when D1.c4 is not null then D1.c4 end  as c3,
               D1.c1 as c4,
               D2.c1 as c5,
               D1.c3 as c6,
               ROW_NUMBER() OVER (PARTITION BY D1.c2, D1.c3,
case  when D2.c2 is not null then D2.c2 when D1.c4 is not null
then D1.c4 end  ORDER BY D1.c2 ASC, D1.c3 ASC,
case  when D2.c2 is not null then D2.c2
when D1.c4 is not null then D1.c4 end  ASC) as c7
          from
               SAWITH0 D1 left outer join SAWITH1 D2 On D1.c5 = D2.c3
     ) D1
where  ( D1.c7 = 1 ) )
select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3,
     D1.c4 as c4,
     D1.c5 as c5
from
     SAWITH2 D1
order by c1, c3 NULLS FIRST, c2

3. CALCULATEDMEMBER:

This function is used to derive calculated members using members at any level of a hierarchy. In 10g, we can do this from a pivot table. In 11g, this function has been introduced at the BI Server level itself so that we can leverage these directly from the repository.

To illustrate this, consider the hierarchy shown below

In this hierarchy, we want to find out the difference between Steven King’s Salary and his 2 direct reportees (Neena Kochhar and Lex De Haan). To do this, we need to use a function given below

CALCULATEDMEMBER("HR"."Employee"."EmployeeDim",'Test', MEMBER(100) - MEMBER(101) -MEMBER(102))

This would push the function directly on the parent-child hierarchy and give us the correct values. This function cannot be used if there are other attributes of the same dimension in the report. This is generally used by external application tools. But within BI EE Answers, there is no need for using this as we can achieve the same using Custom Groups and Calculations.

4. EVALUATE_ANALYTIC:

This is another significant new feature that can be useful to push down the oracle database analytic functions. I have covered EVALUATE number of times before for function shipping database functions. EVALUATE_ANALYTIC extends that further and provides us an ability to function ship the analytic functions as well. The syntax of this function remains the same. I will cover this in detail in another blog post as there are quite a few use cases where EVALUATE_ANALYTIC can prove to be very useful.

5. HIERARCHICAL FUNCTIONS:

BI Server now supports hierarchical functions like ISCHILD, ISROOT, ISANCESTOR, ISDESCENDANT & ISLEAF that can be used to traverse a Parent-Child hierarchy. These functions generally return a boolean data type (true/false) and hence can be used in filters to restrict data. They can also be used in column formula case statements.

For example, in a hierarchy if we want to know whether a specific member is a leaf node or not, we can use the below function to display Yes/No as an attribute of the parent-child hierarchy

CASE WHEN ISLEAF(Employee.EmployeeDim) THEN 'Yes' ELSE 'No' END

Next up is some interesting posts on the BI EE integration with Fusion Middleware & Essbase. I will be doing these posts towards the end of next week.

As Mark had pointed out earlier here, one of the significant changes in the BI EE 11g release is the migration of security from the BI EE repository to Weblogic. Users and Groups will no more be maintained within the rpd. Even in an external authentication like LDAP, Groups will not be maintained within the repository. This is good in many ways as it can significantly reduce the size of the repository. The downside though is, in offline mode unless there is a change in any security object (like custom filters to a group etc), the users & groups will not become visible. The use of Web Catalog Groups is no more recommended. It still exists primarily for supporting upgrades. In this blog post, lets see how the new weblogic based security works.

There is a terminology change that we need to get used to while moving to BI EE 11g. Groups now refer to Weblogic groups. BI EE specific groups are called as Application Roles. The figure shown below explains the relationship between Users, Groups & Application Roles.

Weblogic security is much more comprehensive and it can support multiple external directories like LDAP, OID etc. It also supports table authentication. So, if you are planning on setting up external authentication, its preferred to set it up at the Weblogic layer. We can also setup external directories within BI EE (like in 10g).

To start with lets create a simple user bieeuser1 in the Weblogic Console Security Realm (http://localhost:7001/console).

While creating a user we can assign it to a provider. A provider is the authentication directory against which users will be authenticated. In our case, we have created the user in the default weblogic directory. In the same way lets create 2 more users bieerpdadmin & bieewebcatadmin.

The idea is to provide the following privileges to each of the users that we have created

bieeuser1 – Casual user with no access to the repository but with read only access to dashboards (no answers)
bieerpdadmin – Repository Administrator with full control of the Repository objects but with no control on the web catalog.
bieewebcatadmin – Web Catalog Administrator with full control on the web catalog but with no access to the repository

Remember, its not always necessary to assign users to weblogic groups. Users can be directly assigned to application roles. Once the users are created, lets login to the Weblogic enterprise manager and navigate to the coreapplication under the Business Intelligence folder.

Now, right click on the coreapplication and choose the Security->Application Roles option

This should list all the default application roles available within BI EE.

Let’s neglect the existing roles(ideally we will be reusing/reassigning the default roles as much as possible) and create 3 new roles.

1. BIEERoleUser
2. BIEERoleRPDAdmin
3. BIEERoleWebCatAdmin

For now lets not add any users/roles to these roles.

Lets now navigate to the Application Policies and start granting specific rights/grants to the roles that we created. Remember in any ADF application we need to define application policies so that they can be granted to individual users. We start with assigning the following privileges to the BIEERoleUser

1. resourceType=oracle.bi.server.permission,resourceName=oracle.bi.server.queryUserPopulation

This basically provides an ability to query the BI Server repository.

Lets assign this privilege to the BIEERoleUser application role.

In the same way lets assign the following privileges to the BIEERoleRPDAdmin and BIEERoleWebCatAdmin

For BIEERoleRPDAdmin

1. resourceType=oracle.bi.server.permission,resourceName=oracle.bi.server.ManageRepositories

For BIEERoleWebCatAdmin

1. resourceType=oracle.bi.server.permission,resourceName=oracle.bi.server.queryUserPopulation

Now that the roles have the right privileges, lets go back to the Application Role screen and start assigning the 3 users to each of these roles.

After assigning these, if you try to login to the repository in online mode, only the user bieerpdadmin can login. We have satisfied one requirement, that is we have created a user who is the repository admin. For presentation services, by default during install the privileges in presentation services get assigned to only 3 default groups BIAdministrator, BIAuthor and BIConsumer. So, to make bieewebcatadmin a true webcatalog administrator we have to go to the Application Roles and add the BIAuthor & BIAdministrator role to BIEERoleWebCatAdmin

This will automatically make the user bieewebcatadmin user to be the administrator of the webcatalog. Initially it is mandatory to assign a new role to the BIAdministrator role as that is pre-configured to be the only role having access to Privileges in Presentation Services. Granting BIAdministrator to a role also opens up the access to the repository. This can be disabled as well. It is not necessary to always create new roles and assign policies to them. Existing roles can be re-used and re-assigned. The entire security model is now more aligned with Oracle Platform Security framework so that all the fusion applications can integrate with each other seamlessly.

One of the important points to consider when moving from BI EE 10g to 11g is in understanding the BI Server behavior changes. Though there are quite a lot of changes in the underlying code for performance improvement etc, introduction of new features in 11g can potentially alter your generated queries in 11g. In this blog entry i will basically cover 2 main important changes that have been introduced in 11g. There are more actually, but that is something we will try to cover in our training days event.

1. Level Based Measures at Detail Level:

One significant change in 11g is the handling of level based measures at the detail level. For example, lets consider a measure AMOUNT_SOLD that is assigned to the detail day level of the date dimension.

In 10g, whenever we bring this column for reporting, we would basically see something as shown below i.e. the level will always be enforced.

In the case of 11g, there are 2 types of behavior

a. If the report contains normal attribute columns and when the query is at a grain higher than the measure grain (say the report is at the Year level), then normal aggregation will be applied and the level will not be enforced like in 10g. An example screenshot is given below

b. If the report contains hierarchical columns, then if the query is at higher grain than the measure, the measure values will become null. The values will appear at that grain or below. An example screenshot is given below

2. Priority Groups & Order of LTS:

In 11g, there is a new concept called Priority Groups. In 10g, the LTS that determines the final query is dependent on the ordering of the LTS in the dimension logical table(in most cases – but there are cases where we cannot be always be sure to guarantee this behavior). But in 11g, this is determined by the priority order set in each and every LTS.

For example, lets consider a case where there are 2 aliases for the same dimension CHANNELS as shown below

In the business model and mapping layer, lets assume that both the LTS are mapped to all the columns. Both the LTS are at the same detail level. Now, lets assign a priority group of 0 for the Channel Alias – B LTS and a priority group of 1 for the Channel Alias – A LTS. Also the SALES LTS of the Fact table will also be assigned to the priority group 0. In 11g, the LTS with the least number is considered as the one that has to be chosen for querying (highest priority)

Now, lets create a report containing columns from Channels dimension and the Sales fact.

If you look at the query of the report, you will notice that Channels Alias – B will be chosen as it is the one having the highest priority.

Lets now go back to the repository and reassign the highest the priority to Channels Alias – A as shown below

For the same report, if you look at the query now, you will notice that Channels Alias – A will be chosen instead of the B alias.

Mark has already explained the new Parent-Child hierarchy new feature in 11g here. Parent Child hierarchies are different from normal level based hierarchies not only in the nature of their structure but also in the way aggregations are done. For example, in a level based hierarchy, the most common type of aggregation that is done involves rolling up the numbers from the lower most level. But in the case of parent-child hierarchy, an aggregation by default enforces the roll-up across all descendant members. To understand this lets try to take this through an example.

Lets consider a sales department with the employee hierarchy shown below

Its a simple parent child employee hierarchy found in the oracle HR schema. Now the requirement for us is to model the Salary attribute of an employee in 3 ways

1. As an attribute – Show the salary of each employee as an attribute in the parent child hierarchy.
2. As a measure (All intermediary Roll-ups) – Roll-up the salary figures of all the employees who roll into the employee. For example, Scott King will have a Salary equal to sum of salaries of all the employees who report to him, including the ones who are his direct & in-direct reporting employees.
3. As a measure (Level-0 Roll-up) – Roll-up the salary figures from the lower most level till the top node For example, in the above figure, Scott King will have a Salary Roll-up figure of all the employees who are the lowermost level (not including the intermediary nodes – very similar to normal level based hierarchy roll-ups)

The above requirement is quite straightforward, but to achieve similar such aggregations in a normal level-based hierarchy, we will have introduce fragmentation & other techniques. But with parent-child hierarchy modeling now available in BI EE, all the above 3 requirements can quite easily be achieved.

Following is the employees parent child table that stores all the attributes of an employee

We start off with modeling this as a normal dimension table. As shown in Mark’s blog entry, we then create the Parent-Child table that stores all the relationships within the employee hierarchy (if you look at the script used in creating this, it will be containing a Connect By statement that will connect a root hierarchical node to every member that is part of its hierarchy).

Once this is created, we start with including the Salary as an attribute within the Employee dimension itself

This will serve our first requirement i.e show salary as an attribute of every employee. Now, to model the 2nd requirement i.e. to model salary as a measure and then make it to rollup from all the intermediary hierarchical members, we start with defining an alias on the main Employee table. Then join this table to the custom parent child employee table. The join structure of all the 3 tables is given below

We would then be using the salary column from the new employee alias table in our logical fact table as shown below

Lets now look at the report now and see what BI EE generates.

As you see, BI EE has generated the correct results with the roll-up of intermediary members as well. This is something that is quite difficult to model in BI EE 10g. If we look at the query generated (just the key one that generates the measure as BI EE now generates lots of intermediary sub-queries to support contextual drilling) you would notice that the default Parent-Child behavior is to achieve a roll-up across a hierarchy

To achieve the 3rd requirement i.e. to model a level based hierarchy kind of rollups from level-0, we create another alias in the physical layer and then model it the same way as the 2nd requirement. In the fact logical table, we then create a new measure that will map to the new physical alias as shown below

To ensure that we roll-up only from level-0 members, we then apply a filter in the fact logical table source as shown below

Lets now take a look at what BI EE generates

As you see, BI EE now generates the roll-ups only from level-0 members as against every descendant. We can have even more granular roll-ups based on custom defined attributes.

BI EE 11g is a significant feature filled release when compared with the previous 10g release. Most customers who are planning to upgrade to 11g have an automated way of upgrading their 10g Web Catalogs and repositories to the 11g version. When moving from 10g to 11g, we will have to go through the upgrade utility as there are significant changes to the repository structure and the web catalog structures. A direct copy & paste into the 11g version will not work.

The upgrade process involves the following steps

1. Install the new 11g version in either the same machine or a different machine.
2. Run the upgrade assistant utility to upgrade the repository & Web Catalogs.
3. Migrate other 10g specific customizations manually to the 11g instance. The different customizations are

a. Any Static files added to the 10g app server
b. Custom XML messages
c. Styles & Skins (The upgrades for this will have to be done carefully as some CSS & files have changed in this release)

4. Upgrade the Scheduler schema (Both BI Delivers & BI Publisher) through the upgrade utility
5. Upgrade BI Publisher

In this blog entry we shall see how to go about upgrading the RPD and Web Catalogs from the 10g to the 11g version. After installing 11g, start up the upgrade assistant utility from {ORACLE_HOME}/bin/ua (or ua.bat if you are on windows)

The upgrade assistant can be used to upgrade the scheduler schema, BI Publisher as well as the RPD & Web Catalogs. Lets start with the first option, i.e. upgrading RPD and Web Catalogs

In the upgrade screen, choose the 10g RPD along with the Web Catalog Path. In the Deliveries folder, choose the 10g deliveries path ({OracleBIData}/web/catalog/deliveries

In the next screen, provide the details of the weblogic server on which we will be deploying the Repository and Web Catalog to.

The upgrade utility does not seem to support remote deployments of the RPD and Web Catalog i.e. Upgrade Utility will only deploy the repository and web catalog on the machine containing the weblogic admin server.

The upgrade process will stop all the components first and then will do the individual RPD and Web Catalog Upgrade.

This is just a utility. We cannot expect all the reports to start working out of the box immediately after an upgrade. Good amount of testing is required to make sure all the older queries of the most commonly used reports are working without any major issues. Also, not all the reports would get migrated seamlessly due to the significant changes across the versions which the utility cannot recognize. Care has to be taken to make sure that all the warnings logged as part of the upgrade process are properly analyzed and corrected manually. A sample upgrade log is shown below (containing 4 warnings)

One important new feature of BI EE 11g is the ability to report on CLOBs and other large objects. The main difference between normal Varchar/Numeric columns and LOB datatype columns is, LOB columns do not support GROUP BY, DISTINCT & ORDER BY operations on them (from a reporting perspective). Any query that BI EE generates will always have a GROUP BY or a DISTINCT in it. So, in 10g reporting on LOB columns have always been an issue(will result in Ora-XXXX errors). One had to resort to different types of workarounds for reporting on LOB columns.

In BI EE 11g, its possible to isolate different columns from the GROUP BY/DISTINCT operations by using the SPARSE/DENSE lookup operators. I have already blogged about how Sparse & Dense lookups work in general here. Lets see how the same can be used for CLOB reporting. To demonstrate this i shall be using the v$sql view. This view basically contains a CLOB called SQL_FULLTEXT. This column contains the SQL queries stored in the form of CLOB, that are fired to the Oracle database by multiple users. Our aim is to report on this view and display the SQL Queries from the CLOB column.

We start with granting select on the v_$sql view to SH schema and then importing it in BI EE.

Then edit the column properties of the SQL_FULLTEXT column and increase the size to 100000 (or higher) so that BI EE can display the values from CLOB.

Start with modeling a dimension and a fact in the BMM layer as shown below

Then create another logical table (same as the dimension) containing SQL_ID as the primary key. The idea is to use this primary key and show the SQL_FULLTEXT using a DENSE lookup. Mark the new logical table as a lookup table.

In the dimension logical table create a new column and use the following formula to derive the SQL_FULLTEXT as a lookup column. This will always push a separate SQL without DISTINCT/GROUP BYs on the SQL_FULLTEXT column.

Lookup
(DENSE
"LOB Reporting"."Lookup - CLOB"."SQL Full Text" ,
"LOB Reporting"."Lookup - CLOB"."SQL ID" )

Lets now include this new column in the presentation layer and use it for reporting.

As you see, we should now be able to see the CLOB data without any restrictions. If we look at the query generated, you can basically see 2 separate SQL queries, one with the group-by and the other with just the lookup.

select sum(T43900.ROWS_PROCESSED) as c1,
     T43900.SQL_ID as c2
from
     SYS.V_$SQL T43900
group by T43900.SQL_ID

select T43900.SQL_FULLTEXT as c1,
     T43900.SQL_ID as c2
from
     SYS.V_$SQL T43900

In BI EE 11g, usage tracking can now log logical sql’s greater than 4000 characters into a separate CLOB column.