OraNA :: Oracle News Aggregator

On one of our cloned 11i Instance, I faced issues while trying to start Database Control (aka Enterprise Manager Console) using “emctl start dbconsole“.
This post covers issues and resolution (Issues mentioned here are specific to particular Database and Operating System Version)
.

Instance Details

Operating System : Solaris Sparc 64 Bit
Database : Oracle Database Version 10.2.0.2
Apps Version : 11.5.10.2

.
Things good to know about Enterprise Manager

Database Control : is Web-based tool to manage database. Using Database Control (OEM) you can perform administrative task like adding tablespace, table, index, managed space or view performance statistics…

Files/Directory for Database Control (DB Console) :
a) $ORACLE_HOME/sysman
b) $ORACLE_HOME/[hostname_sid]/sysman
c) $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_[hostname]_[sid]

Files useful in troubleshooting : $ORACLE_HOME/[hostname_sid]/sysman/log/

How to use Database Control

a) ORACLE_HOME/bin/emctl start dbconsole  [To start DB Control]
b) ORACLE_HOME/bin/emctl status dbconsole [To check status of DB Control]
c) ORACLE_HOME/bin/emctl stop dbconsole   [To stop DB Control]

Issue 1 : On running “emctl start dbconsole“, I received error message

OC4J Configuration issue. $OH/oc4j/j2ee/OC4J_DBConsole_$hostname_$sid not found

This means enterprise manager was not configured and Fix was to configure enterprise manager using emca (Enterprise Manger Configuration Assistant).
Steps to create enterprise manager configuration files & repository are explained in “Oracle Enterprise Manager Advanced Configuration Guide“  here

.

How to create Database Control(DB Console)

emca (Enterprise Manager Configuration Assistant) is utility to create Database Control Configuration files and Repository Objects

For 10.2.X Database I used
emca -config dbcontrol db -repos create

.

Issue 2 : While running emca, I got below error message

UnsatisfiedLinkError exception loading native library: njni10
Exception in thread “main” java.lang.UnsatisfiedLinkError: get
at oracle.net.common.NetGetEnv.get(Native Method)
at oracle.sysman. assistants. util.sqlEngine. SQLEngine. getEnvParams (SQLEngine.java:409)

as per Metalink note 553154.1, Fix was to set environment variable and relink executable

Fix 2 :
export LD_LIBRARY_PATH=$ORACLE_HOME/lib32
export LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
cd $ORACLE_HOME/network/lib
make -f ins_net_client.mk client_sharedlib
relink all

Issue 3 : While running “relink all“, there was another error

ld: fatal: library -ljava: not found
ld: fatal: library -ljvm: not found
ld: fatal: File processing errors. No output written to $OH/10.2.0/sysman/lib/libnmemso.so
*** Error code 1
make: Fatal error: Command failed for target `$OH/10.2.0/sysman/lib/libnmemso.so’
Current working directory $OH/10.2.0/sysman/lib
*** Error code 1
make: Fatal error: Command failed for target `libnmemso’
Fix 3 : As per Metalink note 743833.1, issue was because there are references of source system in make file and fix was to change references from source to target instance. (DB was cloned as part of 11i Cloning steps)

cp $ORACLE_HOME/sysman/lib/env_sysman.mk $ORACLE_HOME/sysman/lib/env_sysman.mk.bak

change
JRE_LIB_DIR=$SOURCE_ORACLE_HOME/10.2.0/jdk/jre/lib/sparc  (It was pointing to source Directory)
to
JRE_LIB_DIR=$TARGET_ORACLE_HOME/10.2.0/jdk/jre/lib/sparc

and relink again using “relink all”
.

Issue 4 : Just when I was expecting all issues to be resolved by now I got another (different) error message while  running “relink all”

ld: warning: file $OH/10.2.0/lib/libclntsh.so.10.1: wrong ELF class: ELFCLASS64
Undefined                       first referenced
symbol                             in file
ztch                              $OH/10.2.0/lib32//libnmemso.so

ld: fatal: Symbol referencing errors. No output written to $OH/10.2.0/sysman/lib/emagent
*** Error code 1
make: Fatal error: Command failed for target `$OH/10.2.0/sysman/lib/emagent’
Current working directory $OH/10.2.0/sysman/lib
*** Error code 1
make: Fatal error: Command failed for target `emagent’
ld libclntsh.so.10.1 wrong ELF class ELFCLASS64

Fix 4 : another note from Metalink 395731.1, and fix was to create soft link for file “ldflags” like

ln -s $ORACLE_HOME/lib/ldflags $ORACLE_HOME/lib32/ldflags

After creating softlink, I did manage to relink all executable in ORACLE_HOME successfully and it was time to run emca for enterprise manager configuration

“emca -config dbcontrol db -repos create”

.

Issue 5 : While running above emca command, scripting was prompting for dbsnmp password again and again stating

“Invalid username/password” (Even though I could connect to dbsnmp/ dbsnmp_passsword from sqlplus)

Error message 5 : On checking emca logs at  $OH/cfgtoollogs/emca/ emca_$timestamp.log

oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-01034: ORACLE not available

.
Fix 6 : It was time to search in metalink again and this time I got note # 337260.1 according to this note issue was because of user profile (password limit and resource limit)

SQL> select u.username, u.profile, p.resource_name, p.limit
from dba_profiles p, dba_users u
where p.profile=u.profile
and u.username in (’DBSNMP’)
and p.resource_type = ‘PASSWORD’
order by u.username, p.resource_name;

Profile for user dbsnmp was set according to metalink 337260.1 which means issue was something else, on doing some hit & trial I realized I was using SID in uppercase where as in database listener.ora SID was in lower case.

When I enetered SID name in lowercase, emca script accepted dbsnmp password and finally after 3 hours I managed to install/configure enterprise manager console on cloned target 11i Database Instance.

30-Oct-2008 16:51:05 oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://machinename:<port_number>/em <<<<<<<<<<<
Enterprise Manager configuration completed successfully
FINISHED EMCA at 30-Oct-2008 16:51:05

—
More on how to monitor database using database control (dbconsole) coming soon …

References

• 278100.1  How To Drop, Create And Recreate DB Control In A 10g Database
• 395162.1  How to manage DB Control 10.2 for RAC Database with emca
• 743833.1  Rapid Clone & Em Dbconsole
• 549079.1  Troubleshooting Database Control Startup Issues
• 553154.1  “UnsatisfiedLinkError: njni10? When Tring to Create the dbconsole With EMCA
• 337260.1  Dbsnmp Password Not Accepted
• Enterprise Manager Advanced Configuration Guide

Share This

As year 2008 comes to an end I look back at stats for year 2008,  this blog had 795000 Page Hits, 370000 page views, 1800 comments and approx. 1000 feedburner subscription.
.

Here is top Six posts of year 2008 (ranking by page views)

• Step by Step RAC Build
• 1Z0-233 Dump/Questions
• Instance Home Overview in R12
• Install 11.5.10/11i on Linux
• Difference between 11i & R12
• How to become Apps DBA

Happy New Year to all of You !

Share This

Oracle Clusterware (Cluster Ready Services in 10g/ Cluster Manager in 9i) - provides infrastructure that binds multiple nodes that then operate as single server. Clusterware monitors all components like instances and listeners. There are two important components in Oracle clusterware, Voting Disk and OCR (Oracle Cluster Registry).

.

Voting Disk - is file that resides on shared storage and Manages cluster members.  Voting disk reassigns cluster ownership between the nodes in case of failure.

OCR (Oracle Cluster Registry) - resides on shared storage and maintains information about cluster configuration and information about cluster database. OCR contains information like which database instances run on which nodes and which services runs on which database.

CRS Resource - anything that Oracle Clusterware manages is classified as CRS resource like database, instance, service, listener, VIP address and so on.

.
Cluster-Aware Storage - is storage solution for Oracle RAC like RAW device, OCFS, ASM… to know more about storage option in RAC click here
.
Interconnect - is private network that connects all the servers in cluster. Interconnect uses switch that only nodes in cluster can access. Instances in cluster communicate to each other via interconnect.
.

Cache Fusion - is disk less cache coherency mechanism in Oracle RAC that provides copies of data blocks directly from one instance’s memory cache (in which that block is available) to other instance (instance which is request for specific data block).  Cache Fusion provides single buffer cache (for all instances in cluster) through interconnect.

In Single Node oracle database, an instance looking for data block first checks in cache, if block is not in cache then goes to disk to pull block from disk to cache and return block to client.

In RAC Database there is remote cache so instance should look not only in local cache (cache local to instance) but on remote cache (cache on remote instance). If cache is available in local cache then it should return data block from local cache; if data block is not in local cache, instead of going to disk it should first go to remote cache (remote instance) to check if block is available in local cache (via interconnect)

This is because accessing data block from remote cache is faster than accessing it from disk.

.

Cache Fusion Model
Cache fusion Model is dependent on three services
– Global Resource Directory (GRD)
– Global Cache Service (GCS)
– Global En-queue Service (GES) and –

More on Cache Fusion Model coming soon …..
.
SSH User Equivalency - means assigning same properties (username, userid, group, group id and same password) to operating system user (installing & owning RAC database) across all nodes in cluster

CVU (Cluster Varification Utility) - is utility to verify that system meets all the criteria for Oracle Clusterware Installation.

Share This

BEA’s AquaLogic Service Bus (ALSB) and Oracle’s Enterprise Service Bus (ESB) are now called as Oracle Service Bus (OSB).

.

Use of Service Bus (SB):
Service Bus offers connectivity options (for standard interfaces like SOAP, messages .. and packaged applications & legacy system via adapters), messaging & routing (efficient, reliable and secure flow of data) , Transformation (data format from one service to another service), aggregation and auditing of messages.
Service Bus is one of component in SOA Suite (BPEL, Service Registry, BAM) which you could install as part of SOA Suite or as stand alone product.

.
For Oracle Service Bus statement of direction Click Here

.
Documentation

• For Oracle Service Bus 10g R3 (10.3)
  Configuration, Designing, Deployment and Upgrades Guide click here

• For Oracle Enterprise Service Bus 10g (10.1.3.X)
  OESB Quick Start Guide click here
  OESB Installation Guide click here
  OESB Developer Guide click here
  .

Download

Download Oracle Service Bus 10g R3 (standalone) or BEA’s AquaLogic Service Bus 3.0 Click Here

To download Oracle Enterprise Service Bus 10.1.3.1.0 click here

Share This

While starting Apache (Web Server) in E-Business Suite 11i using “adapcctl.sh start“, I got below error message

***********
Cannot reconnect to gateway
Cause:  Application Object Library is unable to reconnect to your gateway ORACLE account after you unsuccessfully attempted to sign-on.
Action:  Check that your gateway environment variable is set correctly.
***********

My first attempt was to check in metalink where I found note “413529.1  Apache adstrtal.sh adstpal.sh Scripts Give Cannot Reconnect To Gateway With Wrong Apps Username or Password“.

I was using ”adapcctl.sh start“  (No apps password, I don’t remember ever using apps password with Apache) so Metalink Note 413529.1 was not good hit.

My next suspect was GSM (Generic Service Manager - 210062.1 ) so started looking in to adapcctl.sh ($OAD_TOP/ admin/ scripts/ $CONTEXT_NAME) and realized that culprit was FNDSVCRG .

FNDSVCRG : is executable under $FND_TOP/bin introduced as part of seeded GSM services and provides improved coordination between the GSM monitoring of services (like Apache) and their command-line control scripts (like adapcctl.sh)

FNDSVCRG uses dbc file in $FND_SECURE to connect to database using GUEST account. Entry in dbc file was wrong and quick fix was to run AutoConfig (More on Autoconfig Here) to regenerate dbc file.

.

Workaround 

If for some reason you do not want to run Autoconfig then remove FNDSVCRG and start Apache.

mv $FND_TOP/bin/FNDSVCRG $FND_TOP/bin/FNDSVCRG.bak

.

Entry in adapcctl.sh

if test “$control_code” = “start”; then
pre_command=”PRESTART”
post_command=”POSTSTART”

sh -c ‘. $ORA_TOP/8.0.6/$SID_$hostname.env; $FND_TOP/bin/FNDSVCRG $pre_command APACHE $HOST $Apaceh_Port’
svcrg_code=$?

sh -c ‘. $ORA_TOP/8.0.6/$SID_$hostname.env; $FND_TOP/bin/FNDSVCRG $post_command APACHE $HOST $Apaceh_Port’
svcrg_code=$?
.
References

• 413529.1  Apache adstrtal.sh adstpal.sh Scripts Give “Cannot Reconnect To Gateway” With Wrong Apps Username or Password
• 210062.1  Generic Service Management (GSM) in Oracle Applications 11i

Share This