oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full

Yesterday, the users of an EBS R12.2 instance got this error when they logged in:

Error Page
You have Encountered an unexpected error.  Please contact the System Administrator for assistance.

On checking the $EBS_DOMAIN_HOME/servers/oacore_server1/logs/oacore_server1.out, we found this error:


adopreports utility in R12.2

I discovered a utility in R12.2 when I was looking for the directory of adop :

which adop
$NE_BASE/EBSapps/appl/ad/bin/adop

cd $NE_BASE/EBSapps/appl/ad/bin/
$ ls
adop  adopreports

Curious I executed adopreports
$ adopreports

Enter the APPS username: apps
Enter the APPS Password:



    Online Patching Diagnostic Reports Main Menu
    --------------------------------------------

    1.  Run edition reports
    2.  Patch edition reports
    3.  Other generic reports
    4.  Exit

    Enter your (more...)

R12.2 URL Validation failed. The error could have been caused through the use of the browser’s navigation buttons ( the browser Back button or refresh, for example). If the error persists,

Sravan came across this error while using SSO enabled URL for R12.2


URL validation failed. The error could have been caused through the use of the browser's navigation buttons ( the browser Back button or refresh, for example). If the error persists, Please contact system administrator.


This occurs for any user that is newly created. The user_guid column is not populated. Here is the Solution:

ssoreg_r12_linux erppgai1 DEV appspassword cleanup
ssoreg_r12_linux erppgai1 DEV appspassword (more...)

R12.2 adapcctl.sh: opmnctl: not found after fresh install / upgrade



This problem occurs during rapidwiz where it fails to create apache instance and the error is logged in deploytech.log:

from deploytech.log:

ERROR: Failed to create the EBS domain.
Failed to deploy the E-Business Suite domain

Oracle would (more...)

R12.2 Dictionary Corrupted error during adop phase=prepare

Do not follow the document 1576086.1.  I followed it and increased my invalid counts from 1000 to 100,000 and wasted 10 hours to get back to where I started.  Instead follow these instructions:


Reference : Ora-04023 (more...)

ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table. ORA-06512: at "APPS.AD_ZD_ADOP", line 523

I reported this error to Oracle on an SR:

The adop phase=prepare fails before generating any log files. Whatever we get on the screen is what we have. I'll paste it for you again:

$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Please wait. Validating credentials...


RUN file system context file: $INST_TOP/appl/admin/tsoadba1_alpgetapp049d.xml
PATCH file system context file: $INST_TOP/appl/admin/tsoadba1_alpgetapp049d.xml


************* Start of session *************
version: 12.2. (more...)

Product Restrictions during R12.2 online patching

This information is from page 75 of Oracle® E-Business Suite Maintenance Guide Release 12.2 Part No. E22954-13, published in July 2013 available here  http://docs.oracle.com/cd/V39571_01/current/acrobat/122ebsmt.zip

For patches that have manual steps, the patch readme file instructs you to use Oracle Patch Application Assistant (PAA) to create customized instructions for your system. PAA consolidates and displays only the relevant manual steps for all the patches you want to apply, including steps that you (more...)

R12.2 upgrade – icxr12pd.sql takes a very long time

During an R12.2 upgrade, the worker icxr12pd.sql takes very long time.  This is document in My Oracle Support Article ICXR12PD.Sql Runs Slowly During Upgrade to 12.1.1 (Doc ID 864898.1)
 
Cause: The pre-upgrade steps for iProcurement were not implemented as documented in the upgrade guide
Though this article talks about R12.1.1, it also applies to R12.2.  If you read the 11i to R12.2 (more...)

weblogic.store.PersistentStoreFatalException: The persistent file store "WLS_DIAGNOSTICS" canno t open file WLS_DIAGNOSTICS000000.DAT

After bouncing an EBS R12.2 VM, weblogic adminserver wouldn't start up.

I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:

BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
        at weblogic.store. (more...)

R12.2 Failure of server APACHE bridge

Failure of server APACHE bridge:


No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent.

If there is an issue with the oacore managed server, meaning apache is not able to connect to the oacore managed server, you'll see this (more...)

R12.2 Apache and Weblogic log file locations

Apache Logs

$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/EBS_web_*/*log

OPMN Log
$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/*

Weblogic Logs

$IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*
$EBS_DOMAIN_HOME/servers/forms*/logs/*
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*
$EBS_DOMAIN_HOME/sysman/log/*

Source: Oracle® E-Business Suite
Maintenance Guide
Release 12.2

adpreclone.pl fails with segmentation fault, bus error, out of memory

Manoj pinged me about this issue last night.  Whenever he was trying to do adpreclone.pl dbTier or AppsTier, when he entered the apps password, the process would crash with either segmentation fault, bus error or out of memory.

I tried executing top and it was also ending up with (more...)

R12.2.2 is now available on edelivery.oracle.com

At last, R12.2.2 is now available on edelivery.oracle.com


550 5.7.1 Relaying denied. IP name lookup failed

Recently we configured a BigIP virtual for SMTP called smtp.justanexample.com

Whenever I tried sending mail through this it would complain:

# telnet smtp.justanexample.com 25


Trying 192.168.10.201...

Connected to smtp.justanexample.com
Escape character is '^]'.

220 smtp1.justanexample.com ESMTP Sendmail 8.13. (more...)

Path not found (/shared/FTI Reports/_Portal/Main Dashboard/dashboard layout)

In OTM FTI you may get an error like this:

OTM > Transportation Intelligence > Dashboard

Path not found (/shared/FTI Reports/_Portal/Main Dashboard/dashboard layout)


Error Details

Error Codes: U9KP7Q94    

From OTM install guide In the $OBIEE_DATA/web/config/ directory, edit the instanceconfig.xml as follows:

a. Change the CatalogPath to point the new “aa” directory that was created when you extracted the aa_webcat.zip file. The template shows the following line.

vi instanceconfig.xml



Search for string CatalogPath and you'll find this
$OBIEE_DATA/web/catalog/samplesales
Edit this line so that it looks like this:
$OBIEE_DATA/web/catalog/aa


Note: Do not put in the $OBIEE_DATA only add (more...)

ORA-06502: PL/SQL: numeric or value error: hex to raw conversion error ORA-06512: at "APPS.WF_EVENT", line 3600

Akhilesh pinged me today and told me that autoconfig was not working on a newly upgraded R12 instance.  It was failing with these errors:

AutoConfig could not successfully execute the following scripts:
   Directory: /r12ascp/erpapp/10.1.3/perl/bin/perl -I /r12ascp/erpapp/10.1.3/perl/lib/5.8.3 -I /r12ascp/erpapp/10.1.3/perl/lib/site_perl/5.8.3 -I /usnapss
1/erpapp/appl/au/12.0.0/perl -I /r12ascp/erpapp/10.1.3/Apache/Apache/mod_perl/lib/site_perl/5.8.3/sun4-solaris-thread-multi /r12ascp/erpapp/inst/apps/usnaps
s1_r12ascp/admin/scripts/adexecsql.pl sqlfile=/r12ascp/erpapp/inst/apps/r12ascp_r12ascp/admin/install
     afwebprf.sql            INSTE8_PRF         1
     amscmprf.sql            INSTE8_PRF         1
     amswebprf.sql     (more...)

java.sql.SQLException: ORA-01017: invalid username/password; logon denied for glogdba after cloning OTM

After cloning OTM database and instance, when we started the services and tried accessing

Configuration and Administration > Cluster Management > Scalability Overview, we got this error in glog.exception.log

java.sql.SQLException: ORA-01017: invalid username/password; logon denied

I was able to diagnose that this was coming for glogdba user by switching on auditing on the database.

The encrypted password was correct for glogdba in glog.properties.

After a lot of R&D we found out that this was occurring because the encrypted password of glogdba user in data_source table was incorrect.


It had the encrypted password of the glogdba user in production. (more...)

Oracle error 20001: java.sql.SQLException: ORA-20001: Oracle error -6508: ORA-06508: PL/SQL: could not find program unit being called has been detected in fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG. ORA-06512

Anand  pinged me today.  After cloning an 11i instance, the following error was coming when AppsLocalLogin.jsp was accessed:

PRE>Unable to create anonymous session. Your session is no longer valid. Oracle error 20001: java.sql.SQLException: ORA-20001: Oracle error -6508: ORA-06508: PL/SQL: could not find program unit being called has been detected in fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG. ORA-06512: at "APPS.MO_UTILS", line 27 ORA-06512: at "APPS.MO_UTILS", line 232 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 2109 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 369 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 538 ORA-06512: at "APPS.FND_AOLJ_UTIL", line 217 ORA-06512: at line 1 has been (more...)

Workflow Mailer Notification not sending out mails

Today Amanda pinged me about an issue with Workflow Notification Mailer in an R12.1.3 instance.  Mails were not going out.  I checked the SMTP with commandline:

telnet justanexample.com 25
mail from:john.doe@justanexample.com
rcpt from:jane.doe@justanexample.com
data
Subject: Test mail from smtp commandline of justanexample.com
Test.
.

The mail was going out fine.  So there was no issue on the unix Sendmail side. I suggested that we check the configuration inside Apps.  A search on support.oracle.com revealed this article:

What to Review When Notifications are not Emailed (Outbound Processing)? [ID (more...)

ORA-06512: at "APPS.FND_CORE_LOG", line 318

Shoaib pinged me today. He was getting this error duing AutoConfig on DB node:

begin

*
ERROR at line 1:
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at "APPS.FND_CORE_LOG", line 318
ORA-06512: at "APPS.FND_CORE_LOG", line 62
ORA-06512: at "APPS.FND_CORE_LOG", line 456
ORA-06512: at "APPS.FND_PROFILE", line 110
ORA-06512: at "APPS.ADX_PRF_PKG", line 203
ORA-06512: at line 7

A search on My Oracle Support revealed R12: ORA-06502: PL/SQL: numeric or value error trying to update table fnd_profile_option_values [ID 1501822.1]


However the error described was similar, but not same:

ORA-06502: PL/SQL: numeric or (more...)