Oracle Application DBA

Cloning Reference:-
Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1) – For clone procedure
Sharing the Application Tier File System in Oracle E-Business Suite Release 12.2 (Doc ID 1375769.1) – For Shared Appl top procedure
Issues Refrence:
http://balaoracledba.com/category/oracleappsr12/issues/ - For known issues and support to oracle docs
http://oradbapp.blogspot.in/2015/05/ebs-122-unable-to-start-up-managed.html- For a known issue in 12.2.4
12.2: error while fs_clone of patching(WLS 17495356) cycle (Doc ID 1634239.1)
12.2: fs_clone fails with "ERROR: Number of servers are not in sync between Run and Patch Context" (Doc ID 1582473.1)
Fs_clone Fails with Status 2 (Doc ID 1960769.1)
FS Clone Fails Saying Port In Use: Free The Listed Ports And Retry The Adop Operation (Doc ID 1901109.1)
Clone Oracle Applications Environment
This section explains the detailed steps to clone an Oracle applications 12i (12.2.4) installation using a 11gR2 (11.2.0.4) database to another Oracle 12i (12.2.4) 11gR2(11.2.0.4) environment.
The main steps for cloning Oracle Applications Environment are:
Pre Clone Steps.
Clone Database Tier.
Clone Application Tier.
Post Clone Steps.
Pre Clone Steps
On Source Instance Please perform below activities.
Nodes Task Description Command
Check if instance can be cloned This can be done anytime before the clone
App Source Environment and choose RUN  . /u11/home/ebsdev/EBSapps.env
App Find master node if multi-node system
Master node is RUN FS for instance ls -ltr $NE_BASE/EBSapps/log/status*/adzdshowstatus.out | tail -n 1
check for master node in the file
App Check ADOP status complete on master-node CLEANUP         COMPLETED
Or
PREPARE         NOT STARTED “adop -status” should show all completed,
otherwise use
“adop phase=finalize/cutover/cleanup/abort/fs_clone” to complete
Run Preclone on App-tier and copy This can be done anytime before the clone after ADOP cleanup
App On Master_node RUN FS
Run adpreclone.pl appsTier Following files exist in $RUN_BASE/ /EBSapps/comn/clone/FMW
   FMW_Home.jar
   WLS/EBSdomain.jar
   WLS/plan/moveplan.xml
   OHS/ohsarchive.jar
   OHS/moveplan.xml $INST_TOP/admin/scripts/adadminserver.sh start
$INST_TOP/admin/scripts/adpreclone.pl appsTier
App Copy $RUN_BASE/EBSapps to target Copy the tar/zip file to target instance $RUN_BASE tar & zip the binaries
tar –cf – EBSapps | gzip > EBSapps.tar.gz
DB Run Preclone on DB-tier and copy Subbu
DB Run adpreclone.pl dbTier Subbu $OH/appsutil/scripts/$CTX/adpreclone.pl dbTier
DB Backup DB Subbu
DB Backup RDBMS home Subbu
Clone Database Tier
Clone oracle home & Database
Clone DB & Oracle Homes on target DB tier. This task is performed by DBA team (Subbu)
Enable Auto Config on DB Tier
For RAC node run below steps on both nodes.
NODE 1
1. Create appsutil.zip on source app server
apdbappd01:appldev1$ $ADPERLPRG $AD_TOP/bin/admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at /u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/log/MakeAppsUtil_04131514.log
output located at /u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip
MakeAppsUtil completed successfully.
2. Transfer appsutil.zip file to target DB server
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $ scp mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00
cd $ORACLE_HOME
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $ scp mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $
3. Unzip appsutil.zip file
cd $ORACLE_HOME
unzip -o appsutil.zip
4. Purge Old nodes information
Login as apps user
$ sqlplus apps/apps
SQL> exec fnd_conc_clone.setup_clean
5. Install JRE
Install jre under $ORACLE_HOME/appsutil
cd /bio/ebs/str/db/11.2.0.4/appsutil
cat /ora_nfs/jre/jre-7u11-linux-x64.tar.gz | gzip -d | tar xf - &
mv jre1.7.0_11 jre
6. Create Context file
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ adbldxml.pl appsuser=apps appspass=appss2015
Starting context file generation for db tier..
Using JVM from /bio/ebs/str/db/11.2.0.4/appsutil/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/bio/ebs/str/db/11.2.0.4/appsutil/log/adbldxml_04231351.log
Enter Database Service Name: ebsstr1
Enter the value for Display Variable: apdbadm01:0.0
Context File  /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
 already exists.
Overwriting Context File may cause loss of existing settings, hence
 backing it up as: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml.bak
 The context file has been created at:
/bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Update context file with target db port number
vi /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
update db port from 1521 to 1731 & save the file
Ensure that Util file dir has /usr/tmp
7. Run Auto Config
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ perl adconfig.pl
Enter the full file path to the Context file: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Enter the APPS user password:
The log file for this session is located at: /bio/ebs/str/db/11.2.0.4/appsutil/log/ebsstr1_apdbadm01/04231358/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /bio/ebs/str/db/11.2.0.4
        Classpath                   : :/bio/ebs/str/db/11.2.0.4/jdbc/lib/ojdbc6.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java/xmlparserv2.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java:/bio/ebs/str/db/11.2.0.4/jlib/netcfg.jar:/bio/ebs/str/db/11.2.0.4/jlib/ldapjclnt11.jar
        Using Context file          : /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Context Value Management will now update the Context file
        Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.
NODE 2:
1. Transfer appsutil.zip file to target DB server
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $ scp` mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00
cd $ORACLE_HOME
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $ scp mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00`
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $
2. Unzip appsutil.zip file
cd $ORACLE_HOME
unzip -o appsutil.zip
3. Purge Old nodes information
Login as apps user
$ sqlplus apps/apps
SQL> exec fnd_conc_clone.setup_clean
4. Install JRE
Install jre under $ORACLE_HOME/appsutil
cd /bio/ebs/str/db/11.2.0.4/appsutil
cat /ora_nfs/jre/jre-7u11-linux-x64.tar.gz | gzip -d | tar xf - &
mv jre1.7.0_11 jre
5. Create Context file
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ adbldxml.pl appsuser=apps appspass=appss2015
Starting context file generation for db tier..
Using JVM from /bio/ebs/str/db/11.2.0.4/appsutil/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/bio/ebs/str/db/11.2.0.4/appsutil/log/adbldxml_04231351.log
Enter Database Service Name: ebsstr1
Enter the value for Display Variable: apdbadm01:0.0
Context File  /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
 already exists.
Overwriting Context File may cause loss of existing settings, hence
 backing it up as: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml.bak
 The context file has been created at:
/bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Update context file with target db port number
vi /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
update db port from 1521 to 1731 & save the file
Ensure that Util file dir has /usr/tmp
6. Run Auto Config
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ perl adconfig.pl
Enter the full file path to the Context file: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Enter the APPS user password:
The log file for this session is located at: /bio/ebs/str/db/11.2.0.4/appsutil/log/ebsstr1_apdbadm01/04231358/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /bio/ebs/str/db/11.2.0.4
        Classpath                   : :/bio/ebs/str/db/11.2.0.4/jdbc/lib/ojdbc6.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java/xmlparserv2.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java:/bio/ebs/str/db/11.2.0.4/jlib/netcfg.jar:/bio/ebs/str/db/11.2.0.4/jlib/ldapjclnt11.jar
        Using Context file          : /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Context Value Management will now update the Context file
        Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.
Clone Application Tier – Primary node apdbablog01
Create RUN File System
1. Untar/zip EBSApps dump file on target server
                      Copied fs2(Run Base) from source to /u11/home/ebsprd/fs2/
tar -cf - EBSapps | gzip > EBSapps.tar.gz
cp -pr EBSapps.tar.gz /patch/erp/bala/ebsprd_files/
cp -pr /patch/erp/bala/ebsprd_files/EBSapps.tar.gz ./
tar –xvf EBSapps.tar.gz
To determine which file system (fs1 or fs2) is currently the Run Edition File System, run the command:
echo $RUN_BASE
Note: If the Source Run Edition File System is under fs1, you should configure the Target Run Edition File System under fs1 as well.
Similarly, if the Source Run Edition File System is under fs2, you should configure the Target Run Edition File System under fs2.
2. Create RUN file system
cd /u11/home/ebspat/fs1/EBSapps/comn/clone/bin
[applpat@apdbablog03 bin]$ perl adcfgclone.pl appsTier
                     Copyright (c) 2011 Oracle Corporation
                        Redwood Shores, California, USA
                        Oracle E-Business Suite Rapid Clone
                                 Version 12.2
                      adcfgclone Version 120.63.12020000.39
Enter the APPS password :
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper -encryptpwd /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../FMW/tempinfoApps.txt
Enter the Weblogic AdminServer password :
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../FMW/tempinfo.txt
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../FMW/EBSDataSource
Do you want to add a node (yes/no) [no] :
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -cp /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/xmlparserv2.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojdbc6.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.context.CloneContext -e /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../context/apps/CTXORIG.xml -validate -pairsfile /tmp/adpairsfile_11610.lst -stage /u11/home/ebspat/fs1/EBSapps/comn/clone  2> /tmp/adcfgclone_11610.err; echo $? > /tmp/adcfgclone_11610.res
Log file located at /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/CloneContext_0414180413.log
Target System File Edition type [run] :
Provide the values required for creation of the new APPL_TOP Context file.
Target System Hostname (virtual or normal) [apdbablog03] :
Target System Database SID : ebspat
Target System Database Server Node [apdbablog03] : edp01-scan
Target System Database Domain Name [apdba.com] :
Target System Base Directory : /u11/home/ebsprd
Target System Base Directory set to /u11/home/ebsprd
Target System Current File System Base set to /u11/home/ebspat/fs1
Target System Other File System Base set to /u11/home/ebspat/fs2
Target System Fusion Middleware Home set to /u11/home/ebspat/fs1/FMW_Home
Target System Web Oracle Home set to /u11/home/ebspat/fs1/FMW_Home/webtier
Target System Appl TOP set to /u11/home/ebspat/fs1/EBSapps/appl
Target System COMMON TOP set to /u11/home/ebspat/fs1/EBSapps/comn
Target System Instance Home Directory [/u11/home/ebspat] :
Target System Instance Top set to /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03
Do you want to preserve the Display [apdbappd01:0.0] (y/n)  : n
Target System Display [apdbablog03:0.0] :
Target System Root Service [enabled] :
Target System Web Administration [enabled] :
Target System Web Entry Point Services [enabled] :
Target System Web Application Services [enabled] :
Target System Batch Processing Services [enabled] :
Target System Other Services [disabled] :
Do you want the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 50
Checking the port pool 50
done: Port Pool 50 is free
Report file located at /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/admin/out/portpool.lst
Complete port information available at /u11/home/ebspat/fs1/inst/apps/ebsprd_apdbablog03/admin/out/portpool.lst
RC-40201: Unable to connect to Database ebspat.
Enter the Database listener port [1571] : 1725
UTL_FILE_DIR on database tier consists of the following directories.
1. /bio/ebs/pat/tmp
2. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 2 – select /usr/tmp
Creating the new APPL_TOP Context file from :
  /u11/home/ebspat/fs1/EBSapps/comn/clone/context/apps/adxmlctx.tmp
The new APPL_TOP context file has been created :
  /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml
Log file located at /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/CloneContext_0414180413.log
Check Clone Context logfile /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/CloneContext_0414180413.log for details.
Running Rapid Clone with command:
Running:
perl /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/adclone.pl java=/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre mode=apply stage=/u11/home/ebspat/fs1/EBSapps/comn/clone component=appsTier method=CUSTOM appctxtg=/u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml showProgress contextValidated=true
Faced an Error:-
############################ ALERT ###################################
The oraInst.loc file is not present under /etc. Without this file cloning cannot proceed further
Please run the oraInstFileGenerator.sh located under /tmp with ROOT privileges
This script will create the oraInst.loc file under /etc
ERROR: Failed to execute /u11/home/ebsprd/fs2/EBSapps/comn/clone/bin/adclone.pl
 Please check logfile.
Reason for this error:
This is the first time we are cloning in this machine so we need to run the orainstFileGenerator.sh
Raising a ticket to UNIX team to run this as root user.
Fixed the issue in this server:
/u11/home/oraInventory
Manually create this dir
I renamed the existing oraInventory to oraInventory.obsolotete and mkdir oraInventory to create a new directory. Then restarted the process
Running the adcfgclone.pl again:
Again faced an error:
No write permissions to the Inventory pointed by:
        /etc
Rapid Clone needs to be able to write to the inventory
Please run the script /tmp/orainstRoot.sh as root
Restarted adcfg clone:
Success!!
Issue:3
adcfgclone in Run FS
Enter value for 1: Enter value for 2: Enter value for 3: ERROR:
ORA-12505: TNS:listener does not currently know of SID given in connect
descriptor
Fix:
1) Verified the Autocofig erroed in DB second nodesync
2) Ran the Autoconfig in DB nodes and ran the adcfgclone seems completed successfully
Post Clone Steps
Source ENV and run the cmclean in sqlplus
sqlplus apps/apps_pw
SQL>@cmclean.sql
SQL>commit;
SQL>exit;
Run the script to keep the jobs on hold
sqlplus apps/apps_pw
SQL> create table dba_conc_req_hold as select * from fnd_concurrent_requests;
SQL> update fnd_Concurrent_requests set hold_flag='Y' where PHASE_CODE='P' and hold_flag='N' and request_id in (select request_id from dba_conc_req_hold);
SQL> commit;
SQL>exit;
Change apps & sysadmin password
Source the Run FS
$FNDCPASS apps/appss2015 0 Y system/<target_system_pw> SYSTEM APPLSYS <target_apps_pw>
$FNDCPASS apps/<target_apps_pw> 0 Y system/<target_system_pw> USER   SYSADMIN  <target_sysadmin_pw>
verify the log and make sure there is no errors in FND log
Update wf tables
Source the Run FS
sqlplus apps/apps_pw
SQL>create table wf_notifications20130709 as select * from wf_notifications;
SQL>update wf_notifications set mail_status='SENT' where mail_status is not null;
SQL> Update wf_resources
Set text=’ FND_RESP|FND|FNDWF_ADMIN_WEB|STANDARD’
Where name =’ WF_ADMIN_ROLE’  ;
SQL>commit;
SQL>exit;
Run adcmclean.sql script
Source the Run FS
Configure Load Balancer – Configured by Load Balancer team
Source the Run FS primary node
Update auto config context file and run auto config(Please take a backup of currecnt context file)
<externURL oa_var="s_external_url">http://ebsprd.apdba.com</externURL>
<login_page oa_var="s_login_page">http://epsprd.apdba.com/OA_HTML/AppsLogin</login_page>
<activewebport oa_var="s_active_webport" oa_type="DUP_PORT" base="8000" step="1" range="-1" label="Active Web Port">80</activewebport>
<webentryhost oa_var="s_webentryhost">ebsprd</webentryhost>
<APPL_TOP_NAME oa_var="s_atName">ebsprd</APPL_TOP_NAME>
Cd $INST_TOP/admin/scripts
./adautocfg.sh appspass=<pw>
Start application services
cd $INST_TOP/admin/scr*
./adstrtall.sh apps/<pw>
Starting Application for the first time.
You are running admanagedsrvctl.sh version 120.14.12020000.10
Starting oacore_server1...
admanagedsrvctl.sh: exiting with status 1
admanagedsrvctl.sh: check the logfile /u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog01/logs/appl/admin/log/adoacorectl.txt for more infmation ...
.end std out.
.end err out.
****************************************************
adstrtal.sh: Exiting with status 1
Fix:
http://oradbapp.blogspot.in/2015/05/ebs-122-unable-to-start-up-managed.html
It is observed that Apps Password is changed as part of execution of adcfgclone.pl on DBTier and this new password change is somehow not reflected when managed servers are created during execution of rapidclone on Apps Tier.
Now this is fixed manually by performing below steps.
As ADMIN Server is up and runnning,
1. Login to EBS Weblogic console -> Perform Lock & Edit -> Under Domain Structure -> Services -> Datasources -> Select "EBSDataSource"
2. Navigate to Configuration -> Connection pool  -> Update with new password in password fields.
3. Then Save changes.
4. Perform the above steps of other Datasource OAADatasource.
Fixed:-
Local URL: http://ebsprd.apdba.com/OA_HTML/AppsLocalLogin.jsp
Login & validate
Log into application
Run  a concurrent request and view output
Checked and verified Primary node Run FS application
Create Patch File System
Note:- Do not delete or move the orainventory.xml(central inventory) during patch file system clone
1. Export the environment variablesa as below
export CONFIG_JVM_ARGS="-Xms512m -Xmx1024m"
export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/tmp -Xms512m -Xmx1024m"
export IGNORE_FMW_DISK_WARNING=true
export TIMEDPROCESS_TIMEOUT=-1
2. Run preclone on Run FS and take backup of EBSapps. Move it to Patch files system.
3. Run adcfgclone.pl in Patch FS
cd /u11/home/ebspat/fs2/EBSapps/comn/clone/bin
[applpat@apdbablog03 bin]$ ./adcfgclone.pl appsTier
                     Copyright (c) 2011 Oracle Corporation
                        Redwood Shores, California, USA
                        Oracle E-Business Suite Rapid Clone
                                Version 12.2
                     adcfgclone Version 120.63.12020000.39
Enter the APPS password :
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper -encryptpwd /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../FMW/tempinfoApps.txt
Enter the Weblogic AdminServer password :
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../FMW/tempinfo.txt
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../FMW/EBSDataSource
Do you want to add a node (yes/no) [no] :
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -cp /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/xmlparserv2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojdbc6.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.context.CloneContext -e /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../context/apps/CTXORIG.xml -validate -pairsfile /tmp/adpairsfile_27612.lst -stage /u11/home/ebspat/fs2/EBSapps/comn/clone  2> /tmp/adcfgclone_27612.err; echo $? > /tmp/adcfgclone_27612.res
Log file located at /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/CloneContext_0419034910.log
Target System File Edition type [run] : patch (we should enter patch)
Enter the full path of Run File System Context file : /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml
Provide the values required for creation of the new APPL_TOP Context file.
Target System Fusion Middleware Home set to /u11/home/ebspat/fs2/FMW_Home
Target System Web Oracle Home set to /u11/home/ebspat/fs2/FMW_Home/webtier
Target System Appl TOP set to /u11/home/ebspat/fs2/EBSapps/appl
Target System COMMON TOP set to /u11/home/ebspat/fs2/EBSapps/comn
Target System Instance Top set to /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03
Target System Port Pool [0-99] : 60(in FS1 the port pool is 50 so here we should enter 60)
Checking the port pool 56
done: Port Pool 56 is free
Report file located at /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/admin/out/portpool.lst
Complete port information available at /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/admin/out/portpool.lst
UTL_FILE_DIR on database tier consists of the following directories.
1. /bio/ebs/pat/tmp
2. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 2 – select /usr/tmp
Creating the new APPL_TOP Context file from :
  /u11/home/ebspat/fs2/EBSapps/comn/clone/context/apps/adxmlctx.tmp
The new APPL_TOP context file has been created :
/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml
Log file located at /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/CloneContext_0419034910.log
Check Clone Context logfile /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/CloneContext_0419034910.log for details.
Running Rapid Clone with command:
Running:
perl /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/adclone.pl java=/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre mode=apply stage=/u11/home/ebspat/fs2/EBSapps/comn/clone component=appsTier method=CUSTOM appctxtg=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml showProgress contextValidated=true
FMW Pre-requisite check log file location : /u11/home/ebspat/fs2/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Running: /u11/home/ebspat/fs2/EBSapps/comn/clone/FMW/t2pjdk/bin/java -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/engine.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereq.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereqChecks.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstaller.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstallerNet.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/srvm.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl-log4j.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/xmlparserv2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/share.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java oracle.apps.ad.clone.util.FMWOracleHomePreReqCheck -prereqCheckFMW -e /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml -stage /u11/home/ebspat/fs2/EBSapps/comn/clone -log /u11/home/ebspat/fs2/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Beginning application tier Apply - Sun Apr 19 03:52:15 2015
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=true -Doracle.installer.oui_loc=/oui -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/xmlparserv2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojdbc6.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/oui/OraInstaller.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/oui/ewt3.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/oui/share.jar:/u11/home/ebspat/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.ApplyAppsTier -e /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml -stage /u11/home/ebspat/fs2/EBSapps/comn/clone    -showProgress -nopromptmsg
Log file located at /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/admin/log/clone/ApplyAppsTier_04190352.log
100% completed
Completed Apply...
1. Run con clean in DB nodes:
1) Concclean
2) Auto config
2. Run auto config in all application nodes – Run fs
1) Run Auto config
3. Check and make sure that admin server runs only in Primary server and web is disabled in CC nodes.
select NODE_NAME,SUPPORT_CP,SUPPORT_WEB,SUPPORT_ADMIN from fnd_nodes;
If managed server oacore_server2 exists in primary node then,
1) perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -accessgate=apdbablog11.apdba.com:6891
2) perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -ekanban=apdbablog11.apdba.com:6891
3) perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -yms=apdbablog11.apdba.com:6891
After you run these three make sure there are no entries of host apdbablog11.apdba.com  in mod_wl_ohs.conf under /u13/home/ebsdev/fs2/FMW_Home/webtier/instances/EBS_web_ebsdev_OHS1/config/OHS/EBS_web_ebsdev/
Then run below to remove oaea_server2 managed server and confirm the server is removed.
4) perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-managedserver -contextfile=$CONTEXT_FILE -managedsrvname=oaea_server2 -servicetype=accessgate -logfile=/tmp/oaeaserver2.log
FS_CLONE:
Source the Run env
Primary RUN FS amdmin server should run
[applprd@apdbablog01 ~]$ adop phase=fs_clone force=yes
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials...
Initializing...
    Run Edition context  : /u11/home/ebsprd/fs1/inst/apps/ebsprd_apdbablog01/appl/admin/ebsprd_apdbablog01.xml
    Patch edition context: /u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog01/appl/admin/ebsprd_apdbablog01.xml
    Patch file system freespace: 341.25 GB
Validating system setup...
    Node registry is valid.
Generating log report...
    Output: /u11/home/ebsprd/fs_ne/EBSapps/log/adop/43/fs_clone_20150819_022924/ebsprd_apdbablog01/adzdshowlog.out
adop phase=fs_clone - Completed Successfully
adop exiting with status = 0 (Success)
Issue 1:
-----------------------------
ERROR: The following required ports are in use:
-----------------------------
7451 : Managed Server Port 2
7651 : Managed Server Port 1
7851 : Managed Server Port 3
7251 : Managed Server Port 4
Corrective Action: Free the listed ports and retry the adop operation.
FIX :
1) Down all the services in Primary Run fs and restrated the clone
2) Backup all the Context file Change the below server port numbers ( old 7451 : New 7457) make sure your given port number must be
oacorestatus, formsstatus, oafmstatus,forms-c4wsstatus
once the port number changed in all the context files run auto config in all the nodes and start the services(make sure the old port number is not in any context files )
Issue 2:
/u11/home/ebsprd/fs_ne/EBSapps/log/adop/43/fs_clone_20150818_221556/ebsprd_apdbablog01/TXK_SYNC_create/fsclone_stage/FSCloneStageAppsTier_08182219.log:
--------------------------------------------------------------------------------------------------------------------------------------------------------
Lines #(23-27):
patchMSList size: 34
ERROR: Number of servers are not in sync between Run and Patch Context.
ERROR: Managed Servers would be in-sync at apply phase
Lines #(24-28):
patchMSList size: 34
ERROR: Number of servers are not in sync between Run and Patch Context.
ERROR: Managed Servers would be in-sync at apply phase
Fix:
Run the below command in all the nodes to sync the Servers with patch FS and run the auto config in all the Run FS(Please change tha node name in the command)
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/u11/home/ebsprd/fs1/inst/apps/ebsprd_apdbablog05/appl/admin/ebsprd_apdbablog05.xml logfile=/tmp/patchctxupload.log
Issue 3:
Lines #(73-77):
diffSrvName : oaea_server2
runMSSrvPortsVal : oaea_server1:6851
ERROR: Run fs Context variable s_oaea_server_ports value cannot be NULL for oaea_server2
ERROR:  Derived Patch managed server oaea_server2 port : NULL
ERROR: Failed to clone Run Context file to refresh Patch context file
Lines #(74-78):
runMSSrvPortsVal : oaea_server1:6851
ERROR: Run fs Context variable s_oaea_server_ports value cannot be NULL for oaea_server2
ERROR:  Derived Patch managed server oaea_server2 port : NULL
ERROR: Failed to clone Run Context file to refresh Patch context file
Lines #(75-79):
ERROR: Run fs Context variable s_oaea_server_ports value cannot be NULL for oaea_server2
ERROR:  Derived Patch managed server oaea_server2 port : NULL
ERROR: Failed to clone Run Context file to refresh Patch context file
FIX: please do to the following
1) Stop the services in all the Run FS
2) cd /u11/home/ebsprd/fs1/FMW_Home/user_projects/domains/EBS_domain_ebsprd/servers/
3) mv oaea_server2 oaea_server2_old
4) run the autoconfig in all the nodes
5) run the FS_CLONE again
Configure Shared APPL_TOP
Source RUN_FS of Primary application server
1. The value for the context variable "s_atName" must be same across all the application tier node context filesEnsure
2. that value of the AutoConfig variable " s_shared_file_system" is set to "true" ( without the quote) on the primary application tier node.
vi $CONTEXT_FILE
Change variable "s_shared_file_system" is set to "true"
Configure Node 2 - apdbablog02 Web
3. Start weblogic on primary application server if it is not already running and run Preclone on Primary Run FS
Before executing these steps, ensure the AdminServer on both the Run Edition File System and the Patch Edition File System is running. This is required for adcfgclone.pl to properly re-register the new node on the Target System
Source Run file system
Start weblogic administration server
./adadminsrvctl.sh start
./adpreclone.pl appsTier
4. Start weblogic on primary application Patch FS server if it is not already running and run Preclone on Primary Patch FS
Source Patch file system
Start weblogic administration server
./adadminsrvctl.sh start forcepatchfs
./adpreclone.pl appsTier
stop weblogic admin server
./adadminsrvctl.sh stop forcepatchfs
5. Create context files for RUN file system second application node.
[applpat@apdbablog02 scripts]$ cd /u11/home/ebspat/
[applpat@apdbablog02 ebspat]$ ls
EBSapps.env  fs1  fs2  fs_ne  sqlnet.log
[applpat@apdbablog02 ebspat]$ mkdir -p pairsfiles/run
[applpat@apdbablog02 ebspat]$ mkdir -p pairsfiles/patch
[applpat@apdbablog02 ebspat]$
[applpat@apdbablog02 run]$ cd /u11/home/ebspat/pairsfiles/run
[applpat@apdbablog02 run]$ cp /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03_run.txt ./
1.Ensure the correct perl utility is in the $PATH. You must use the perl utility installed in the Fusion Middleware Oracle Home.
 For example:
$ export PATH=/u11/home/ebsprd/fs1/FMW_Home/webtier/perl/bin:$PATH
Edit the Pairs run and patch file
Enter the new node information to be added in pairsfile of run and patch
Run File system
S_batch status should be enabled only in the concurrent nodes in order to configure PCP.
Adminserver runs only in the primary node
Export path: on Secondary node
export PATH=/u11/home/ebsprd/fs2/FMW_Home/webtier/perl/bin:$PATH
For Run FS
2.Ensure the WebLogic Administration Server from the run file system is running on the primary application tier node.
cd $INST_TOP/admin/scripts
./adadminsrvctl.sh start
3.Execute the Clone Context Utility to configure the run file system. For example:
$ cd /u11/home/ebsprd/fs1/EBSapps/comn/clone/bin
$ /u11/home/ebsprd/fs1/FMW_Home/webtier/perl/bin/perl ./adclonectx.pl \
 addnode contextfile=/u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog01/appl/admin/ebsprd_apdbablog01.xml \
 pairsfile=/u11/home/ebspat/pairsfiles/run/ebspat_apdbablog02_run.txt \
 outfile=/u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog02/appl/admin/ebsprd_apdbablog02.xml
Now node added succesffully for Run fs
Create context files for PATCH file system second application node.
export PATH=/u11/home/ebspat/fs2/FMW_Home/webtier/perl/bin:$PATH
[applpat@apdbablog03 admin]$ cd /u11/home/ebspat/pairsfiles/patch
[applpat@apdbablog03 patch]$ cp /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03_patch.txt ./
[applpat@apdbablog03 scripts]$ ./adadminsrvctl.sh start forcepatchfs
You are running adadminsrvctl.sh version 120.10.12020000.9
Enter the WebLogic Admin password:
Enter the APPS Schema password:
[applpat@apdbablog04 secure]$ cd /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/                             [applpat@apdbablog04 bin]$ /u11/home/ebspat/fs2/FMW_Home/webtier/perl/bin/perl ./adclonectx.pl \
addnode contextfile=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml \ pairsfile=/u11/home/ebspat/pairsfiles/patch/ebspat_apdbablog03_patch.txt \ outfile=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog04/appl/admin/ebspat_apdbablog04.xml
Node added successfully
[applprd@apdbablog02 ebsprd]$ perl /u11/home/ebsprd/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh                  -contextfile=/u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog02/appl/admin/ebsprd_apdbablog02.xml -hosts=apdbablog01,apdbablog02,apdbablog03,apdbablog04,apdbablog05,apdbablog06
Enter SSH User password for the OS user applprd:
Logfile location /u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog02/logs/appl/rgf/TXK/enablessh.log
SSH equivalence setup successful.
6. Run auto config on second RUN file system
cd $INST_TOP/admin/scripts
./adautocfg.sh appspass=<pw>
7. Run auto config on second patch file system
cd $AD_TOP/bin
./adconfig.sh contextfile=<patchcontextfile.xml> -syncctx
Eg:- ./adconfig.sh contextfile=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog02/appl/admin/ebspat_apdbablog02.xml -syncctx
Configure PCP
1. Start application services in all nodes
cd $INST_TOP/admin/scripts
./adstrtal.sh apps/<pw>
2. Stop adcmctl in both concurrent nodes.
3. Take all CM that are active and set workshift as Standard - 1 and set primary and secondary servers(Refer source CMs)
4. Make sure the nodes are defined as below
5. Assign primary/secondary nodes for each manager
6. Restart the managers and validate
cd $INST_TOP/admin/scripts
./adcmctl.sh stop apps/<pw>
./adcmctl.sh start apps/<pw>
4. Start all application services now. Check if concurrent process runs only in 05 and 06 nodes.

CREATE PUBLIC DATABASE LINK dev_db CONNECT TO apps IDENTIFIED BY apps
USING 'HOST:PORT/SID';  

select * from dba_db_links;
ping 123.45.6789
ping remoreserv
TNSping to ping the TNS service name
tnsping mydb
query using the database link:
select * from dual@mydb;
select count(*) from remtab@mydb;

List Database Links.

SET PAUSE ON
SET PAUSE 'Press Return to Continue'
SET PAGESIZE 60
SET LINESIZE 300

COLUMN owner FORMAT A30
COLUMN db_link FORMAT A50
COLUMN username FORMAT A30
COLUMN host FORMAT A30

SELECT owner,
       db_link,
       username,
       host
FROM   dba_db_links
ORDER BY owner, db_link

ORA-20041: Error while inserting into control table from VCP :
ORA-20002: Too many rows found while retrieving db link for instance id
+---------------------------------------------------------------------------+
Start of log messages from FND_FILE
+---------------------------------------------------------------------------+
Start of ins_control_table
Start of get_source_db_link
ORA-06512: at "APPS.BIO_MSC_PUB_NOTES_PKG", line 182
ORA-06512: at "APPS.BIO_MSC_PUB_NOTES_PKG", line 1434
ORA-06512: at "APPS.BIO_MSC_PUB_NOTES_PKG", line 182
ORA-06512: at "APPS.BIO_MSC_PUB_NOTES_PKG", line 1434
Solution:
Atp Inquiry Form Shows 'Dblink Setup Is Wrong' (Doc ID 551819.1).
CAUSE
There are 2 rows in MRP_AP_APPS_INSTANCES_ALL because the instance was cloned from a different instance.
SOLUTION
To implement the solution, please execute the following steps:
1. In SQL Developer, run the following query:
select * from mrp_ap_apps_instances_all;
You should see 2 rows in the table. Only 1 row should exist in the table.
2. Delete the incorrect record from the table.  The problem row is from the cloned instance.
3. Retest the issue.