Oracle Application DBA

Starting WLS with line:
/erp/apps/java/jdk1.8.0_65/bin/java -server -d64 -Xms4096m -Xmx4096m -Dweblogic.Name=AdminServer -Djava.security.policy=/erp/apps/obi12/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dopss.version=12.2.1 -
............
141107> <Version: WebLogic Server 12.2.1.0.0 Tue Oct 6 10:05:47 PDT 2015 1721936>
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.credstore.ssp.CsfWalletManager openWallet
WARNING: Opening of wallet based credential store failed. Reason java.io.IOException
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.config.OpssCommonStartup preStart
INFO: Jps startup failed.
<Nov 17, 2016 1:09:04 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>
<Nov 17, 2016 1:09:04 PM CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
oracle.security.jps.JpsException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.config.OpssCommonStartup.preStart(OpssCommonStartup.java:334)
at oracle.security.jps.JpsStartup.preStart(JpsStartup.java:286)
Caused by: oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:187)
Caused by: java.io.IOException
at oracle.security.pki.OracleWallet.open(Unknown Source)
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:175)
... 30 more
Caused by: java.lang.ExceptionInInitializerError
at oracle.security.pki.OracleFileSSOWalletImpl.a(Unknown Source)
... 32 more
Caused by: java.lang.RuntimeException: java.io.IOException: Disc quota exceeded
at oracle.security.pki.FileLocker.<clinit>(Unknown Source)
... 33 more
Caused by: java.io.IOException: Disc quota exceeded
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at java.io.File.createTempFile(File.java:2070)
... 34 more
SOLUTION
Review the OS level quota setting for the user and either increase it or move unwanted files to make more space
For example "quota" command will display the users' disk usage and limits on Linux OS

Jan 7, 2014 9:09:20 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:30 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:40 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:50 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:10:00 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
SOLUTION
When a server is starting, a .lok file is created. This is found under $FMW_HOME/servers/SERVERNAME/tmp/SERVERNAME.lok. If the file exists , then it cannot be created and the server cannot start.
To correct this delete the existing file or kill the running process that is holding such file.

After killing the Weblogic server, the following errors were raised on restart:
<23-Jul-009 12:57:00 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. R
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
        at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:75)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:374)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:125)
And then on a re-attempt:
<23-Jul-2009 12:58:06 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:44)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:107)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Caused by: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.Heap.open(Heap.java:166)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:85)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:350)
        at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:202)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:61)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:42)
        ... 4 more
Caused by: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.StoreDir.throwIOException(StoreDir.java:213)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:103)
        at weblogic.store.io.file.Heap.open(Heap.java:164)
        ... 9 more
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.open(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:51)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:138)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:142)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:95)
        ... 10 more

This problem sometimes occurs if you killed the bin/startWebLogic process via:
SOLUTION
There are two possible solutions:
remove the lock files and restart
kill any remaining server processes (if they exist) and restart
Solution1: Removing the lock files and restarting
1. Find -name "*.DAT" files in user_projects/domains/<your domain name> directory (as appropriate) and move or remove them, for example:
$ find . -name "*DAT"
./servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
./servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT
NOTE: Gentle reminder that Unix is case sensitive. you do need to make sure you do NOT delete the file <CM HOME>/security/SerializedSystemIni.dat, it is the files ending with "*.DAT" not "*.dat"
Deleting .DAT files: with reference to Note 1332274.1. Please be aware of the implications of removing the .DAT files. "...There may be error messages for transactions that were already committed and finished during the WebLogic Server run. However, the transaction entries will still be available in the transaction persistent store file (.DAT file)..."
2. Remove lock files: "EmbeddedLDAP.lok" and "AdminServer.lok" (maybe some more). These files are usually under paths such as below
/opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok
opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok
3. You can use the following shell script to find out which process is listening to your port, and kill this via kill -TERM:
#!/bin/ksh
line='---------------------------------------------'
pids=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')
if [ $# -eq 0 ]; then
   read ans?"Enter port you would like to know pid for: "
else
   ans=$1
fi
for f in $pids
do
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $ans"
   if [ $? -eq 0 ]; then
      echo $line
      echo "Port: $ans is being used by PID:\c"
      /usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
   fi
done
exit 0
4. Execute:
nohup ./startWebLogic.sh &
Solution2: kill any remaining server processes (if they exist) and restart
Using the example of a managed server
1. Find the PID of the managed server
ps -ef | grep ManagedServer
2. Kill the managed server process with a -KILL
kill -9 <pid of managed server>
3. Restart the Admin server and Managed servers
nohup ./startWebLogic.sh &
Note: the startWebLogic.sh script is usually run with nohup so the user can exit the shell leaving the process running.
For Windows based servers please see the following
1. The AdminServer.lok file default location is usually: C:\Oracle\Middleware\user_projects\domains\<domain_name>\servers\AdminServer\tmp\
2. Start startWebLogic.cmd
Note: If AdminServer.lok cannot be deleted:
i) end all java.exe processes in Task Manager on WebLogic server,
ii) delete AdminServer.lok file

JPS-01538: The default policy provider was not found
JPS-01514: The default context is missing in jps-config.xml.
JPS-04301: Cannot read from policy store
<Jun 22, 2016 12:19:11 AM PDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.>
<Jun 22, 2016 12:19:11 AM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1394)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1018)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:273)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:705)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01514: The default context is missing in jps-config.xml.
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:665)
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:650)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:650)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-04301: Cannot read from policy store.
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.buildFromFile(XmlPolicyStore.java:298)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.<init>(XmlPolicyStore.java:177)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:106)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:76)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
SOLUTION
Delete the pki*.lck files present under /tmp folder and restart Admin Server
Note: Verify the TEMP folder used by Weblogic and look for the lock files under that folder, as it may not be /tmp always

startup weblogic fails with:
Starting server oacore_server1 ...
Error Starting server oacore_server1: weblogic.nodemanager.NMException: Exception while starting server 'oacore_server1'
ERROR: Unable to start up the managed server oacore_server1
07/06/14-13:37:05 :: admanagedsrvctl.sh: exiting with status 1
The oacore_server1.out log file under $EBS_DOMAIN_DOMAIN\servers\oacore_server1\logs\ shows:
<Jul 6, 2014 1:33:46 PM EDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 23.25-b01 from Oracle Corporation>
<Jul 6, 2014 1:37:04 PM EDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on ..EBS_domain/servers/oacore_server1/tmp/oacore_server1.lok. Server may already be running
    at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:206)
    at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:67)
    at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:74)
    at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
    at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
    at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
    at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
    at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
    at weblogic.Server.main(Server.java:71)
The previous shutdown was not clean and the .lok file from the previous startup was still there.
SOLUTION
1. Remove or rename the file: $EBS_DOMAIN_HOME/servers/oacore_server1/tmp/oacore_server1.lok.
2. Re-teststarting EBS again.

LogFileName = adop_20140311_152649.log
ERROR
[UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: <perl /u01/EBS/VIS/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/EBS/VIS/fs2/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -patchcontextfile=/u01/EBS/VIS/fs1/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -promptmsg=hide -console=off -mode=create -sessionid=5 -timestamp=20140311_152649 -outdir=/u01/EBS/VIS/fs_ne/EBSapps/log/adop/5/fs_clone_20140311_152649/UPG_apcappsx21>.
LogFileName = fsclone_stage/FSCloneStageAppsTier_03111527.log
ERROR
AdminServer is in RUNNING mode.
# Set WLS password...
# Passing the apps password to apps Tier...
runMSList size6
patchMSList size5
ERROR: Number of servers are not in sync between Run and Patch Context.
ERROR: FSClone would exit...
CAUSE
runMSList size6 means there are 6 managed servers in Run environment, patchMSList size5 means there are 5 managed servers in Patch environment, which caused error :
ERROR: Number of servers are not in sync between Run and Patch Context.
Use following sql to confirm :
SELECT
extractValue(XMLType(TEXT),'//oa_service_name[@oa_var="s_adminservername"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='patch'
and CTX_TYPE = 'A';
-------------------------
AdminServer
oacore_server1:7201
forms_server1:7401
oafm_server1:7601
forms-c4ws_server1:7801

SELECT
extractValue(XMLType(TEXT),'//oa_service_name[@oa_var="s_adminservername"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='run'
and CTX_TYPE = 'A';
-------------------------
AdminServer
oacore_server1:7202,oacore_server2:7204
forms_server1:7402
oafm_server1:7602
forms-c4ws_server1:7802

Run environment has one more managed server "oacore_server2" than Patch environment.
SOLUTION
Solution A: Remove/delete the managed server "oacore_server2" from the RUN environment filesystem
1.  Managed server oacore_server2 should be shutdown before it can be deleted(in this case, i shut down apache service with adapcctl.sh)
2.  Execute the below command on the Application tier node where the managed server resides:
# perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-managedserver -contextfile=$RUN_BASE/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -managedsrvname=oacore_server2 -servicetype=oacore -logfile=/tmp/remove_manage_server.log
NOTE: check the run filesystem context file for confirmation of no "oacore_server2" information
3. On the node containing the Oracle HTTP Server, run autoconfig:
# sh $INST_TOP/admin/scripts/adautocfg.sh
NOTE: The database server and database listener must remain available during the AutoConfig run, but all other database tier services should be shut down
4.  Execute the following script to remove details of the deleted managed server from the Oracle HTTP Server configuration files:
# perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -ctxfile=$RUN_BASE/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -outfile=/tmp/delete_http_config.log
5.  On the node that contains the Oracle HTTP Server, bounce the Oracle HTTP Server:
$ sh $ADMIN_SCRIPTS_HOME/adapcctl.sh stop
$ sh $ADMIN_SCRIPTS_HOME/adapcctl.sh start
6.  Re-issue fs_clone phase

12.2.4 version, When attempting to run adop phase=fs_clone the following error occurs:
In adop log file:
[UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: "perl /DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/DATA/app/appldev/TEST1/fs1/inst/apps/TEST1_abc/appl/admin/TEST1_abc.xml -patchcontextfile=/DATA/app/appldev/TEST1/fs2/inst/apps/TEST1_abc/appl/admin/TEST1_abc.xml -promptmsg=hide -console=off -mode=create -sessionid=18 -timestamp=20150325_010624 -outdir=/DATA/app/appldev/TEST1/fs_ne/EBSapps/log/adop/18/fs_clone_20150325_010624/TEST1_abc".
In txkADOPPreparePhaseSynchronize log file
===============================
Inside runADOPCustomSyncUp()...
===============================
Executing SYSTEM command: perl /DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCustomSyncUp.pl -contextfile=/DATA/app/appldev/TEST1/fs1/inst/apps/TEST1_abc/appl/admin/TEST1_abc.xml -logfile=/DATA/app/appldev/TEST1/fs_ne/EBSapps/log/adop/18/fs_clone_20150402_232202/TEST1_abc/TXK_SYNC_create_Thu_Apr_2_23_27_26_2015/txkADOPCustomSyncUp.log -promptmsg=hide
EXIT STATUS: 1
/DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCustomSyncUp.pl did not go through successfully.
LOG FILE: /DATA/app/appldev/TEST1/fs_ne/EBSapps/log/adop/18/fs_clone_20150402_232202/TEST1_abc/TXK_SYNC_create_Thu_Apr_2_23_27_26_2015/txkADOPCustomSyncUp.log.
*******FATAL ERROR*******
PROGRAM : (/DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Fri Apr 3 00:22:46 2015
FUNCTION: main::runADOPCustomSyncUp [ Level 1 ]
ERRORMSG: /DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCustomSyncUp.pl did not go through successfully.
In FSCloneStageAppsTier log file:
Attempting to delete: /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc/FMW/t2pjdk
Creating prereq/webtier directory
Attempting to delete: /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc//prereq
Copying /DATA/app/appldev/TEST1/fs1/FMW_Home/webtier/oui to /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc//prereq/webtier
Copying /DATA/app/appldev/TEST1/fs1/FMW_Home/webtier/inventory/Scripts to /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc//prereq/webtier
START: Creating FMW archive.
Running /DATA/app/appldev/TEST1/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc/FMW/t2pjdk -al /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc/FMW/FMW_Home.jar -smw /DATA/app/appldev/TEST1/fs1/FMW_Home -ldl /DATA/app/appldev/TEST1/fs1/inst/apps/TEST1_abc/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 200 milliseconds, returning status 1
ERROR: Script failed, exit code 1
CAUSE
The issue is caused during clone staging process t2pjdk directly couldn't deleted because the customer placed the SSL certificate file under /DATA/app/appldev/INNOV/fs2/EBSapps/comn/adopclone_abc/FMW/t2pjdk/jre/lib/security/ and the file couldn't deleted.
During WLS component staging copyConfig utility is failing. The reason is /FMW/t2pjdk directory didn't stage correctly. During t2pjdk directory staging ,if t2pjdk directory already exist the system are trying to delete it. But if for some reason some files within t2pjdk is in already used state then code couldn't able to delete the t2pjdk. The customer enabled SSL on EBS and they copied the custom certificate file to the directory /DATA/app/appldev/INNOV/fs2/EBSapps/comn/adopclone_abc/FMW/t2pjdk/jre/lib/security/ to import it to the cacerts, The file is not seeded file so the t2pjdk directly couldn't deleted as normal.
The workaround is to remove the t2pjdk directory manually. After that, fs_clone complete normally. But after the customer complete a full adop cycle and run fs_clone again, the issue is reproduced.
SOLUTION
To implement the solution, please execute the following steps:
1. Move the custom certificate file from $COMMON_TOP/util/jdk64/jre/lib/security directory and place it outside of $COMMON_TOP/util/jdk64/ directory. (for both RUN and PATCH file system)
2.. Remove /DATA/app/appldev/INNOV/fs1/EBSapps/comn/adopclone_abd/FMW/t2pjdk/directory. (as fs1 is RUN File system)
3. Run fs_clone again.

The actual error is from FSCloneStageAppsTier_<time stamp>.log
START: Creating WLS config archive.
Script Executed in 200 milliseconds, returning status 1
ERROR: Script failed, exit code 1
CAUSE
The t2pjdk directory is not created completely.
SOLUTION
1. First refer to FSCloneStageAppsTier_<time stamp>.log file to get the t2pjdk directory path.
Following content might be found:
Creating t2pjdk directory
Directory /DATA/app/appldev/ORACLE/fs1/EBSapps/comn/adopclone_test/FMW/t2pjdk already exists.
So the t2pjdk directory path is /DATA/app/appldev/ORACLE/fs1/EBSapps/comn/adopclone_test/FMW/t2pjdk
2. Move the t2pjdk directory to a temp location
For example:
mkdir /tmp/t2pjdk_backup
mv /DATA/app/appldev/ORACLE/fs1/EBSapps/comn/adopclone_test/FMW/t2pjdk /tmp/t2pjdk_backup/FMW/t2pjdk
3. Rerun "adop phase=prepare".
Please be aware :
The same solution can be used for the adpreclone steps.

Error can be seen in adop_<TIME_STAMP>.log
[UNEXPECTED]Error occurred running "perl /u01/oracle/OCBPRD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/OCBPRD/fs1/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -patchcontextfile=/u01/oracle/OCBPRD/fs2/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -promptmsg=hide -console=off -mode=create -sessionid=3 -timestamp=20151223_192751 -outdir=/u01/oracle/OCBPRD/fs_ne/EBSapps/log/adop/3/fs_clone_20151223_192751/OCBPRD_ebsapp01"
[UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u01/oracle/OCBPRD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/OCBPRD/fs1/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -patchcontextfile=/u01/oracle/OCBPRD/fs2/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -promptmsg=hide -console=off -mode=create -sessionid=3 -timestamp=20151223_192751 -outdir=/u01/oracle/OCBPRD/fs_ne/EBSapps/log/adop/3/fs_clone_20151223_192751/OCBPRD_ebsapp01"
CAUSE
The context variable s_invPtrLoc is not getting populated in patch context file. FSCloneApplyAppsTier log prints correct value for s_invPtrLoc but the value is missing in patch context file.
Context file customization is present.
From stage logs(FSCloneStage.log) below can be seen:
Copying /u01/oracle/OCBPRD/fs1/EBSapps/appl/ad/12.0.0/admin/template/custom/adxmlctx.tmp to /u01/oracle/OCBPRD/fs1/EBSapps/comn/adopclone_ebsapp01//context/apps/adxmlctx.tmp
The adxmlctx.tmp is being copied from custom folder.
Custom folder has old version of adxmlct.tmp (version 120.401.12020000.9.1202010.10)
While the actual version is 120.401.12020000.54
During clonecontext the template being used is from the stage which have the older version. Due that reason the patch context file is not getting created properly.
That's the reason the version of context files which can be seen in FND_OAM_CONTEXT_FILES are different between RUN and PATCH filesystems.
RUN fs version : 120.401.12020000.54
PATCH fs version : 120.401.12020000.9.1202010.10
SOLUTION
To solve the issue please do the below:
1.Remove the file $AD_TOP/admin/template/custom/adxmlctx.tmp
2.Run fs_clone with option force=yes

On Oracle Applications 12.2.4,
when attempting to open the status diagram,
the following error occurs:
[1953]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Exception in getHtml: java.lang.IllegalStateException: Cipher not initialized
[1954]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Complete Exception stack is:
[1957]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:java.lang.IllegalStateException: Cipher not initialized at javax.crypto.Cipher.checkCipherState(Cipher.java:1672)
The issue can be reproduced at will with the following steps:
1. Workflow Administrator Web Applications
2. Administrator Workflow
3. Status Monitor
CAUSE
The Workflow code wasn't able to open the URL for the workflow diagram due to SSL setting was missing.
The debug information in NewWorkflowError.docx shows the following error when trying to connect to the url:
[1505]:STATEMENT:[fnd.wf.WFBaseUtil]:Done openConnection [1505]:STATEMENT:[fnd.wf.WFBaseUtil]:String representation of urlConn: weblogic.net.http.SOAPHttpsURLConnection:https://<serverName>:<portNumber>/forms/frmservlet?appletmode=nonforms&HTMLpageTitle=&HTMLpreApplet=&code=oracle.apps.fnd.wf.Monitor&width=800&height=400&archive=/OA_JAVA/oracle/apps/fnd/jar/wfmon.jar,/OA_JAVA/oracle/apps/fnd/jar/fndewt.jar,/OA_JAVA/oracle/apps/fnd/jar/fndswing.jar,/OA_JAVA/oracle/apps/fnd/jar/fndbalishare.jar,/OA_JAVA/oracle/apps/fnd/jar/fndctx.jar...
1505]:STATEMENT:[fnd.wf.WFBaseUtil]:Done setUseCaches
[1953]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Exception in getHtml: java.lang.IllegalStateException: Cipher not initialized
[1954]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Complete Exception stack is:
[1957]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:java.lang.IllegalStateException: Cipher not initialized at javax.crypto.Cipher.checkCipherState(Cipher.java:1672)
SOLUTION
To implement the solution, please execute the following steps:
1. Bring up the WLS Admin console for the EBS instance ->
http://host:port/console and login as the weblogic user.
2. Click on Lock & Edit.
3. Under the Domain Structure, navigate to EBS Domain -> Environment -> Servers.
4. Click on oacore_server1, and click on the SSL tab.
5. Click on Advanced to expand that section.
6. Look for the 'Use JSSE SSL' parameter and enable this by placing a check in the box next to it.
7. Click on Activate Changes.
8. Restart the managed server.
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server1
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server1
9. Retest the issue.
10. Migrate the solution as appropriate to other environments.

When attempting to apply 12.2.4 upgrade patch(es) (adop phase=apply apply_mode=downtime patches=17919161,21900859 merge=yes)
the following error occurs.
The following Oracle Forms objects did not generate successfully:
pa forms/US PAXURVPS.fmx
The patch has FAILED,
Please check the adpatch logs for more details.
Object APPS.PA_STATUS_COMMITMENTS_V is invalid
SOLUTION
Please recompile APPS.PA_STATUS_COMMITMENTS_V and advise if there are errors, if not then use adadmin to compile form PAXURVPS.fmx
One can use the following to display errors if there are any errors when compiling the view
SQL>alter view APPS.PA_STATUS_COMMITMENTS_V compile;
show error;
OR
SQL> select owner, name, type, text from dba_errors where name = 'PA_STATUS_COMMITMENTS_V' order by 1,2,3,line;

Customer faces 5000~8000 open cursor every day and due to this Java Listeners goes down.

SELECT 
  OC.SID, 
  OC.USER_NAME, 
  S.PROGRAM, 
  COUNT(*) COUNTER, 
  OC.SQL_ID, 
  OC.SQL_TEXT 
  FROM 
  V$OPEN_CURSOR OC, 
  V$SESSION S 
  WHERE 
  OC.SQL_TEXT NOT LIKE ''%obj#,%'' 
  AND OC.SQL_TEXT NOT LIKE ''%grantee#,%'' 
  AND OC.SQL_TEXT NOT LIKE ''%privilege#%'' 
  AND OC.SQL_TEXT NOT LIKE ''%/*+ rule */%'' 
  AND OC.SQL_TEXT NOT LIKE ''%col#%'' 
  AND OC.SQL_TEXT NOT LIKE ''%sys.mon_mods$%'' 
  AND OC.SQL_TEXT NOT LIKE ''%obj#=%'' 
  AND OC.SQL_TEXT NOT LIKE ''%update$,%'' 
  AND OC.SID=S.SID 
  AND OC.USER_NAME NOT IN 
 (''SYS'',''DBSNMP'',''SYSTEM'',''SYSMAN'',''RMAN'') 
  AND lower(SQL_TEXT) like ''%enqueue%'' 
  GROUP BY 
  OC.SID, 
  OC.USER_NAME, 
  S.PROGRAM, 
  OC.SQL_ID, 
  OC.SQL_TEXT 
  HAVING COUNT(*)>1 
  ORDER BY 
  COUNT(*) desc, 
  OC.USER_NAME, 
  OC.SID, 
  OC.SQL_TEXT;

Oracle Applications R12, users are not able to access Forms directly via the following url:
http://hostname.domain:port/forms/frmservlet
The following error would pop out:
APP-FND-01542: This Applications Server is not authorized to access this database.
The goal of this document is to allow users access to the Oracle Applications R12 directly via Forms especially in cases where access to the following url is not available:
http://hostname.domain:port/OA_HTML/AppsLogin
NOTE: Accessing Forms directly is not supported and should only be used for diagnostic and troubleshooting purposes.
SOLUTION
1. Edit $CONTEXT_FILE which is located in $INST_TOP/appl/admin
2. Change
<appserverid_authentication oa_var="s_appserverid_authentication">SECURE</appserverid_authentication>
to
<appserverid_authentication oa_var="s_appserverid_authentication">OFF</appserverid_authentication>
Application Server Security Authentication can take one of the following values {ON, OFF, SECURE}.
OFF - Server security is not checked. Any application server machine can access the database.
ON - Some level of trust is required to access the database. Either the Application Server is registered with the database or the module and version ID are known to be trusted.
SECURE - Full trust is required for access to the database. Only registered Application Server machines and trusted code modules may connect.
3. Run $INST_TOP/admin/scripts/adautocfg.sh to instantiate the changes
4. Users are now able to access Forms directly.
NOTE:
Remember to set the Application Server Security Authentication back to SECURE by settting the value of s_appserverid_authentication to SECURE and running autoconfig.

ADOP cleanup reports following error at ad.plsql.ad_zd_sys.drop_covered_object.
[EVENT]     Calling cleanup in STANDARD mode...
   [EVENT]     Log: @ADZDSHOWLOG.sql "2016/01/01 01:01:01"
   [ERROR]     ORA-04045: errors during recompilation/revalidation of APPS.HXT_HOUR_DEDUCTION_RULES_WHO
ORA-04023: Object HXT.HXT_HOUR_DEDUCTION_RULES
   [ERROR]     Failed to execute SQL statement :
                               declare
                              begin
                                 ad_zd_log.message('ad.bin.adop','WARNING','');
                              exception
                                 when others then
                                    raise_application_error(-20001,'Error while calling ad_zd_log.message(ad.bin.adop,WARNING,).' || sqlerrm);
                              end;
   [ERROR]     Error Message :
   [ERROR]     *** Error occurred while calling ad_zd_log. Please check log file for the details. ***
Steps to Reproduce:
The issue can be reproduced at will with the following steps:
 Run ADOP Cleanup phase.
 Error is printed in the log.
SOLUTION
1. Run the command below to compile the trigger.
alter trigger APPS.HXT_HOUR_DEDUCTION_RULES_WHO compile;
2. Run the query below and make sure the status is Enabled.
select owner, trigger_name, status
from dba_triggers
where trigger_name = 'HXT_HOUR_DEDUCTION_RULES_WHO';
  Expected result:
   OWNER      TRIGGER_NAME                   STATUS
   ---------- ------------------------------ ---------
   APPS       HXT_HOUR_DEDUCTION_RULES_WHO   ENABLED
   APPS       HXT_HOUR_DEDUCTION_RULES_WHO   ENABLED
3. Run the query below and make sure the status is valid.
select owner,object_name, object_type,status
from dba_objects
where object_name = 'HXT_HOUR_DEDUCTION_RULES_WHO';
   Expected result:
   OWNER OBJECT_NAME                    OBJECT_TYPE STATUS
   ----- ------------------------------ ----------- -------
   APPS  HXT_HOUR_DEDUCTION_RULES_WHO   TRIGGER     VALID

CEST]:1409674186377:-1:-1:hostname.domain:10.32.241.5:-1:-1:-1:-1:GUEST(6):1769119955:Thread[[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads]:10.32.241.5:49173:1409674008272:4:UNEXPECTED:[fnd.framework.OAException]:oracle.adf.mds.exception.MDSRuntimeException: Unable to find component with absolute reference = /oracle/apps/fnd/attributesets/Buttons/Submit, XML Path = {1}. Please verify that the reference is valid and the definition of the component exists either on the File System or in the MDS Repository.  
/oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
[Sep 2, 2014 6:09:46 PM CEST]:1409674186379:-1:-1:hostname.domain:10.32.241.5:-1:-1:-1:-1:GUEST(6):1769119955:Thread[[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads]:10.32.241.5:49173:1409674008272:4:UNEXPECTED:[UNEXPECTED_OA_EXCEPTION: user name = GUEST: responsibility name = null]:oracle.apps.fnd.framework.OAException: oracle.adf.mds.exception.MDSRuntimeException: Unable to find component with absolute reference = /oracle/apps/fnd/attributesets/Buttons/Submit, XML Path = {1}. Please verify that the reference is valid and the definition of the component exists either on the File System or in the MDS Repository.  
/oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
at oracle.apps.fnd.framework.OAException.wrapperException(OAException.java:912)
at oracle.apps.fnd.framework.webui.OAPageErrorHandler.prepareException(OAPageErrorHandler.java:1169)
The oacore server log file (oacore_server1.out) contains the following error stack:
(WebAppServletContext.java:2181)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: org.xml.sax.SAXParseException; systemId: /oracle/apps/fnd/attributesets/Buttons; lineNumber: 1; columnNumber: 40; /oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
at oracle.xml.parser.v2.XMLError.flushErrorHandler(XMLError.java:422)
at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:287)
at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:414)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:355)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:226)
at oracle.cabo.share.xml.ParserAdapter.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:283)
at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:115)
at oracle.adf.mds.adapters.DBAdapter.getElementData(DBAdapter.java:324)
... 81 more
Cause:
org.xml.sax.SAXParseException; systemId: /oracle/apps/fnd/attributesets/Buttons; lineNumber: 1; columnNumber: 40; /oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
at oracle.xml.parser.v2.XMLError.flushErrorHandler(XMLError.java:422)
at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:287)
at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:414)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:355)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:226)
at oracle.cabo.share.xml.ParserAdapter.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
If you run:
SQL> exec jdr_utils.printdocument('/oracle/apps/ar/hz/attributesets/HzParties');
the results differ if VPD policies are enabled/disabled, i.e. with VPD policies enabled this returns no rows.
CAUSE
The SQL profiles are setting optimizer_features_enable to 10.1.0.5 and this setting is not going to work with EBR , which is an 11g feature.
This is explained in:
Bug 19830345 - QUERY RETURNS NO ROWS AGAINST SOME TABLES PROTECTED BY VPD
SOLUTION
To implement the solution, please execute the following steps:
1) run this query to discover all the profiles that set the OFE parameter to 10.1.0.5 :
SELECT o.name, d.comp_data
FROM SQLOBJ$ o, SQLOBJ$DATA d
WHERE o.signature = d.signature
AND o.category = d.category
and COMP_DATA like '%10.1%'
ORDER BY o.name;
2) for each o.name run this :
exec DBMS_SQLTUNE.DROP_SQL_PROFILE ( '')
3) Re-test the issue.

E-Business Suite 12.2 Applications DBA, Online Patching issues
While applying a patch (For example Patch 23510855 - ISG R12.2 CONSOLIDATED PATCH), a worker fails with the below error:
drop_error_codes_map.sql &un_fnd'
Connected.
PL/SQL procedure successfully completed.
drop view APPLSYS.ERROR_CODES_MAP#
ERROR at line 1:
ORA-00942: table or view does not exist
CAUSE
The view APPLSYS.ERROR_CODES_MAP has already been dropped.
A number of different patches could have been applied which would drop the view, including the following:
17909318 R12.ATG_PF.C.delta.4 R12.ATG_PF.C.delta.4 ATG_PF
17919161 12.2.4 ORACLE E-BUSINESS SUITE 12.2.4 RELEASE UPDATE PACK AU
17765665 R12.OWF.C ISG AGENT CONSOLIDATED ONE-OFF PATCH FOR 12.2.3 OWF
The following select statements can be used to check for the above patches:
SELECT ad_patch.is_patch_applied('R12',-1,17909318) FROM dual;
SELECT ad_patch.is_patch_applied('R12',-1,17919161) FROM dual;
SELECT ad_patch.is_patch_applied('R12',-1,17765665) FROM dual;
SOLUTION
To resolve the issue test the following steps in a development instance and then migrate accordingly:
1. Verify view does not exist:
SQL> SELECT OWNER, VIEW_NAME FROM ALL_VIEWS WHERE upper(VIEW_NAME) LIKE 'ERROR_CODES_MAP%';
2. use adctrl hidden option 8 to skip the job.
3. resume the adop session using options:
restart=yes abandon=no
4. Confirm the patch completes successfully.

On : 12.2.3 version, Upgrade related issues
When attempting to perform "adop phase=finalize"
the following error occurs.
ERROR
-----------------------
[UNEXPECTED]adop cannot detect any application tier nodes in FND_NODES table. Ensure ICM is running and run autoconfig on all nodes
[UNEXPECTED]Error while checking if this is a multi node instance
STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Login to server
2. adop phase=finalize
CAUSE
The issue occurs due to a null value for the concurrent/admin node status column in table FND_NODES.
More details are available via the following unpublished defect:
Bug 18533116 - AD:ADOP IN HOTPATCH FAILED FOR R12.TXK.C.DELTA.5
"In AD.Delta.4+one off patches we have a dependency on FND_NOTES.STATUS column
The column will be automatically set to 'Y' when ICM is up.
When the ICM is down the query will not detect any nodes with status 'Y',  and adop proceeds with only the current/admin node (inserts rows into AD_ADOP_SESSIONS for current/admin node only."
SOLUTION
    A. Set the "run" Application Tier environment
    B. Start ICM
    C. Run "adop phase=finalize" again.

On EBS 12.2 when applying a patch using adop.sh in the prepare phase it fails with the error:
ERROR at line 1:
ORA-20001: Error: while calling ad_zd.cleanup(normal).ORA-20007: Ancestors of
Current RUN Edition are not retired
ORA-06512: at line 7
The issue can be reproduced at will with the following steps:
1. Apply patch using adop.sh.
CAUSE
The following data from customer environment shows V_20120409_0455 not retired. The other edition, ORA$BASE is correctly retired.
select apps.ad_zd.get_edition('OLD') "OLD",
  apps.ad_zd.get_edition('RUN') "RUN",
  apps.ad_zd.get_edition('PATCH') "PATCH"
from dual;
EDITION_NAME PARENT_EDITION_NAME USA
------------------------------ ------------------------------ ---
ORA$BASE YES
V_20120409_0455 ORA$BASE YES
V_20120521_0936 V_20120409_0455 YES
column CURRENT_EDITION format a30
column EDITION_TYPE format a15
select apps.ad_zd.get_edition "CURRENT_EDITION",
  apps.ad_zd.get_edition_type "EDITION_TYPE"
from dual;
OLD RUN PATCH
-------------------- -------------------- --------------------
V_20120409_0455 V_20120521_0936
Previous patch application did not get retired.
SOLUTION
1. Please run the following command to retire the edition:
SQL> exec sys.ad_zd_sys.retire_old_editions;
After this the following sql should not return any row for V_20120409_0455:
 select *
 from dba_tab_privs
 where table_name in (select edition_name from dba_editions);
2. Once the V_20120409_0455 is retired successfully, one will be able to do the cleanup and proceed further with the adop cycles.

SOLUTION
The system profile 'Local Login Mask' has been obsoleted in R12 and cannot be used to disable Login Assistance link.  Please follow the steps below to disable this link:
1. Logon with Functional Administrator responsibility.
2. Click on 'Personalization' Tab.
3. Enter /oracle/apps/fnd/sso/login/webui in document path and click on 'Go' button.
4. Click on 'Pencil' icon under 'Personalize Page' for oracle/apps/fnd/sso/login/webui/MainLoginPG.
5. Make sure that 'Include' for Site is checked and click on 'apply'
6. Look for (LoginRN.MainLoginRN.ForgotPasswordUrlRowLayoutRN1) and click on 'Pencil' icon in
'Personalize' column.
7. Change 'rendered' property value to 'false' at Site level from LOV and click on Apply.
8. This will disable the link and users logging in after the change will not see the link.

SET PAGESIZE 124
COL DB_NAME FORMAT A8
COL HOSTNAME FORMAT A12
COL LOG_ARCHIVED FORMAT 999999
COL LOG_APPLIED FORMAT 999999
COL LOG_GAP FORMAT 9999
COL APPLIED_TIME FORMAT A12
SELECT DB_NAME, HOSTNAME, LOG_ARCHIVED, LOG_APPLIED,APPLIED_TIME,LOG_ARCHIVED-LOG_APPLIED LOG_GAPFROM(SELECT NAME DB_NAMEFROM V$DATABASE),(SELECT UPPER(SUBSTR(HOST_NAME,1,(DECODE(INSTR(HOST_NAME,’.’),0,LENGTH(HOST_NAME),(INSTR(HOST_NAME,’.’)-1))))) HOSTNAMEFROM V$INSTANCE),(SELECT MAX(SEQUENCE#) LOG_ARCHIVEDFROM V$ARCHIVED_LOG WHERE DEST_ID=1 AND ARCHIVED=’YES’),(SELECT MAX(SEQUENCE#) LOG_APPLIEDFROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND APPLIED=’YES’),(SELECT TO_CHAR(MAX(COMPLETION_TIME),’DD-MON/HH24:MI’) APPLIED_TIMEFROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND APPLIED=’YES’);
================================================================================
SQL> select status,instance_name,database_role fromv$instance,v$database;
STATUS       INSTANCE_NAME    DATABASE_ROLE
——————————————–
OPEN         prim             PRIMARY
SQL> select max(sequence#) from v$archived_log;
MAX(SEQUENCE#)
————–
40
Check the status of the physical standby database and the latest sequence applied on the physcial standby database.
SQL> select status,instance_name,database_role fromv$database,v$instance;
STATUS   INSTANCE_NAME DATABASE_ROLE
——————————————
MOUNTED  stnd          PHYSICAL STANDBY
SQL> select max(sequence#) from v$archived_log where applied=’YES’;
MAX(SEQUENCE#)
————–
40
Check if the Managed Recovery Process (MRP) is active on the physcial standby database.
SQL> select process,status,sequence# from v$managed_standby;
PROCESS   STATUS        SEQUENCE#
——————————-
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
RFS       IDLE          41
RFS       IDLE          0
RFS       IDLE          0
RFS       IDLE          0
MRP0      WAIT_FOR_LOG  41
9rows selected.
Here, MRP is active. The PROCESS Column above shows that MRP is active and is waiting for the log sequence 41.
Step 4:
Cancel the MRP on the physical standby database and open the standby database. The standby database would be opened in the READ-ONLY Mode.
SQL> alter database recover managed standby database cancel;
Database altered.
SQL> alter database open;
Database altered.
SQL> select status,instance_name,database_role,open_mode fromv$database,v$instance;
STATUS INSTANCE_NAME  DATABASE_ROLE    OPEN_MODE
—————————————————
OPEN   stnd           PHYSICAL STANDBY READ ONLY
Step 6:
Now start the MRP on the physical standby database.
SQL> alter database recover managed standby database disconnectfrom session;
Database altered.
SQL> select process,status,sequence# from v$managed_standby;
PROCESS   STATUS        SEQUENCE#
——————————-
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
RFS       IDLE          41
RFS       IDLE          0
RFS       IDLE          0
RFS       IDLE          0
MRP0      WAIT_FOR_LOG  41
9rows selected.
Here, you can see that the MRP is active and is waiting for the log sequence 41 and also the physical standby database is opened in READ-ONLY mode which would allow users to use the physical standby database for fetching reports.
Managing dataguard – monitoring scripts
  LOGS             TIME—————- ——————
Last applied   :  16-JUL-09:09:24:16
Last received :  16-JUL-09:09:28:36
select ‘Last applied  : ‘ Logs, to_char(next_time,’DD-MON-YY:HH24:MI:SS’) Timefrom v$archived_log
where sequence# = (select max(sequence#) from v$archived_log where applied=’YES’)
union
select ‘Last received : ‘ Logs, to_char(next_time,’DD-MON-YY:HH24:MI:SS’) Time
from v$archived_log
where sequence# = (select max(sequence#) from v$archived_log);
NAME                   VALUE                  UNIT———————- ———————- ———————————–
apply finish time      +00 00:02:07.2         day(2) to second(1) interval
apply lag              +00 00:01:59           day(2) to second(0) interval
estimated startup time 16                     second
standby has been open  N
transport lag          +00 00:00:00           day(2) to second(0) interval
Time Computed: 16-JUL-2009 09:33:16
selectNAME Name,
VALUE Value,
UNIT Unit
from v$dataguard_stats
union
select null,null,’ ‘ from dual
union
select null,null,’Time Computed: ‘||MIN(TIME_COMPUTED)
from v$dataguard_stats;
  Redo onsite
——————–
16-JUL-2009 09:42:44
select to_char(max(last_time),’DD-MON-YYYY HH24:MI:SS’) “Redo onsite”from v$standby_log