R12.2 Apps DBA. Powered by Blogger.

Weblogic Server Startup Fails with Error JPS-01050, "Caused by: java.io.IOException: Disc quota exceeded"

, , No comments :
Starting WLS with line:
/erp/apps/java/jdk1.8.0_65/bin/java -server -d64 -Xms4096m -Xmx4096m -Dweblogic.Name=AdminServer -Djava.security.policy=/erp/apps/obi12/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dopss.version=12.2.1 -
............
141107> <Version: WebLogic Server 12.2.1.0.0 Tue Oct 6 10:05:47 PDT 2015 1721936>
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.credstore.ssp.CsfWalletManager openWallet
WARNING: Opening of wallet based credential store failed. Reason java.io.IOException
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.config.OpssCommonStartup preStart
INFO: Jps startup failed.
<Nov 17, 2016 1:09:04 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>
<Nov 17, 2016 1:09:04 PM CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
oracle.security.jps.JpsException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.config.OpssCommonStartup.preStart(OpssCommonStartup.java:334)
at oracle.security.jps.JpsStartup.preStart(JpsStartup.java:286)
Caused by: oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:187)
Caused by: java.io.IOException
at oracle.security.pki.OracleWallet.open(Unknown Source)
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:175)
... 30 more
Caused by: java.lang.ExceptionInInitializerError
at oracle.security.pki.OracleFileSSOWalletImpl.a(Unknown Source)
... 32 more
Caused by: java.lang.RuntimeException: java.io.IOException: Disc quota exceeded
at oracle.security.pki.FileLocker.<clinit>(Unknown Source)
... 33 more
Caused by: java.io.IOException: Disc quota exceeded
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at java.io.File.createTempFile(File.java:2070)
... 34 more
SOLUTION
Review the OS level quota setting for the user and either increase it or move unwanted files to make more space
For example "quota" command will display the users' disk usage and limits on Linux OS

Weblogic AdminServer fails with "unable to get file lock, will retry" error message

, , No comments :
Jan 7, 2014 9:09:20 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:30 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:40 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:50 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:10:00 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
SOLUTION
When a server is starting, a .lok file is created. This is found under $FMW_HOME/servers/SERVERNAME/tmp/SERVERNAME.lok. If the file exists , then it cannot be created and the server cannot start.
To correct this delete the existing file or kill the running process that is holding such file.

Weblogic Fails To Start With Error Unable To Obtain Lock

, , No comments :
After killing the Weblogic server, the following errors were raised on restart:
<23-Jul-009 12:57:00 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. R
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
        at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:75)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:374)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:125)
And then on a re-attempt:
<23-Jul-2009 12:58:06 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:44)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:107)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Caused by: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.Heap.open(Heap.java:166)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:85)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:350)
        at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:202)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:61)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:42)
        ... 4 more
Caused by: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.StoreDir.throwIOException(StoreDir.java:213)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:103)
        at weblogic.store.io.file.Heap.open(Heap.java:164)
        ... 9 more
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.open(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:51)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:138)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:142)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:95)
        ... 10 more

This problem sometimes occurs if you killed the bin/startWebLogic process via:
SOLUTION
There are two possible solutions:
remove the lock files and restart
kill any remaining server processes (if they exist) and restart
Solution1: Removing the lock files and restarting
1. Find -name "*.DAT" files in user_projects/domains/<your domain name> directory (as appropriate) and move or remove them, for example:
$ find . -name "*DAT"
./servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
./servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT
NOTE: Gentle reminder that Unix is case sensitive. you do need to make sure you do NOT delete the file <CM HOME>/security/SerializedSystemIni.dat, it is the files ending with "*.DAT" not "*.dat"
Deleting .DAT files: with reference to Note 1332274.1. Please be aware of the implications of removing the .DAT files. "...There may be error messages for transactions that were already committed and finished during the WebLogic Server run. However, the transaction entries will still be available in the transaction persistent store file (.DAT file)..."
2. Remove lock files: "EmbeddedLDAP.lok" and "AdminServer.lok" (maybe some more). These files are usually under paths such as below
/opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok
opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok
3. You can use the following shell script to find out which process is listening to your port, and kill this via kill -TERM:
#!/bin/ksh
line='---------------------------------------------'
pids=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')
if [ $# -eq 0 ]; then
   read ans?"Enter port you would like to know pid for: "
else
   ans=$1
fi
for f in $pids
do
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $ans"
   if [ $? -eq 0 ]; then
      echo $line
      echo "Port: $ans is being used by PID:\c"
      /usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
   fi
done
exit 0
4. Execute:
nohup ./startWebLogic.sh &
Solution2: kill any remaining server processes (if they exist) and restart
Using the example of a managed server
1. Find the PID of the managed server
ps -ef | grep ManagedServer
2. Kill the managed server process with a -KILL
kill -9 <pid of managed server>
3. Restart the Admin server and Managed servers
nohup ./startWebLogic.sh &
Note: the startWebLogic.sh script is usually run with nohup so the user can exit the shell leaving the process running.
For Windows based servers please see the following
1. The AdminServer.lok file default location is usually: C:\Oracle\Middleware\user_projects\domains\<domain_name>\servers\AdminServer\tmp\
2. Start startWebLogic.cmd
Note: If AdminServer.lok cannot be deleted:
i) end all java.exe processes in Task Manager on WebLogic server,
ii) delete AdminServer.lok file

WebLogic Admin Server Startup Fails.

, , No comments :
JPS-01538: The default policy provider was not found
JPS-01514: The default context is missing in jps-config.xml.
JPS-04301: Cannot read from policy store
<Jun 22, 2016 12:19:11 AM PDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.>
<Jun 22, 2016 12:19:11 AM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1394)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1018)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:273)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:705)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01514: The default context is missing in jps-config.xml.
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:665)
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:650)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:650)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-04301: Cannot read from policy store.
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.buildFromFile(XmlPolicyStore.java:298)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.<init>(XmlPolicyStore.java:177)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:106)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:76)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
SOLUTION
Delete the pki*.lck files present under /tmp folder and restart Admin Server
Note: Verify the TEMP folder used by Weblogic and look for the lock files under that folder, as it may not be /tmp always

Starting EBS Fails with Exception while starting server oacore_server1 Unable to obtain lock on oacore_server1.lok

, , No comments :
startup weblogic fails with:
Starting server oacore_server1 ...
Error Starting server oacore_server1: weblogic.nodemanager.NMException: Exception while starting server 'oacore_server1'
ERROR: Unable to start up the managed server oacore_server1
07/06/14-13:37:05 :: admanagedsrvctl.sh: exiting with status 1
The oacore_server1.out log file under $EBS_DOMAIN_DOMAIN\servers\oacore_server1\logs\ shows:
<Jul 6, 2014 1:33:46 PM EDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 23.25-b01 from Oracle Corporation>
<Jul 6, 2014 1:37:04 PM EDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on ..EBS_domain/servers/oacore_server1/tmp/oacore_server1.lok. Server may already be running
    at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:206)
    at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:67)
    at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:74)
    at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
    at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
    at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
    at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
    at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
    at weblogic.Server.main(Server.java:71)
The previous shutdown was not clean and the .lok file from the previous startup was still there.
SOLUTION
1. Remove or rename the file: $EBS_DOMAIN_HOME/servers/oacore_server1/tmp/oacore_server1.lok.
2. Re-teststarting EBS again.

Error Number of servers are not in sync between Run and Patch Context While fs_clone of patching

, , , No comments :
LogFileName = adop_20140311_152649.log
ERROR
[UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: <perl /u01/EBS/VIS/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/EBS/VIS/fs2/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -patchcontextfile=/u01/EBS/VIS/fs1/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -promptmsg=hide -console=off -mode=create -sessionid=5 -timestamp=20140311_152649 -outdir=/u01/EBS/VIS/fs_ne/EBSapps/log/adop/5/fs_clone_20140311_152649/UPG_apcappsx21>.
LogFileName = fsclone_stage/FSCloneStageAppsTier_03111527.log
ERROR
AdminServer is in RUNNING mode.
# Set WLS password...
# Passing the apps password to apps Tier...
runMSList size6
patchMSList size5
ERROR: Number of servers are not in sync between Run and Patch Context.
ERROR: FSClone would exit...
CAUSE
runMSList size6 means there are 6 managed servers in Run environment, patchMSList size5 means there are 5 managed servers in Patch environment, which caused error :
ERROR: Number of servers are not in sync between Run and Patch Context.
Use following sql to confirm :
SELECT
extractValue(XMLType(TEXT),'//oa_service_name[@oa_var="s_adminservername"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='patch'
and CTX_TYPE = 'A';
-------------------------
AdminServer
oacore_server1:7201
forms_server1:7401
oafm_server1:7601
forms-c4ws_server1:7801

SELECT
extractValue(XMLType(TEXT),'//oa_service_name[@oa_var="s_adminservername"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='run'
and CTX_TYPE = 'A';
-------------------------
AdminServer
oacore_server1:7202,oacore_server2:7204
forms_server1:7402
oafm_server1:7602
forms-c4ws_server1:7802

Run environment has one more managed server "oacore_server2" than Patch environment.
SOLUTION
Solution A: Remove/delete the managed server "oacore_server2" from the RUN environment filesystem
1.  Managed server oacore_server2 should be shutdown before it can be deleted(in this case, i shut down apache service with adapcctl.sh)
2.  Execute the below command on the Application tier node where the managed server resides:
# perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-managedserver -contextfile=$RUN_BASE/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -managedsrvname=oacore_server2 -servicetype=oacore -logfile=/tmp/remove_manage_server.log
NOTE: check the run filesystem context file for confirmation of no "oacore_server2" information
3. On the node containing the Oracle HTTP Server, run autoconfig:
# sh $INST_TOP/admin/scripts/adautocfg.sh
NOTE: The database server and database listener must remain available during the AutoConfig run, but all other database tier services should be shut down
4.  Execute the following script to remove details of the deleted managed server from the Oracle HTTP Server configuration files:
# perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -ctxfile=$RUN_BASE/inst/apps/UPG_apcappsx21/appl/admin/UPG_apcappsx21.xml -outfile=/tmp/delete_http_config.log
5.  On the node that contains the Oracle HTTP Server, bounce the Oracle HTTP Server:
$ sh $ADMIN_SCRIPTS_HOME/adapcctl.sh stop
$ sh $ADMIN_SCRIPTS_HOME/adapcctl.sh start
6.  Re-issue fs_clone phase

adop fs_clone phase Fails In txkADOPPreparePhaseSynchronize.pl Script When Creating FMW Archive

, , No comments :
12.2.4 version, When attempting to run adop phase=fs_clone the following error occurs:
In adop log file:
[UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: "perl /DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/DATA/app/appldev/TEST1/fs1/inst/apps/TEST1_abc/appl/admin/TEST1_abc.xml -patchcontextfile=/DATA/app/appldev/TEST1/fs2/inst/apps/TEST1_abc/appl/admin/TEST1_abc.xml -promptmsg=hide -console=off -mode=create -sessionid=18 -timestamp=20150325_010624 -outdir=/DATA/app/appldev/TEST1/fs_ne/EBSapps/log/adop/18/fs_clone_20150325_010624/TEST1_abc".
In txkADOPPreparePhaseSynchronize log file
===============================
Inside runADOPCustomSyncUp()...
===============================
Executing SYSTEM command: perl /DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCustomSyncUp.pl -contextfile=/DATA/app/appldev/TEST1/fs1/inst/apps/TEST1_abc/appl/admin/TEST1_abc.xml -logfile=/DATA/app/appldev/TEST1/fs_ne/EBSapps/log/adop/18/fs_clone_20150402_232202/TEST1_abc/TXK_SYNC_create_Thu_Apr_2_23_27_26_2015/txkADOPCustomSyncUp.log -promptmsg=hide
EXIT STATUS: 1
/DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCustomSyncUp.pl did not go through successfully.
LOG FILE: /DATA/app/appldev/TEST1/fs_ne/EBSapps/log/adop/18/fs_clone_20150402_232202/TEST1_abc/TXK_SYNC_create_Thu_Apr_2_23_27_26_2015/txkADOPCustomSyncUp.log.
*******FATAL ERROR*******
PROGRAM : (/DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Fri Apr 3 00:22:46 2015
FUNCTION: main::runADOPCustomSyncUp [ Level 1 ]
ERRORMSG: /DATA/app/appldev/TEST1/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCustomSyncUp.pl did not go through successfully.
In FSCloneStageAppsTier log file:
Attempting to delete: /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc/FMW/t2pjdk
Creating prereq/webtier directory
Attempting to delete: /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc//prereq
Copying /DATA/app/appldev/TEST1/fs1/FMW_Home/webtier/oui to /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc//prereq/webtier
Copying /DATA/app/appldev/TEST1/fs1/FMW_Home/webtier/inventory/Scripts to /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc//prereq/webtier
START: Creating FMW archive.
Running /DATA/app/appldev/TEST1/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc/FMW/t2pjdk -al /DATA/app/appldev/TEST1/fs1/EBSapps/comn/adopclone_abc/FMW/FMW_Home.jar -smw /DATA/app/appldev/TEST1/fs1/FMW_Home -ldl /DATA/app/appldev/TEST1/fs1/inst/apps/TEST1_abc/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true
Script Executed in 200 milliseconds, returning status 1
ERROR: Script failed, exit code 1
CAUSE
The issue is caused during clone staging process t2pjdk directly couldn't deleted because the customer placed the SSL certificate file under /DATA/app/appldev/INNOV/fs2/EBSapps/comn/adopclone_abc/FMW/t2pjdk/jre/lib/security/ and the file couldn't deleted.
During WLS component staging copyConfig utility is failing. The reason is /FMW/t2pjdk directory didn't stage correctly. During t2pjdk directory staging ,if t2pjdk directory already exist the system are trying to delete it. But if for some reason some files within t2pjdk is in already used state then code couldn't able to delete the t2pjdk. The customer enabled SSL on EBS and they copied the custom certificate file to the directory /DATA/app/appldev/INNOV/fs2/EBSapps/comn/adopclone_abc/FMW/t2pjdk/jre/lib/security/ to import it to the cacerts, The file is not seeded file so the t2pjdk directly couldn't deleted as normal.
The workaround is to remove the t2pjdk directory manually. After that, fs_clone complete normally. But after the customer complete a full adop cycle and run fs_clone again, the issue is reproduced.
SOLUTION
To implement the solution, please execute the following steps:
1. Move the custom certificate file from $COMMON_TOP/util/jdk64/jre/lib/security directory and place it outside of $COMMON_TOP/util/jdk64/ directory. (for both RUN and PATCH file system)
2.. Remove /DATA/app/appldev/INNOV/fs1/EBSapps/comn/adopclone_abd/FMW/t2pjdk/directory. (as fs1 is RUN File system)
3. Run fs_clone again.

adop phase=prepare Failed on txkADOPPreparePhaseSynchronize.pl

, , No comments :
The actual error is from FSCloneStageAppsTier_<time stamp>.log
START: Creating WLS config archive.
Script Executed in 200 milliseconds, returning status 1
ERROR: Script failed, exit code 1
CAUSE
The t2pjdk directory is not created completely.
SOLUTION
1. First refer to FSCloneStageAppsTier_<time stamp>.log file to get the t2pjdk directory path.
Following content might be found:
Creating t2pjdk directory
Directory /DATA/app/appldev/ORACLE/fs1/EBSapps/comn/adopclone_test/FMW/t2pjdk already exists.
So the t2pjdk directory path is /DATA/app/appldev/ORACLE/fs1/EBSapps/comn/adopclone_test/FMW/t2pjdk
2. Move the t2pjdk directory to a temp location
For example:
mkdir /tmp/t2pjdk_backup
mv /DATA/app/appldev/ORACLE/fs1/EBSapps/comn/adopclone_test/FMW/t2pjdk /tmp/t2pjdk_backup/FMW/t2pjdk
3. Rerun "adop phase=prepare".
Please be aware :
The same solution can be used for the adpreclone steps.

fsclone fails while running txkADOPPreparePhaseSynchronize.pl

, , No comments :
Error can be seen in adop_<TIME_STAMP>.log
[UNEXPECTED]Error occurred running "perl /u01/oracle/OCBPRD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/OCBPRD/fs1/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -patchcontextfile=/u01/oracle/OCBPRD/fs2/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -promptmsg=hide -console=off -mode=create -sessionid=3 -timestamp=20151223_192751 -outdir=/u01/oracle/OCBPRD/fs_ne/EBSapps/log/adop/3/fs_clone_20151223_192751/OCBPRD_ebsapp01"
[UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u01/oracle/OCBPRD/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oracle/OCBPRD/fs1/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -patchcontextfile=/u01/oracle/OCBPRD/fs2/inst/apps/OCBPRD_ebsapp01/appl/admin/OCBPRD_ebsapp01.xml -promptmsg=hide -console=off -mode=create -sessionid=3 -timestamp=20151223_192751 -outdir=/u01/oracle/OCBPRD/fs_ne/EBSapps/log/adop/3/fs_clone_20151223_192751/OCBPRD_ebsapp01"
CAUSE
The context variable s_invPtrLoc is not getting populated in patch context file. FSCloneApplyAppsTier log prints correct value for s_invPtrLoc but the value is missing in patch context file.
Context file customization is present.
From stage logs(FSCloneStage.log) below can be seen:
Copying /u01/oracle/OCBPRD/fs1/EBSapps/appl/ad/12.0.0/admin/template/custom/adxmlctx.tmp to /u01/oracle/OCBPRD/fs1/EBSapps/comn/adopclone_ebsapp01//context/apps/adxmlctx.tmp
The adxmlctx.tmp is being copied from custom folder.
Custom folder has old version of adxmlct.tmp (version 120.401.12020000.9.1202010.10)
While the actual version is 120.401.12020000.54
During clonecontext the template being used is from the stage which have the older version. Due that reason the patch context file is not getting created properly.
That's the reason the version of context files which can be seen in FND_OAM_CONTEXT_FILES are different between RUN and PATCH filesystems.
RUN fs version : 120.401.12020000.54
PATCH fs version : 120.401.12020000.9.1202010.10
SOLUTION
To solve the issue please do the below:
1.Remove the file $AD_TOP/admin/template/custom/adxmlctx.tmp
2.Run fs_clone with option force=yes

Error java.lang.IllegalStateException: Cipher not initialized R12.2

, , , , No comments :
On Oracle Applications 12.2.4,
when attempting to open the status diagram,
the following error occurs:
[1953]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Exception in getHtml: java.lang.IllegalStateException: Cipher not initialized
[1954]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Complete Exception stack is:
[1957]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:java.lang.IllegalStateException: Cipher not initialized at javax.crypto.Cipher.checkCipherState(Cipher.java:1672)
The issue can be reproduced at will with the following steps:
1. Workflow Administrator Web Applications
2. Administrator Workflow
3. Status Monitor
CAUSE
The Workflow code wasn't able to open the URL for the workflow diagram due to SSL setting was missing.
The debug information in NewWorkflowError.docx shows the following error when trying to connect to the url:
[1505]:STATEMENT:[fnd.wf.WFBaseUtil]:Done openConnection [1505]:STATEMENT:[fnd.wf.WFBaseUtil]:String representation of urlConn: weblogic.net.http.SOAPHttpsURLConnection:https://<serverName>:<portNumber>/forms/frmservlet?appletmode=nonforms&HTMLpageTitle=&HTMLpreApplet=&code=oracle.apps.fnd.wf.Monitor&width=800&height=400&archive=/OA_JAVA/oracle/apps/fnd/jar/wfmon.jar,/OA_JAVA/oracle/apps/fnd/jar/fndewt.jar,/OA_JAVA/oracle/apps/fnd/jar/fndswing.jar,/OA_JAVA/oracle/apps/fnd/jar/fndbalishare.jar,/OA_JAVA/oracle/apps/fnd/jar/fndctx.jar...
1505]:STATEMENT:[fnd.wf.WFBaseUtil]:Done setUseCaches
[1953]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Exception in getHtml: java.lang.IllegalStateException: Cipher not initialized
[1954]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Complete Exception stack is:
[1957]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:java.lang.IllegalStateException: Cipher not initialized at javax.crypto.Cipher.checkCipherState(Cipher.java:1672)
SOLUTION
To implement the solution, please execute the following steps:
1. Bring up the WLS Admin console for the EBS instance ->
http://host:port/console and login as the weblogic user.
2. Click on Lock & Edit.
3. Under the Domain Structure, navigate to EBS Domain -> Environment -> Servers.
4. Click on oacore_server1, and click on the SSL tab.
5. Click on Advanced to expand that section.
6. Look for the 'Use JSSE SSL' parameter and enable this by placing a check in the box next to it.
7. Click on Activate Changes.
8. Restart the managed server.
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server1
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server1
9. Retest the issue.
10. Migrate the solution as appropriate to other environments.

Invalid Objects and Forms Generation Errors During R12.2

, , No comments :
When attempting to apply 12.2.4 upgrade patch(es) (adop phase=apply apply_mode=downtime patches=17919161,21900859 merge=yes)
the following error occurs.
The following Oracle Forms objects did not generate successfully:
pa forms/US PAXURVPS.fmx
The patch has FAILED,
Please check the adpatch logs for more details.
Object APPS.PA_STATUS_COMMITMENTS_V is invalid
SOLUTION
Please recompile APPS.PA_STATUS_COMMITMENTS_V and advise if there are errors, if not then use adadmin to compile form PAXURVPS.fmx
One can use the following to display errors if there are any errors when compiling the view
SQL>alter view APPS.PA_STATUS_COMMITMENTS_V compile;
show error;
OR
SQL> select owner, name, type, text from dba_errors where name = 'PA_STATUS_COMMITMENTS_V' order by 1,2,3,line;

Query to count the open cursors for the sessions.

, , No comments :
Customer faces 5000~8000 open cursor every day and due to this Java Listeners goes down.

SELECT 
  OC.SID, 
  OC.USER_NAME, 
  S.PROGRAM, 
  COUNT(*) COUNTER, 
  OC.SQL_ID, 
  OC.SQL_TEXT 
  FROM 
  V$OPEN_CURSOR OC, 
  V$SESSION S 
  WHERE 
  OC.SQL_TEXT NOT LIKE ''%obj#,%'' 
  AND OC.SQL_TEXT NOT LIKE ''%grantee#,%'' 
  AND OC.SQL_TEXT NOT LIKE ''%privilege#%'' 
  AND OC.SQL_TEXT NOT LIKE ''%/*+ rule */%'' 
  AND OC.SQL_TEXT NOT LIKE ''%col#%'' 
  AND OC.SQL_TEXT NOT LIKE ''%sys.mon_mods$%'' 
  AND OC.SQL_TEXT NOT LIKE ''%obj#=%'' 
  AND OC.SQL_TEXT NOT LIKE ''%update$,%'' 
  AND OC.SID=S.SID 
  AND OC.USER_NAME NOT IN 
 (''SYS'',''DBSNMP'',''SYSTEM'',''SYSMAN'',''RMAN'') 
  AND lower(SQL_TEXT) like ''%enqueue%'' 
  GROUP BY 
  OC.SID, 
  OC.USER_NAME, 
  S.PROGRAM, 
  OC.SQL_ID, 
  OC.SQL_TEXT 
  HAVING COUNT(*)>1 
  ORDER BY 
  COUNT(*) desc, 
  OC.USER_NAME, 
  OC.SID, 
  OC.SQL_TEXT;

How To Access Forms Directly In Oracle Applications R12

, , No comments :
Oracle Applications R12, users are not able to access Forms directly via the following url:
http://hostname.domain:port/forms/frmservlet
The following error would pop out:
APP-FND-01542: This Applications Server is not authorized to access this database.
The goal of this document is to allow users access to the Oracle Applications R12 directly via Forms especially in cases where access to the following url is not available:
http://hostname.domain:port/OA_HTML/AppsLogin
NOTE: Accessing Forms directly is not supported and should only be used for diagnostic and troubleshooting purposes.
SOLUTION
1. Edit $CONTEXT_FILE which is located in $INST_TOP/appl/admin
2. Change
<appserverid_authentication oa_var="s_appserverid_authentication">SECURE</appserverid_authentication>
to
<appserverid_authentication oa_var="s_appserverid_authentication">OFF</appserverid_authentication>
Application Server Security Authentication can take one of the following values {ON, OFF, SECURE}.
OFF - Server security is not checked. Any application server machine can access the database.
ON - Some level of trust is required to access the database. Either the Application Server is registered with the database or the module and version ID are known to be trusted.
SECURE - Full trust is required for access to the database. Only registered Application Server machines and trusted code modules may connect.
3. Run $INST_TOP/admin/scripts/adautocfg.sh to instantiate the changes
4. Users are now able to access Forms directly.
NOTE:
Remember to set the Application Server Security Authentication back to SECURE by settting the value of s_appserverid_authentication to SECURE and running autoconfig.

ORA-04045 errors during recompilation/revalidation of APPS.HXT_HOUR_DEDUCTION_RULES_WHO

, , No comments :
ADOP cleanup reports following error at ad.plsql.ad_zd_sys.drop_covered_object.
[EVENT]     Calling cleanup in STANDARD mode...
   [EVENT]     Log: @ADZDSHOWLOG.sql "2016/01/01 01:01:01"
   [ERROR]     ORA-04045: errors during recompilation/revalidation of APPS.HXT_HOUR_DEDUCTION_RULES_WHO
ORA-04023: Object HXT.HXT_HOUR_DEDUCTION_RULES
   [ERROR]     Failed to execute SQL statement :
                               declare
                              begin
                                 ad_zd_log.message('ad.bin.adop','WARNING','');
                              exception
                                 when others then
                                    raise_application_error(-20001,'Error while calling ad_zd_log.message(ad.bin.adop,WARNING,).' || sqlerrm);
                              end;
   [ERROR]     Error Message :
   [ERROR]     *** Error occurred while calling ad_zd_log. Please check log file for the details. ***
Steps to Reproduce:
The issue can be reproduced at will with the following steps:
 Run ADOP Cleanup phase.
 Error is printed in the log.
SOLUTION
1. Run the command below to compile the trigger.
alter trigger APPS.HXT_HOUR_DEDUCTION_RULES_WHO compile;
2. Run the query below and make sure the status is Enabled.
select owner, trigger_name, status
from dba_triggers
where trigger_name = 'HXT_HOUR_DEDUCTION_RULES_WHO';
  Expected result:
   OWNER      TRIGGER_NAME                   STATUS
   ---------- ------------------------------ ---------
   APPS       HXT_HOUR_DEDUCTION_RULES_WHO   ENABLED
   APPS       HXT_HOUR_DEDUCTION_RULES_WHO   ENABLED
3. Run the query below and make sure the status is valid.
select owner,object_name, object_type,status
from dba_objects
where object_name = 'HXT_HOUR_DEDUCTION_RULES_WHO';
   Expected result:
   OWNER OBJECT_NAME                    OBJECT_TYPE STATUS
   ----- ------------------------------ ----------- -------
   APPS  HXT_HOUR_DEDUCTION_RULES_WHO   TRIGGER     VALID

Unexpected Error On R12.2 Home Page.

, , No comments :
CEST]:1409674186377:-1:-1:hostname.domain:10.32.241.5:-1:-1:-1:-1:GUEST(6):1769119955:Thread[[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads]:10.32.241.5:49173:1409674008272:4:UNEXPECTED:[fnd.framework.OAException]:oracle.adf.mds.exception.MDSRuntimeException: Unable to find component with absolute reference = /oracle/apps/fnd/attributesets/Buttons/Submit, XML Path = {1}. Please verify that the reference is valid and the definition of the component exists either on the File System or in the MDS Repository.  
/oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
[Sep 2, 2014 6:09:46 PM CEST]:1409674186379:-1:-1:hostname.domain:10.32.241.5:-1:-1:-1:-1:GUEST(6):1769119955:Thread[[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads]:10.32.241.5:49173:1409674008272:4:UNEXPECTED:[UNEXPECTED_OA_EXCEPTION: user name = GUEST: responsibility name = null]:oracle.apps.fnd.framework.OAException: oracle.adf.mds.exception.MDSRuntimeException: Unable to find component with absolute reference = /oracle/apps/fnd/attributesets/Buttons/Submit, XML Path = {1}. Please verify that the reference is valid and the definition of the component exists either on the File System or in the MDS Repository.  
/oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
at oracle.apps.fnd.framework.OAException.wrapperException(OAException.java:912)
at oracle.apps.fnd.framework.webui.OAPageErrorHandler.prepareException(OAPageErrorHandler.java:1169)
The oacore server log file (oacore_server1.out) contains the following error stack:
(WebAppServletContext.java:2181)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: org.xml.sax.SAXParseException; systemId: /oracle/apps/fnd/attributesets/Buttons; lineNumber: 1; columnNumber: 40; /oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
at oracle.xml.parser.v2.XMLError.flushErrorHandler(XMLError.java:422)
at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:287)
at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:414)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:355)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:226)
at oracle.cabo.share.xml.ParserAdapter.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:283)
at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:115)
at oracle.adf.mds.adapters.DBAdapter.getElementData(DBAdapter.java:324)
... 81 more
Cause:
org.xml.sax.SAXParseException; systemId: /oracle/apps/fnd/attributesets/Buttons; lineNumber: 1; columnNumber: 40; /oracle/apps/fnd/attributesets/Buttons<Line 1, Column 40>: XML-20108: (Fatal Error) Start of root element expected.
at oracle.xml.parser.v2.XMLError.flushErrorHandler(XMLError.java:422)
at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:287)
at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:414)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:355)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:226)
at oracle.cabo.share.xml.ParserAdapter.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)
If you run:
SQL> exec jdr_utils.printdocument('/oracle/apps/ar/hz/attributesets/HzParties');
the results differ if VPD policies are enabled/disabled, i.e. with VPD policies enabled this returns no rows.
CAUSE
The SQL profiles are setting optimizer_features_enable to 10.1.0.5 and this setting is not going to work with EBR , which is an 11g feature.
This is explained in:
Bug 19830345 - QUERY RETURNS NO ROWS AGAINST SOME TABLES PROTECTED BY VPD
SOLUTION
To implement the solution, please execute the following steps:
1) run this query to discover all the profiles that set the OFE parameter to 10.1.0.5 :
SELECT o.name, d.comp_data
FROM SQLOBJ$ o, SQLOBJ$DATA d
WHERE o.signature = d.signature
AND o.category = d.category
and COMP_DATA like '%10.1%'
ORDER BY o.name;
2) for each o.name run this :
exec DBMS_SQLTUNE.DROP_SQL_PROFILE ( '')
3) Re-test the issue.

R12.2 ADOP Patch Worker Fails On Command drop view APPLSYS.ERROR_CODES_MAP# With Error ORA-00942 table or view does not exist

, , No comments :
E-Business Suite 12.2 Applications DBA, Online Patching issues
While applying a patch (For example Patch 23510855 - ISG R12.2 CONSOLIDATED PATCH), a worker fails with the below error:
drop_error_codes_map.sql &un_fnd'
Connected.
PL/SQL procedure successfully completed.
drop view APPLSYS.ERROR_CODES_MAP#
ERROR at line 1:
ORA-00942: table or view does not exist
CAUSE
The view APPLSYS.ERROR_CODES_MAP has already been dropped.
A number of different patches could have been applied which would drop the view, including the following:
17909318 R12.ATG_PF.C.delta.4 R12.ATG_PF.C.delta.4 ATG_PF
17919161 12.2.4 ORACLE E-BUSINESS SUITE 12.2.4 RELEASE UPDATE PACK AU
17765665 R12.OWF.C ISG AGENT CONSOLIDATED ONE-OFF PATCH FOR 12.2.3 OWF
The following select statements can be used to check for the above patches:
SELECT ad_patch.is_patch_applied('R12',-1,17909318) FROM dual;
SELECT ad_patch.is_patch_applied('R12',-1,17919161) FROM dual;
SELECT ad_patch.is_patch_applied('R12',-1,17765665) FROM dual;
SOLUTION
To resolve the issue test the following steps in a development instance and then migrate accordingly:
1. Verify view does not exist:
SQL> SELECT OWNER, VIEW_NAME FROM ALL_VIEWS WHERE upper(VIEW_NAME) LIKE 'ERROR_CODES_MAP%';
2. use adctrl hidden option 8 to skip the job.
3. resume the adop session using options:
restart=yes abandon=no
4. Confirm the patch completes successfully.

R12.2 adop phase=finalize Fails As ADOP Cannot Detect Any Application Tier Nodes In FND_NODES

, , No comments :
On : 12.2.3 version, Upgrade related issues
When attempting to perform "adop phase=finalize"
the following error occurs.
ERROR
-----------------------
[UNEXPECTED]adop cannot detect any application tier nodes in FND_NODES table. Ensure ICM is running and run autoconfig on all nodes
[UNEXPECTED]Error while checking if this is a multi node instance
STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Login to server
2. adop phase=finalize
CAUSE
The issue occurs due to a null value for the concurrent/admin node status column in table FND_NODES.
More details are available via the following unpublished defect:
Bug 18533116 - AD:ADOP IN HOTPATCH FAILED FOR R12.TXK.C.DELTA.5
"In AD.Delta.4+one off patches we have a dependency on FND_NOTES.STATUS column
The column will be automatically set to 'Y' when ICM is up.
When the ICM is down the query will not detect any nodes with status 'Y',  and adop proceeds with only the current/admin node (inserts rows into AD_ADOP_SESSIONS for current/admin node only."
SOLUTION
    A. Set the "run" Application Tier environment
    B. Start ICM
    C. Run "adop phase=finalize" again.

Adop Prepare With Error ORA-20001: Error: While Calling Ad_zd.cleanup(normal).ORA-20007

, , No comments :
On EBS 12.2 when applying a patch using adop.sh in the prepare phase it fails with the error:
ERROR at line 1:
ORA-20001: Error: while calling ad_zd.cleanup(normal).ORA-20007: Ancestors of
Current RUN Edition are not retired
ORA-06512: at line 7
The issue can be reproduced at will with the following steps:
1. Apply patch using adop.sh.
CAUSE
The following data from customer environment shows V_20120409_0455 not retired. The other edition, ORA$BASE is correctly retired.
select apps.ad_zd.get_edition('OLD') "OLD",
  apps.ad_zd.get_edition('RUN') "RUN",
  apps.ad_zd.get_edition('PATCH') "PATCH"
from dual;
EDITION_NAME PARENT_EDITION_NAME USA
------------------------------ ------------------------------ ---
ORA$BASE YES
V_20120409_0455 ORA$BASE YES
V_20120521_0936 V_20120409_0455 YES
column CURRENT_EDITION format a30
column EDITION_TYPE format a15
select apps.ad_zd.get_edition "CURRENT_EDITION",
  apps.ad_zd.get_edition_type "EDITION_TYPE"
from dual;
OLD RUN PATCH
-------------------- -------------------- --------------------
V_20120409_0455 V_20120521_0936
Previous patch application did not get retired.
SOLUTION
1. Please run the following command to retire the edition:
SQL> exec sys.ad_zd_sys.retire_old_editions;
After this the following sql should not return any row for V_20120409_0455:
 select *
 from dba_tab_privs
 where table_name in (select edition_name from dba_editions);
2. Once the V_20120409_0455 is retired successfully, one will be able to do the cleanup and proceed further with the adop cycles.

How to Disable Login Assistance Link In R12 ?

, No comments :
SOLUTION
The system profile 'Local Login Mask' has been obsoleted in R12 and cannot be used to disable Login Assistance link.  Please follow the steps below to disable this link:
1. Logon with Functional Administrator responsibility.
2. Click on 'Personalization' Tab.
3. Enter /oracle/apps/fnd/sso/login/webui in document path and click on 'Go' button.
4. Click on 'Pencil' icon under 'Personalize Page' for oracle/apps/fnd/sso/login/webui/MainLoginPG.
5. Make sure that 'Include' for Site is checked and click on 'apply'
6. Look for (LoginRN.MainLoginRN.ForgotPasswordUrlRowLayoutRN1) and click on 'Pencil' icon in
'Personalize' column.
7. Change 'rendered' property value to 'false' at Site level from LOV and click on Apply.
8. This will disable the link and users logging in after the change will not see the link.

Dataguard Monitoring Scripts

, , , No comments :
SET PAGESIZE 124
COL DB_NAME FORMAT A8
COL HOSTNAME FORMAT A12
COL LOG_ARCHIVED FORMAT 999999
COL LOG_APPLIED FORMAT 999999
COL LOG_GAP FORMAT 9999
COL APPLIED_TIME FORMAT A12
SELECT DB_NAME, HOSTNAME, LOG_ARCHIVED, LOG_APPLIED,APPLIED_TIME,LOG_ARCHIVED-LOG_APPLIED LOG_GAPFROM(SELECT NAME DB_NAMEFROM V$DATABASE),(SELECT UPPER(SUBSTR(HOST_NAME,1,(DECODE(INSTR(HOST_NAME,’.’),0,LENGTH(HOST_NAME),(INSTR(HOST_NAME,’.’)-1))))) HOSTNAMEFROM V$INSTANCE),(SELECT MAX(SEQUENCE#) LOG_ARCHIVEDFROM V$ARCHIVED_LOG WHERE DEST_ID=1 AND ARCHIVED=’YES’),(SELECT MAX(SEQUENCE#) LOG_APPLIEDFROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND APPLIED=’YES’),(SELECT TO_CHAR(MAX(COMPLETION_TIME),’DD-MON/HH24:MI’) APPLIED_TIMEFROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND APPLIED=’YES’);
================================================================================
SQL> select status,instance_name,database_role fromv$instance,v$database;
STATUS       INSTANCE_NAME    DATABASE_ROLE
——————————————–
OPEN         prim             PRIMARY
SQL> select max(sequence#) from v$archived_log;
MAX(SEQUENCE#)
————–
40
Check the status of the physical standby database and the latest sequence applied on the physcial standby database.
SQL> select status,instance_name,database_role fromv$database,v$instance;
STATUS   INSTANCE_NAME DATABASE_ROLE
——————————————
MOUNTED  stnd          PHYSICAL STANDBY
SQL> select max(sequence#) from v$archived_log where applied=’YES’;
MAX(SEQUENCE#)
————–
40
Check if the Managed Recovery Process (MRP) is active on the physcial standby database.
SQL> select process,status,sequence# from v$managed_standby;
PROCESS   STATUS        SEQUENCE#
——————————-
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
RFS       IDLE          41
RFS       IDLE          0
RFS       IDLE          0
RFS       IDLE          0
MRP0      WAIT_FOR_LOG  41
9rows selected.
Here, MRP is active. The PROCESS Column above shows that MRP is active and is waiting for the log sequence 41.
Step 4:
Cancel the MRP on the physical standby database and open the standby database. The standby database would be opened in the READ-ONLY Mode.
SQL> alter database recover managed standby database cancel;
Database altered.
SQL> alter database open;
Database altered.
SQL> select status,instance_name,database_role,open_mode fromv$database,v$instance;
STATUS INSTANCE_NAME  DATABASE_ROLE    OPEN_MODE
—————————————————
OPEN   stnd           PHYSICAL STANDBY READ ONLY
Step 6:
Now start the MRP on the physical standby database.
SQL> alter database recover managed standby database disconnectfrom session;
Database altered.
SQL> select process,status,sequence# from v$managed_standby;
PROCESS   STATUS        SEQUENCE#
——————————-
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
ARCH      CONNECTED     0
RFS       IDLE          41
RFS       IDLE          0
RFS       IDLE          0
RFS       IDLE          0
MRP0      WAIT_FOR_LOG  41
9rows selected.
Here, you can see that the MRP is active and is waiting for the log sequence 41 and also the physical standby database is opened in READ-ONLY mode which would allow users to use the physical standby database for fetching reports.
Managing dataguard – monitoring scripts
  LOGS             TIME—————- ——————
Last applied   :  16-JUL-09:09:24:16
Last received :  16-JUL-09:09:28:36
select ‘Last applied  : ‘ Logs, to_char(next_time,’DD-MON-YY:HH24:MI:SS’) Timefrom v$archived_log
where sequence# = (select max(sequence#) from v$archived_log where applied=’YES’)
union
select ‘Last received : ‘ Logs, to_char(next_time,’DD-MON-YY:HH24:MI:SS’) Time
from v$archived_log
where sequence# = (select max(sequence#) from v$archived_log);
NAME                   VALUE                  UNIT———————- ———————- ———————————–
apply finish time      +00 00:02:07.2         day(2) to second(1) interval
apply lag              +00 00:01:59           day(2) to second(0) interval
estimated startup time 16                     second
standby has been open  N
transport lag          +00 00:00:00           day(2) to second(0) interval
Time Computed: 16-JUL-2009 09:33:16
selectNAME Name,
VALUE Value,
UNIT Unit
from v$dataguard_stats
union
select null,null,’ ‘ from dual
union
select null,null,’Time Computed: ‘||MIN(TIME_COMPUTED)
from v$dataguard_stats;
  Redo onsite
——————–
16-JUL-2009 09:42:44
select to_char(max(last_time),’DD-MON-YYYY HH24:MI:SS’) “Redo onsite”from v$standby_log

INVALID Oracle Database Packages and Type in dba_registry

, , No comments :
The timestamp query object.

SQL> select do.obj# d_obj,do.name d_name, do.type# d_type,
2 po.obj# p_obj,po.name p_name,
3 to_char(p_timestamp,'DD-MON-YYYY HH24:MI:SS') "P_Timestamp",
4 to_char(po.stime ,'DD-MON-YYYY HH24:MI:SS') "STIME",
5 decode(sign(po.stime-p_timestamp),0,'SAME','*DIFFER*') X
6 from sys.obj$ do, sys.dependency$ d, sys.obj$ po
7 where P_OBJ#=po.obj#(+)
8 and D_OBJ#=do.obj#
9 and do.status=1 /*dependent is valid*/
10 and po.status=1 /*parent is valid*/
11 and po.stime!=p_timestamp /*parent timestamp not match*/
12 order by 2,1;

D_OBJ D_NAME D_TYPE P_OBJ P_NAME
---------- -------------------- ---------- ---------- --------------------
P_Timestamp STIME X

Solution :
————————-
SQL>sqlplus / as sysdba
SQL>drop table plan_table;
SQL>@?/rdbms/admin/utlxplan
SQL>@?/rdbms/admin/prvtspao.plb
SQL>@?/rdbms/admin/utlrp.sql
This will resolve your issue.
or
SQL> SELECT r.comp_name, r.version, r.status FROM dba_registry r;
COMP_NAME VERSION STATUS
—————————————- —————————— ———————————
Oracle interMedia 10.2.0.4.0 VALID
Oracle Enterprise Manager 10.2.0.4.0 VALID
Oracle Workspace Manager 10.2.0.4.3 VALID
Oracle Database Catalog Views 10.2.0.4.0 VALID
Oracle Database Packages and Types 10.2.0.4.0 INVALID
JServer JAVA Virtual Machine 10.2.0.4.0 VALID
Oracle Database Java Packages 10.2.0.4.0 VALID
Oracle XDK 10.2.0.4.0 VALID
8 rows selected
repair and validate “Oracle Database Packages and Types”.
1. log as SYSDBA
$ sqlplus ” / as sysdba”
2. shutdown running database and startup database in upgrade mode
SQL> shutdown immediate;
SQL>startup upgrade;
For startup migrate;
3. run this two scripts
SQL> spool /tmp/cat_repair_db.txt
SQL> @?/rdbms/admin/catalog.sql
SQL> @?/rdbms/admin/catproc.sql
catalog.sql will recreate Oracle database Catalog Views component and catproc.sql will recreate Oracle database Packages and Types component
4. Recompile all invalid objects
SQL> @?/rdbms/admin/utlrp.sql
SQL> spool off
5. Shutdown database and startup in open mode
SQL> shutdown immediate;
SQL> startup;
SQL> SELECT r.comp_name, r.version, r.status FROM dba_registry r;
COMP_NAME VERSION STATUS
—————————————- —————————— ———————————
Oracle interMedia 10.2.0.4.0 VALID
Oracle Enterprise Manager 10.2.0.4.0 VALID
Oracle Workspace Manager 10.2.0.4.3 VALID
Oracle Database Catalog Views 10.2.0.4.0 VALID
Oracle Database Packages and Types 10.2.0.4.0 VALID
JServer JAVA Virtual Machine 10.2.0.4.0 VALID
Oracle Database Java Packages 10.2.0.4.0 VALID
Oracle XDK 10.2.0.4.0 VALID
8 rows selected

adop phase cutover is failing on starting Apache OHS timing out

, , No comments :
When attempting to run Cutover phase completed with errors/warnings.
the following error occurs.
*******FATAL ERROR*******
PROGRAM : (/localapps/app/oracle/CONV/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl)
TIME : Sat Nov 29 18:24:59 2014
FUNCTION: main::forceStartupServices [ Level 1 ]
ERRORMSG: 1
  [ERROR] Error occurred while executing
  [STATEMENT] Cutover phase completed with errors/warnings. Please check logfiles
 In the log file the following information can be seen:
--------
14/11/30 00:01:57 Start process
--------
/localapps/app/oracle/CONV/fs2/FMW_Home/webtier/ohs/bin/apachectl startssl: execing httpd
httpd.worker: Syntax error on line 1039 of /localapps/app/oracle/CONV/fs2/FMW_Home/webtier/instances/EBS_web_CONV_OHS1/config/OHS/EBS_web_CONV/httpd.conf: Syntax error on line 115 of /localapps/app/oracle/CONV/fs2/FMW_Home/webtier/instances/EBS_web_CONV_OHS1/config/OHS/EBS_web_CONV/ssl.conf: Could not open configuration file /localapps/app/oracle/CONV/fs2/FMW_Home/webtier/instances/EBS_web_CONV_OHS1/config/OHS/EBS_web_CONV/oracle_apache_ssl.conf: No such file or directory
CAUSE
Cutover completed and issue occurred only while starting services.  Apache OHS service is timing out on starting up.
--------
14/11/30 00:01:57 Start process
--------
/localapps/app/oracle/CONV/fs2/FMW_Home/webtier/ohs/bin/apachectl startssl: execing httpd
httpd.worker: Syntax error on line 1039 of /localapps/app/oracle/CONV/fs2/FMW_Home/webtier/instances/EBS_web_CONV_OHS1/config/OHS/EBS_web_CONV/httpd.conf: Syntax error on line 115 of /localapps/app/oracle/CONV/fs2/FMW_Home/webtier/instances/EBS_web_CONV_OHS1/config/OHS/EBS_web_CONV/ssl.conf: Could not open configuration file /localapps/app/oracle/CONV/fs2/FMW_Home/webtier/instances/EBS_web_CONV_OHS1/config/OHS/EBS_web_CONV/oracle_apache_ssl.conf: No such file or directory
SOLUTION
To solve the issue please apply the below steps :
On DB Tier:
------------
1. Login to the DB Tier
2. Source the environment
3. Confirm the database and listener is up and running
On Apps Tier:
--------------
1. Source RUN file system environment
2. Stop all middle tier services using $ADMIN_SCRIPTS_HOME/adstpall.sh
3. Ensure that no process is running from RUN file system
4. Execute below command:
  $FND_TOP/bin/txkrun.pl -script=ChkEBSDependecies -server=ALL_SERVERS
5. Recreate the domain by following below steps:
cd $FND_TOP/patch/115/bin
  perl txkEBSDomainConfig.pl
  Below prompts are shown by the script:
  --------------------------------------
  Enter the full path of Applications Context File [DEFAULT - ]: Complete path of the RUN context file
  Enter the server start mode for the domain [DEFAULT - prod]: prod
  Enter the APPS schema password :
  Enter weblogic admin server password :
6. Start all middle tier services using $ADMIN_SCRIPTS_HOME/adstrtal.sh
7. continue with cleanup and fs_clone steps

RC-50208: Exception in method TimedProcessTimer.run When Running txkEBSDomainConfig.pl Script On R12.2

, , No comments :
On Oracle Applications 12.2.3 version, IAS for Applications Technology,
when attempting to re-deploy the WLS domains using txkEBSDomainConfig.pl,
the following error occurs:
StackTrace:
java.lang.IllegalArgumentException: timeout value is negative
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:905)
at oracle.apps.ad.util.TimedProcess$TimedProcessTimer.run(TimedProcess.java:703)
at java.lang.Thread.run(Thread.java:781)
RC-50208: Exception in method TimedProcessTimer.run
Raised by oracle.apps.ad.util.TimedProcess$TimedProcessTimer
The Admin Server is starting up in the background..
ERROR: Unable to startup the Admin Server within the time specified in the context variable s_adminservertimeout..
ERROR: Unable to startup the Admin Server. Cannot proceed with the deployment of EBS
Creation and deployment of E-Business Suite domain completed..
The issue can be reproduced at will with the following steps:
1. Run the script txkEBSDomainConfig.pl to re-deploy the WLS domains.
2. Review the logs.
CAUSE
The issue is caused by the context parameter value s_adminservertimeout which is set incorrectly to -1.
Due to which the below error is seen:
StackTrace:
java.lang.IllegalArgumentException: timeout value is negative
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:905)
at oracle.apps.ad.util.TimedProcess$TimedProcessTimer.run(TimedProcess.java:703)
at java.lang.Thread.run(Thread.java:781)
RC-50208: Exception in method TimedProcessTimer.run
Raised by oracle.apps.ad.util.TimedProcess$TimedProcessTimer
This is explained in the following bug:
Bug 18485007 - FS_CLONE FAIL WITH RC-50208: EXCEPTION IN METHOD TIMEDPROCESSPROCESS.RUN
SOLUTION
Please perform the below steps:
1. Shutdown the EBS Services.
2. In both the RUN and PATCH file system context file, modify s_adminservertimeout context variable value to 4000.
From
=====
-1
To
==
4000
2. Run AutoConfig from both the RUN and PATCH file system.
Ignore the Autoconfig failures on the patch file system.
NOTE: AutoConfig should be run in full mode (Without any mode specification) , otherwise Context File will not be uploaded to database.
3. Rerun the failed script in this case it is /txkEBSDomainConfig.pl and check whether the script completes successfully.
4. Migrate the solution to appropriate environments.

R12.2 Patch Seems to be Stuck And Not Progressing

, , No comments :
While applying some R12.2 patch it seems to be stuck for long time and don't progress.
ERROR:
No information is logged in the log files.
CHANGES
 In reviewing the patch/worker logs it appears to be running compile objects such as ADZDUTLRECMP.sql.
CAUSE
Compiling STUB invalid objects
STUB are objects related to the editioning capabilities of the database.
SOLUTION
To implement the solution, please execute the following steps :
In R12.2, different than other releases, the stubs objects are not displayed in the dba_objects, so querying this table will not show all the invalid objects
The following query can be used to display the invalid objects:

select * from
  (
    select
        eusr.user_name owner
      , count(decode(obj.type#,88,NULL,decode(obj.status,1,NULL,1))) Actual
      , count(decode(obj.type#,88,decode(obj.status,1,NULL,1),NULL)) Stub
      , count(decode(obj.type#,88,decode(obj.status,1,NULL,1),decode(obj.status,1,NULL,1))) Total
    from
        sys.obj$ obj
      , sys.obj$ bobj
      , (
            select
                xusr.user#
              , xusr.ext_username user_name
              , ed.name edition_name
            from
                (select * from sys.user$ where type# = 2) xusr
              , (select * from sys.obj$ where owner# = 0 and type# = 57) ed
            where xusr.spare2 = ed.obj#
            union
            select
                busr.user#
              , busr.name user_name
              , ed.name edition_name
            from
                (select * from sys.user$ where type# = 1 or user# = 1) busr
              , (select * from sys.obj$ where owner# = 0 and type# = 57) ed
            where ed.name = 'ORA$BASE'
        ) eusr
    where obj.owner# = eusr.user#
      and bobj.obj#(+) = obj.dataobj#
      and obj.type# not in (0, 10)
      and ( obj.type# <> 88 or (obj.type# = 88 and bobj.type# <> 10) )
      and obj.remoteowner is null
    group by eusr.user_name
  ) x
where total > 0
order by 1
/
---
The patch will proceed only when the number of invalids displayed is close to zero.
The following is a sample output from the query:
OWNER    ACTUAL       STUB      TOTAL
---------- ---------- ----------
APPS         2      19296      19298
Note that there are 19296 invalid stub objects to be compiled yet.

ADOP Fails Because Remote Execution Failed

, , No comments :
On : 12.2.4 version, Online Patching - OP
When attempting to run adop phase=prepare or  fs_clone or abort in a multi-node deployment,
the following error occurs.
ERROR
-----------------------
  Executing "adop" action in all nodes
  [ERROR] Node: "erpdb" Status: "failed"
  [STATEMENT] Node: "erpp1" Status: "success"
  [ERROR] Node: "erpp2" Status: "failed"
  [ERROR] Remote execution failed on Node: "erpp2".
  [ERROR] Remote execution failed on Node: "erpdb".
  [EVENT] [START 2015/08/30 17:45:19] Checking if adop can continue with available nodes
  [EVENT] Calling: /engn101/erpprod/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl
  [STATEMENT] Output directory: /engn101/erpprod/fs_ne/EBSapps/log/adop/16/prepare_20150830_164022/ERPPROD_erpp1
 [UNEXPECTED]Error occurred while executing "perl /engn101/erpprod/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -contextfile=/engn101/erpprod/fs2/inst/apps/ERPPROD_erpp1/appl/admin/ERPPROD_erpp1.xml -patchcontextfile=/engn101/erpprod/fs1/inst/apps/ERPPROD_erpp1/appl/admin/ERPPROD_erpp1.xml -phase=prepare -promptmsg=hide -nodelist=erpp1 -console=off -sessionid=16 -timestamp=20150830_164022 -outdir=/engn101/erpprod/fs_ne/EBSapps/log/adop/16/prepare_20150830_164022/ERPPROD_erpp1"
  [UNEXPECTED]Error occurred while Checking if adop can continue with available nodes using command: "perl /engn101/erpprod/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl -contextfile=/engn101/erpprod/fs2/inst/apps/ERPPROD_erpp1/appl/admin/ERPPROD_erpp1.xml -patchcontextfile=/engn101/erpprod/fs1/inst/apps/ERPPROD_erpp1/appl/admin/ERPPROD_erpp1.xml -phase=prepare -promptmsg=hide -nodelist=erpp1 -console=off -sessionid=16 -timestamp=20150830_164022 -outdir=/engn101/erpprod/fs_ne/EBSapps/log/adop/16/prepare_20150830_164022/ERPPROD_erpp1".
  Previous tasks have failed or are incomplete on nodes: ",erpp2,erpdb"
  [UNEXPECTED]Unable to continue with other completed nodes: "erpp1".
  [UNEXPECTED]Error: While executing "adop" on all other nodes.
  [UNEXPECTED]Execution failed when performing remote action.
remote_execution_result.xml:
     [END   2015/08/30 17:40:51] adzdoptl.pl runadop phase=prepare - Completed Successfully
     Log file: /engn101/erpprod/fs_ne/EBSapps/log/adop/16/adop_erpdb_20150830_164837.log
     adop exiting with status = 0 (Success)
     Worker count determination...stty: tcgetattr: Not a typewriter :failed</ResultString>
     <Status>failed</Status>
     <TraceFile/>
  </Node>
  <Node>
     <Name>erpp1</Name>
     <ProcessName>erpp1_patchpool-1-thread-2_erpp1_patch</ProcessName>
     <ResultString>
Please wait. Validating credentials...
Execute SYSTEM command : df -k /engn101/erpprod/fs1
Validation successful. All expected nodes are listed in ADOP_VALID_NODES table.
Worker count determination...
stty: tcgetattr: Not a typewriter
.end err out.
</ResultString>
     <Status>success</Status>
     <TraceFile>NoTraceFile</TraceFile>
  </Node>
  <Node>
     <Name>erpp2</Name>
     <ProcessName>nx1aerpp2_patchpool-1-thread-3_erpp2_patch</ProcessName>
     <ResultString> Please wait. Validating credentials...
     Execute SYSTEM command : df -k /engn101/erpprod/fs1 Validation successful.
     All expected nodes are listed in ADOP_VALID_NODES table.[EVENT]
......
     adop exiting with status = 0 (Success)
     Worker count determination...stty: tcgetattr: Not a typewriter :failed
     </ResultString>
     <Status>failed</Status>
     <TraceFile/>
  </Node>
</NODES>
STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Login to primary node as applmgr
2. adop phase=prepare
CAUSE
In Multi node, non-shared environment, ADOP returns slave node Remote execution status as failed due to "stty: tcgetattr: Not a typewriter :failed"
though the task completed successfully.
It's because of the command stty and due to this its retuning failed status.
SSH in remote nodes were not enabled using txkRunSSHSetup.pl.
After enabling SSH via txkRunSSHSetup.pl, the issue is  resolved.
This is explained in the following bug:
Bug 18138059 - QREP1224:B1:REMOTE EXECUTION ON SLAVE NODE RETURN FAILED STATE IN MN NON SHARED
SOLUTION
Re-enable ssh from the primary node to all secondary nodes using txkRunSSHSetup.pl
For example, a basic command to enable ssh would be:
$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3
To verify ssh operation:
$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl verifyssh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3 \
-invalidnodefile=<filename to report ssh verification failures>

Restarting Adop From A Failed Session R12.2

, , No comments :
When adop has failed what are the steps to restart the same patch with fewer number of workers:
For example to restart using fewer workers.
Is this correct?
$ adop phase=abort
$ adop phase=prepare
$ adop phase=apply patches=12345678 patchtop=/patch_directory
The above steps are not correct.
SOLUTION
This correct process can be found in the 12.2.x maintenance manual:
http://docs.oracle.com/cd/E26401_01/doc.122/e22954.pdf
Which documents the following commands:
 'adop phase=abort'
 'adop phase=cleanup cleanup_mode=full'
 'adop phase=fs_clone'
 'adop restart=yes'
restart the patching cycle by setting the environment again and running
 'adop phase=prepare'
Number of workers can be specified on the command line with the command:  adop phase=apply workers=8 patches=patch number patchtop=<your patch directory>
You can use  'adop -status'  to check the current status of the adop patching cycle.

Fs_clone Fails With txkCloneAcquirePort.pl does not exist

, , No comments :
[UNEXPECTED]$INST_TOP/admin/install/txkCloneAcquirePort.pl does not exist. Cannot continue.
CAUSE
The template file (txkCloneAcquirePort_pl.tmp) is provided from CONSOLIDATED FIXES FOR R12.TXK.C.DELTA.4 Bug 18497540. This file is copied to the file system during the patch. When autoconfig is run this creates txkCloneAcquirePort.pl In this case this didn't happen so txkCloneAcquirePort.pl is missing.
As for 18497540 adpatch lgi file, this error noticed:
Will not apply txkCloneAcquirePort_pl.tmp: Files have same version.
 Patch  : $AD_TOP/admin/template/txkCloneAcquirePort_pl.tmp, v120.0.12020000.2
 On-Site: $AD_TOP/admin/template/txkCloneAcquirePort_pl.tmp, v120.0.12020000.2
As the versions of txkCloneAcquirePort_pl.tmp are the same, txkCloneAcquirePort.pl never gets created.
SOLUTION
1. Run autoconfig from the RUN File System (FS).
2. Make sure the txkCloneAcquirePort.pl file is created after autoconfig is run.
3. Run fs_clone again.

Timeout Error Encountered for pasteBinary.sh While Running "adop phase=fs_clone"

, , No comments :
SYMPTOMS
On Oracle E-Business Suite (EBS) 12.2.2,
when attempting to execute "adop phase=fs_clone",
the following error occurs:
Enabling EBS_LOGON trigger
Error while executing [filesystem]/fs2/inst/apps/[context_name]/appltmp/alter_ebs_logon_trigger.sql
Failed to enable EBS_LOGON trigger
*******FATAL ERROR*******
PROGRAM : ([filesystem]/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Mon Nov 11 15:47:51 2013
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: [filesystem]/fs1/EBSapps/comn/clone/bin/adclone.pl did not go through successfully.
         [UNEXPECTED]Error occurred while executing <perl [filesystem]/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=[filesystem]/fs1/inst/apps/[context_name]/appl/admin/[context_name].xml -patchcontextfile=[filesystem]/fs2/inst/apps/[context_name]/appl/admin/[context_name].xml -promptmsg=hide -console=off -mode=create -sessionid=5 -timestamp=[timestamp] -outdir=[filesystem]/fs_ne/EBSapps/log/adop/5/fs_clone_[timestamp]/[context_name]>
         [UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: <perl [filesystem]/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=[filesystem]/fs1/inst/apps/[context_name]/appl/admin/[context_name].xml -patchcontextfile=[filesystem]/fs2/inst/apps/[context_name]/appl/admin/[context_name].xml -promptmsg=hide -console=off -mode=create -sessionid=5 -timestamp=[timestamp] -outdir=[filesystem]/fs_ne/EBSapps/log/adop/5/fs_clone_[timestamp]/[context_name]>.
         [UNEXPECTED]fs_clone phase completed with errors/warnings. Please check logfiles
         Log file: [filesystem]/fs_ne/EBSapps/log/adop/5/adop_[timestamp].log
adop exiting with status = 2 (Fail)
On further investigation, it is seen in the FSCloneApplyAppsTier_[timestamp].log file:
# Calling ApplyAppsTier...
# Creating the FMW Home from ApplyAppsTier
Creating FMW Home.
Running [filesystem]/fs1/EBSapps/comn/clone/FMW/pasteBinary.sh -javaHome [filesystem]/fs1/EBSapps/comn/clone/FMW/t2pjdk -al [filesystem]/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -tl [filesystem]/fs2/FMW_Home -invPtrLoc /etc/oraInst.loc -ldl [filesystem]/fs1/EBSapps/comn/clone/FMW/logs -silent true -executeSysPrereqs false
Script Executed in 7200015 milliseconds, returning status -1
Script timed out.
The issue can be reproduced at will with the following steps:
1. Execute "adop phase=fs_clone".
The issue has the following business impact:
Due to this issue, users cannot EBS 12.2.2 upgrade.
CAUSE
The issue is caused by the following setup:
pasteBinary is run within a TimedProcess:
TimedProcess pasteBinaryProc = new TimedProcess(l_pasteBinarycmd, 7200000);
SOLUTION
To implement the solution, please perform the following:
1. Open a new terminal.
2. Execute:
      export TIMEDPROCESS_TIMEOUT=-1
3. Execute:
      adop phase=fs_clone
NOTE: The script may take some time depending on the server resources (CPU and RAM).

Error During Script txkADOPPreparePhaseSynchronize.pl of adop phase=fs_clone

, , No comments :
SYMPTOMS
When running adop phase=fs_clone you see the following [UNEXPECTED] error during txkADOPPreparePhaseSynchronize.pl in the main adop log file:
[UNEXPECTED]Error occurred while executing <perl $APPL_TOP/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/d01/R122UPGR/fs1/inst/apps/<SID>_<HOST>/appl/admin/<context file> -patchcontextfile=/d01/R122UPGR/fs2/inst/apps/<SID>_<HOST>/appl/admin/<context file> -promptmsg=hide -console=off -mode=create -sessionid=5 -timestamp=20131204_174004 -outdir=/d01/R122UPGR/fs_ne/EBSapps/log/adop/5/fs_clone_20131204_174004/<SID>_<HOST>
If one then checks the FSCloneStageAppsTier_<datestamp>.log, the following error will be seen:
/d01/R122UPGR/fs1/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /d01/R122UPGR/fs1/EBSapps/comn/clone/FMW/t2pjdk -al /d01/R122UPGR/fs1/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /d01/R122UPGR/fs1/FMW_Home -ldl /d01/R122UPGR/fs1/EBSapps/comn/clone/FMW/logs -invPtrLoc /etc/oraInst.loc -silent true
Script Executed in 28420 milliseconds, returning status 255
Script failed, exit code 255
If one then checks the latest log file in:  /d01/R122UPGR/fs1/EBSapps/comn/clone/FMW/logs the following error is seen:
SEVERE : Dec 4, 2013 6:17:41 PM - ERROR - CLONE-20275   Insufficient space to create /tmp/CLONINGCLIENT3005869153400434600
CHANGES
 Running adop phase=fs_clone following a patch.
CAUSE
 Insufficient disk space in the /tmp directory.
SOLUTION
 Create space in /tmp or assign more disk storage to /tmp.

R12.2 Collecting Online Patching and fs_clone Log Files.

, , No comments :
To debug Online Patching issues which utilize the adop (AD Online Patching) tool we need to collect adop log files located in the <INSTALL BASE>/fs_ne/EBSapps/log/adop directory
e.g. /u01/PROD/fs_ne/EBSapps/log/adop
Each cycle of adop creates a subdirectory corresponding to the patch session ID, e.g.
/u01/PROD/fs_ne/EBSapps/log/adop/1
/u01/PROD/fs_ne/EBSapps/log/adop/2
etc..etc
When running adop the on-screen terminal output will mention which adop session ID  is in use.
e.g. /u01/PROD/fs_ne/EBSapps/log/adop/9/apply_20121011_024437
Online Patching Log Analyzer Utility (this new feature is delivered by AD and TXK C Delta 4 patches)
This utility analyzes adop log directories for errors and warnings, and displays messages to help the user quickly identify any problems that may have occurred during an adop run. It thereby offers an alternative to reviewing log files manually.
The Log Analyzer utility can be run without options:
    To scan all log directories of the latest adop session for errors:
    $ adopscanlog
The utility can also be run with various options. Examples include:
    To scan log directories relating to the latest run of adop in the latest session:
    $ adopscanlog -latest=yes
    To scan log directories relating to the latest run of the specified phase, in the latest session:
    $ adopscanlog -latest=yes -phase=<phase_name>
    To scan all log directories of a given session (represented by a session_id) for errors:
    $ adopscanlog -session_id=<number>
    To see a complete list of supported parameters:
    $ adopscanlog -help
STEP 1:  Obtain all of the log files within the session ID directory. It is best to obtain a zip of the entire directory.
e.g. obtain a zip of /u01/PROD/fs_ne/EBSapps/log/adop/9
The session ID directory will contain:
a) A trace file for each phase;
e.g.
adop_20130316_085026.log
adop_20130316_091340.log
adop_20130316_210950.log
b) Logs grouped in phase directories
e.g.
prepare_20130316_085026
apply_20130316_091340
cutover_20130316_210950
You can see the timestamps match between each log directory and the respective trace file.
prepare_20130316_085026 directory matches adop_20130316_085026.log
apply_20130316_091340 directory matches adop_20130316_091340.log
etc etc
The same goes for fs_clone activities. i.e. you will see something like this
fs_clone_20130319_233614 --> this is a directory
adop_20130319_233614.log
In almost all cases, when debugging an adop failure the patch log directory will contain the information you need to determine root cause
e.g.
/u01/PROD/fs_ne/EBSapps/log/adop/2/cutover_20130316_210950/
In some cases the onscreen error will indicate which subroutine has failed. You will likely find a subdirectory containing log files which matches the failing routine. This should be where you focus your attention
e.g.
/u01/PROD/fs_ne/EBSapps/log/adop/2/cutover_20130316_210950/VIS_ufc/TXK_CTRL_forceshutdown_Sat_Mar_16_21_14_02_2013
STEP 2: Run the following SQL statements
This will show you the status for each adop phase along with its corresponding session id. This is effectively a history of online patching in an environment.
a) select ADOP_SESSION_ID,PREPARE_STATUS,APPLY_STATUS,FINALIZE_STATUS,CUTOVER_STATUS,CLEANUP_STATUS,ABORT_STATUS,STATUS,ABANDON_FLAG,NODE_NAME from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;
Note:
       Y denotes that the phase is done
       N denotes that the phase has not been completed
       X denotes that the phase is not applicable
       R denotes that the phase is running (in progress)
       F denotes that the phase has failed
       P (is applicable only to APPLY phase) denotes at least one patch is already applied for the session id
       C denotes that the status of this ADOP session has completed
Note: Numerical statuses are only relevant for the cutover phase...
These status values are updated when a step has completed, and are as follows:
N denotes that the phase has not been completed
0 denotes that cutover/force_shutdown has started
1 denotes the "force_shutdown" step has successfully executed
3 denotes the "db_cutover" step has successfully executed
4 denotes the "fs_cutover" step has successfully executed
6 denotes the "force_startup" step has successfully executed
Y denotes that the phase is done
cutover statuses
cutover_status='Y' 'COMPLETED'
cutover_status not in ('N','Y','X') and status='F' 'FAILED'
cutover_status='0' 'CUTOVER STARTED'
cutover_status='1' 'SERVICES SHUTDOWN COMPLETED'
cutover_status='3' 'DB CUTOVER COMPLETED'
cutover_status='D' 'FLIP SNAPSHOTS COMPLETED'
cutover_status='4' 'FS CUTOVER COMPLETED'
cutover_status='5' 'ADMIN STARTUP COMPLETED'
cutover_status='6' 'SERVICES STARTUP COMPLETED'
cutover_status='N' 'NOT STARTED'
cutover_status='X' 'NOT APPLICABLE'
STEP 3: Check the current status of the adop cycle
Source the run filesystem environment file and run command
adop -status
usage
adop -status  generates a summary report
adop -status <sessionID> generates a summary report for that session ID
adop -status -detail generates a detailed report
 STEP 4: Check AD and TXK C Patch levels
 SELECT codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in ('txk','ad');
STEP 5: Check middle tier technology patch levels
/u01/VIS/fs1/EBSapps/appl/ad/12.0.0/sql> perl $FND_TOP/patch/115/bin/TXKScript.pl -script=/u01/VIS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkInventory.pl -txktop=$APPLTMP -contextfile=$CONTEXT_FILE -appspass=apps -outfile=/tmp/Inventory_Report.html
Appendix
A.1 General problems with Finalize phase and Abort command
For phase=finalize issues run the following command and attach the adzdshowlog.out file generated:
sqlplus <apps_schema_name>/<apps_Schema_password> @$AD_TOP/sql/ADZDSHOWLOG.sql
Note: the contents of the table will be truncated every time cleanup/prepare phase is run.
Older log information is stored in adzdshowlog.out
or
select * from ad_zd_logs order by log_sequence desc;
A.2 'Duplicate keys found' during Finalize
ERROR at line 1:
ORA-20001: Error: while calling ad_zd.finalize .ORA-01452: cannot CREATE UNIQUE
INDEX; duplicate keys found
ORA-06512: at line 8
Run the following:
sqlplus <apps_schema_name>/<apps_Schema_password> @$AD_TOP/sql/ADZDSHOWLOG.sql
The output will highlight the unique index which is failing. You can then use the following sql to identify the duplicate key values
select <list of columns for which unique index creation failed>, count(*)
from <schema_name>.<table_name> group by <list of columns again> having count(*)>1
e.g.
select REPRESENTATION_CODE, TRX_NUMBER#2, ORG_ID, count(*) from OKL.OKL_TRX_CONTRACTS_ALL group by REPRESENTATION_CODE, TRX_NUMBER#2, ORG_ID having count(*)>1
A.3 Problems with Online Enablement
For failed attempts at Online Enablement i.e. failures of patch 13543062, gather the following information:
1. Patch log
2. Worker logs
3. Output of
select * from ad_zd_logs order by log_sequence desc;
4. output from:
@$AD_TOP/sql/ADZDSHOWDDLS.sql
If enablement appears to be having performance issues or seems to have hanged you can run the following script to determine if enablement is progressing or has hanged:
select count(1) from ad_zd_logs;
Run the scripts every 5 or  10 minutes. If the count is increasing then enablement is progressing.If enablement is progressing but very slowly ensure the DB initialization parameters are set as per the DB 11.2.0.3 and 12.2 requirements.
A.4 To show which patches were applied in each ADOP_SESSION_ID (patching cycle)
select * from ad_adop_session_patches order by end_date desc;
or
set pagesize 200;
set linesize 160;
column adop_session_id format 999999999999;
column bug_number format a15;
column status format a15;
column applied_file_system_base format a23;
column patch_file_system_base format a23;
column adpatch_options format a15;
column node_name format a15;
column end_date format a15;
column clone_status format a15;
select ADOP_SESSION_ID, BUG_NUMBER, STATUS, APPLIED_FILE_SYSTEM_BASE, PATCH_FILE_SYSTEM_BASE, ADPATCH_OPTIONS, NODE_NAME, END_DATE, CLONE_STATUS
from ad_adop_session_patches
order by end_date desc;
Note: STATUS
N - Not Applied In the current node but applied in other nodes
R - Patch Application is going on.
H - Patch failed in the middle. (Hard Failure)
F - Patch failed in the middle but user tried to skip some failures.
S - Patch Application succeeded after skipping the failed jobs.
Y - Patch Application succeeded.
C - Reserved for clone and config_clone. Indicates clone completed
A.5 Issues with FS_CLONE
a) select ADOP_SESSION_ID,BUG_NUMBER,CLONE_STATUS,STATUS,NODE_NAME from AD_ADOP_SESSION_PATCHES order by ADOP_SESSION_ID;
b) sqlplus <apps_schema_name>/<apps_Schema_password> @$AD_TOP/sql/ADZDSHOWLOG.sql
c) following files/dirs
$COMMON_TOP/clone/FMW/logs/CLONE<timestamp>.error
$COMMON_TOP/clone/FMW/logs/CLONE<timestamp>.log
A zip of files $INST_TOP/admin/log/clone directory
A zip of directory $APPLRGF/TXK/ohsCloneLog ( this captures cloning failures for OHS )
d) RUN and PATCH context files from all the nodes
e) if one of the scripts fails within fs_clone you can add some parameters and run it manually to obtain extra debug:
here are some examples
note the addition of -silent false and -debug true
/u01/applmgr/fs2/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /u01/applmgr/fs2/EBSapps/comn/util/jdk64 -al /u01/applmgr/fs1/EBSapps/comn/clone/FMW/WLS/EBSdomain.jar -tdl /u01/applmgr/fs2/FMW_Home/user_projects/domains/EBS_domain_sa3 -tmw /u01/applmgr/fs2/FMW_Home -mpl /u01/applmgr/fs1/EBSapps/comn/clone/FMW/WLS/plan/moveplan.xml -ldl /u01/applmgr/fs1/inst/apps/sa3_spitfire2/admin/log/clone/wlsT2PApply -silent false -domainAdminPassword /u01/applmgr/fs1/EBSapps/comn/clone/FMW/tempinfo.txt -debug true
or
/oa12uat/R122/fs2/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome
/oa12uat/R122/fs2/EBSapps/comn/util/jdk32 -al
/oa12uat/R122/fs1/EBSapps/comn/clone/FMW/OHS/ohsarchive.jar -mpl
/oa12uat/R122/fs1/EBSapps/comn/clone/FMW/OHS/moveplan.xml -ldl
/oa12uat/R122/fs1/inst/apps/oa12uat_coreuatap01/admin/log/clone/ohsT2PApply
-silent false -tcn EBS_web_oa12uat -tih
/oa12uat/R122/fs2/FMW_Home/webtier/instances/EBS_web_VIS_OHS1 -tinEBS_weVIS_OHS1 -toh /oa12uat/R122/fs2/FMW_Home/webtier -debug true
review the log and out files as well as the console output
A.6 ADOP Remote Invocation Issues and running ADOP on Multi Web Node Systems
Upload the following log files:
$APPL_TOP/admin/<SID>_patch/log/remote_execution_result.xml
$APPL_TOP/admin/<SID>/log/remote_execution_result.xml
Adop log directory corresponding to the failed adop id
Other key things to consider in multi web node instances.
- SSH should be enabled to allow communication from the primary node to the secondary nodes. Use txkRunSSHSetup.pl to enable SSH.
- If SSH is not enabled pay very special attention to the specific requirements for running adop in this case (see the maintenance guide)
- When patching ensure you copy patches to the patch home directory of all application tiers e.g. fs_ne/EBSapps/patch
- When debugging issues make sure to you review the adop logs on all application tiers.
Subscribe to: Posts ( Atom )