12.2 E-Business Suite Adop Phase=Prepare Fails with FATAL ERROR FUNCTION: main::runFSCloneApply and CLONE-20603 Error in validating "SERVER_CONFIG" (Doc ID 2163053.1)

SYMPTOMS
E-Business Suite 12.2.4 Appplications DBA, ADOP Utility issues
When attempting to run ADOP Prepare, the following error occurs:
Data Collection
================
>> from txkADOPPreparePhaseSynchronize.log
Creating the directory: /01/oracle/TEST/EBSapps/log/adop/fs_clone/TEST/TXK_SYNC_create/fsclone_apply
Copying the directory
---------------------
SOURCE : /01/oracle/TEST/fs2/inst/apps/TEST/admin/log/clone
TARGET : /01/oracle/TEST/EBSapps/log/adop/21/fs_clone/TEST/TXK_SYNC_create/fsclone_apply
/01/oracle/TEST/fs2/EBSapps/comn/adopclone/bin/adclone.pl did not go through successfully.
LOG DIRECTORY: /01/oracle/TEST/EBSapps/log/adop/21/fs_clone/TEST/TXK_SYNC_create/fsclone_apply.
*******FATAL ERROR*******
PROGRAM : (/01/oracle/TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Fri Jul 08 01:12:07 2016
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /01/oracle/TEST/fs2/EBSapps/comn/adopclone/bin/adclone.pl did not go through successfully.
...........................
>> from adzdshowlog.out
PROCEDURE [END] Check and Stop patch Node Manager
204559085 1:12:43 00:00:00 ad.bin.adop PROCEDURE [END] Check and stop patch Node Manager
204559085 1:12:43 00:00:00 ad.bin.adop UNEXPECTED fs_clone has failed.
204559085 1:12:43 00:00:00 ad.bin.adop PROCEDURE [START] Unlocking sessions table
204559085 1:12:43 00:00:00 ad.bin.adop STATEMENT Unlocking ad_adop_sessions table for nferptap with wait interval of 60 seconds and number of tries 2
204559085 1:12:43 00:00:00 AD_ZD_ADOP.UNLOCK_SESSIONS_TABLE WARN WARN: Trying to unlock ad_adop_sessions table when no lock in place.
204559085 1:12:43 00:00:00 ad.bin.adop PROCEDURE [END] Unlocking sessions table
204560088 6:24:03 05:11:19 ad.plsql.ad_zd.set_edition ERROR Unable to set edition. Invalid edition type : PATCH
204560118 6:31:14 00:07:11 ad.plsql.ad_zd.set_edition ERROR Unable to set edition. Invalid edition type : PATCH
.........................
>> from FSCloneApplyAppsTier_07080046.log
START: Creating new WLS domain.
Running /01/oracle/TEST/fs1/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /01/oracle/TEST/fs1/EBSapps/comn/util/jdk64 -al /01/oracle/TEST/fs2/EBSapps/comn/adopclone/FMW/WLS/EBSdomain.jar -tdl /01/oracle/TEST/fs1/FMW_Home/user_projects/domains/EBS_domain_TEST -tmw /01/oracle/TEST/fs1/FMW_Home -mpl /01/oracle/TEST/fs2/EBSapps/comn/adopclone/FMW/WLS/plan/moveplan.xml -ldl /01/oracle/TEST/fs2/inst/apps/TEST/admin/log/clone/wlsT2PApply -silent true -debug true -domainAdminPassword /01/oracle/TEST/fs2/EBSapps/comn/adopclone/FMW/tempinfo.txt
Script Executed in 7190 milliseconds, returning status 255
ERROR: Script failed, exit code 255
START: Inside exitClone....
Updating status INCOMPLETE for ApplywlsTechStack
...............
>> from CLONE2016-07-08_01-12-01_1836266301.error
SEVERE : Jul 8, 2016 01:12:07 - ERROR - CLONE-20603 Error in validating "SERVER_CONFIG".
SEVERE : Jul 8, 2016 01:12:07 - CAUSE - CLONE-20603 Specified "Custom Identity Keystore File" file for server "Server1" was not valid.
SEVERE : Jul 8, 2016 01:12:07 - ACTION - CLONE-20603 Make sure that "Custom Identity Keystore File" file is valid.
java.lang.Exception: Specified "Custom Identity Keystore File" file for server "Server1" was not valid.
at oracle.as.clone.cloner.component.j2ee.config.GenericPasteConfigSteps.getServerList(GenericPasteConfigSteps.java:1326)
at oracle.as.clone.cloner.component.j2ee.config.GenericPasteConfigSteps.getServerConfig(GenericPasteConfigSteps.java:1189)
at oracle.as.clone.cloner.component.j2ee.config.GenericPasteConfigSteps.executeGenericPasteConfig(GenericPasteConfigSteps.java:366)
at oracle.as.clone.cloner.component.J2EEComponentApplyCloner.doClone(J2EEComponentApplyCloner.java:252)
at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)
at oracle.as.clone.request.ApplyCloneRequest.applyArchive(ApplyCloneRequest.java:188)
at oracle.as.clone.request.ApplyCloneRequest._clone(ApplyCloneRequest.java:69)
at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)
at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)
at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:257)
at oracle.as.clone.client.CloningClient.main(CloningClient.java:119)
oracle.as.t2p.exceptions.FMWT2PPasteConfigException: PasteConfig failed. Make sure that the move plan and the values specified in moveplan are correct.
at oracle.as.clone.cloner.component.J2EEComponentApplyCloner.doClone(J2EEComponentApplyCloner.java:302)
at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)
at oracle.as.clone.request.ApplyCloneRequest.applyArchive(ApplyCloneRequest.java:188)
at oracle.as.clone.request.ApplyCloneRequest._clone(ApplyCloneRequest.java:69)
at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)
at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)
at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:257)
at oracle.as.clone.client.CloningClient.main(CloningClient.java:119)
SEVERE : Jul 8, 2016 01:12:07 - ERROR - CLONE-20237 Restoring the sourceid "J2EECOMPONENT@EBS_domain_TEST" has failed.
SEVERE : Jul 8, 2016 01:12:07 - CAUSE - CLONE-20237 An internal operation failed.
SEVERE : Jul 8, 2016 01:12:07 - ACTION - CLONE-20237 Check the clone log for more details.
SEVERE : Jul 8, 2016 01:12:07 - ERROR - CLONE-20218 Cloning is not successful.
SEVERE : Jul 8, 2016 01:12:07 - CAUSE - CLONE-20218 An internal operation failed.
SEVERE : Jul 8, 2016 01:12:07 - ACTION - CLONE-20218 Provide the clone log and error file for investigation.
..............................
>> from prepare adop log
Lines #(196-200):
  [PROCEDURE] Calling: /01/oracle/TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
  [EVENT] Log: /01/oracle/TEST/EBSapps/log/adop/19/fs_clone/TEST
  [UNEXPECTED]Error occurred running "perl /01/oracle/TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/01/oracle/TEST/fs2/inst/apps/TEST/appl/admin/TEST.xml -patchcontextfile=/01/oracle/TEST/fs1/inst/apps/TEST/appl/admin/TEST.xml -promptmsg=hide -console=off -mode=create -sessionid=19 -timestamp=20 -outdir=/01/oracle/TEST/fs_ne/EBSapps/log/adop/19/fs_clone/TEST"
  [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /01/oracle/TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/01/oracle/TEST/fs2/inst/apps/TEST/appl/admin/TEST.xml -patchcontextfile=/01/oracle/TEST/fs1/inst/apps/TEST/appl/admin/TEST.xml -promptmsg=hide -console=off -mode=create -sessionid=19 -timestamp=20160629_174631 -outdir=/01/oracle/TEST/EBSapps/log/adop/19/fs_clone/TEST".
  [EVENT] Releasing managed server ports.
Lines #(197-201):
  [EVENT] Log: /01/oracle/TEST/EBSapps/log/adop/19/fs_clone/TEST
  [UNEXPECTED]Error occurred running "perl /01/oracle/TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/01/oracle/TEST/fs2/inst/apps/TEST/appl/admin/TEST_nferptap.xml -patchcontextfile=/01/oracle/TEST/fs1/inst/apps/TEST/appl/admin/TEST.xml -promptmsg=hide -console=off -mode=create -sessionid=19 -timestamp=20 -outdir=/01/oracle/TEST/fs_ne/EBSapps/log/adop/19/fs_clone/TEST"
  [UNEXPECTED]
STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Login to UNIX Shell as APPLMGR user
2. Source to environment file
3. Execute command: adop phase=prepare

CAUSE
From AdminServer.out:
..............
<Channel "DefaultSecure" is now listening on 100.10.1.100:7000 for protocols iiops, t3s, ldaps, https.>
<Channel "Default" is now listening on 100.10.1.100:7000 for protocols iiop, t3, ldap, snmp, http.>
................
>> from config.xml in diagnostic_info_test_URL.tar.gz
AdminServertruetruefalsefalseBuiltinSSLValidationOnlyBuiltinSSLValidationOnly
trueAdminServertrue
================================
-->> SSL was enabled on the Admin server as described in the log.
Refer to the following unpublished defect for additional details:
Bug 20468451 - ADOP PREPARE PHASE FAILED.
AdminServer is SSL enabled which is not correct.
Content from config.xml:
========================
<server>
<name>AdminServer</name>
<ssl> <------
<enabled>true</enabled> <------
</ssl> <------
<machine>test_url</machine>
<listen-port>7000</listen-port>
<listen-address>test_url.us.oracle.com</listen-address>
<server-start>
<name>AdminServer</name>
<java-vendor>Oracle</java-vendor>
<java-home>/appmgr/EBS/fs2/EBSapps/comn/util/jdk64</java-ho
me>
</server-start>
<client-cert-proxy-enabled>true</client-cert-proxy-enabled>
<weblogic-plugin-enabled>true</weblogic-plugin-enabled>
<server-diagnostic-config>
<name>AdminServer</name>
<diagnostic-context-enabled>true</diagnostic-context-enabled>
</server-diagnostic-config>
<data-source>
<rmi-jdbc-security xsi:nil="true"></rmi-jdbc-security>
</data-source>
</server>
AdminServer.out shows:
======================
####<Jan 29, 2015 9:14:07 PM PST> <Notice> <Server> <slc01ref> <AdminServer>
<[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default
(self-tuning)'> <<WLS Kernel>> <>
<cb6affb6:-4589:14b394:-800-000000000000000>
<1422> <BEA-002613> <Channel "DefaultSecure" is now listening on
100.10.1.100:7000 for protocols iiops, t3s, ldaps, https.>
SOLUTION
To resolve the issue test the following steps in a development instance and then migrate accordingly:
1. Backup the admin server config.xml file.
2. Disable SSL setup for the Admin server config.xml
2. For more information on on SSL protocols for Fusion Middleware reference the following document:
Note 1936300.1 - How to Change SSL Protocols (to Disable SSL 2.0/3.0) in Oracle Fusion Middleware Products (Doc ID 1936300.1)