R12.2 adop patching and Cloning issue and solution

Prerequisites before starting patching activity.
1. Check Patch top size ,filesystem free space  , it should be minimum ~ 26.5- 27 GB approx. free space, otherwise Prepare phase will fail
2. Check context_file and remove all custom entry below CUSTOM TOP
3. Check inventory which should point correctly
4. In case of DMZ instance(APDROD & APDTST)  download all patch on both internal and external (DMZ) node under PATCH_TOP (Double check patch on both nodes before apply phase)
Issue:
1. If prepare/apply phase/Cutover fails, check log under $ADOP_LOG_HOME/current_session_id
Find the issue and fix then trigger the adop cycle prepare/apply/cutover etc.
2. If patch is applied in internal node but failing in external (DMZ) node, THEN Crosscheck patch is downloaded in external node under PATCH_TOP. Check log in external node also for any issue. Download patch on both node and triggered apply phase.
3. If node is abandon during adop_phase, please check log for issue
Run fs_clone and start the prepare/apply phase
4. If node is abandon during Cutover and filesystem got interchanged in primary node but not in another node, check log for issue.
This is absolutely not recommended and will lead to ADOP corruption if not handled based on the scenario.
Take backup of ad_adop_session table before updating ad_adop_session table (strongly recommended)
Manual cutover after mocking values in ADOP tables
• Update ad_adop_session table set abandon flag=NULL
• Run fs_clone with option force=yes
• Run cutover manually by option allnodes=no

                       Cloning issue and solution
1. ADcfgclone is failed while creating FMW Home
Error:
==============
/d02/app/APDT122/fs2START: Creating FMW Home.
Running /d02/app/APDT122/fs2/EBSapps/comn/clone/FMW/pasteBinary.sh -javaHome /d02/app/APDT122/fs2/EBSapps/comn/util/jdk64 -al /d02/app/APDT122/fs2/EBSapps/comn/clone/FMW/FMW_Home.jar -tl /d02/app/APDT122/fs2/FMW_Home -invPtrLoc /etc/oraInst.loc -ldl /d02/app/APDT122/fs2/inst/apps/APDTST_apdbasrv02/admin/log/clone/fmwT2PApply -silent true -debug true -executeSysPrereqs false
Script Executed in 28420 milliseconds, returning status 255
Script failed, exit code 255
Solution:  Make sure APPLTMP location should have atleast 10GB free space
Export T2P_JAVA_OPTIONS="Djava.io.tmpdir=/d02/app/APDT122/fs_ne/inst/APDTST_apdbasrv02/temp"
and ran the adcfgclone.pl
2. Adcfgclone got complete but at the end it failed for autoconfig with the below error.
Error :
=====================
WARNING: [AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
      <filename>  <phase>  <return code where appropriate>
               [PROFILE PHASE]
                AutoConfig could not successfully execute the following scripts:
Directory: /d02/app/APDT122/fs2/inst/apps/APDTST_apdbasrv02/admin/install
      adadmprf.sh             INSTE8_PRF
Solution:  work around
1. Take backup of ad_timestamps, ad_appl_tops
2. Remove the rows which correspond to current node (on which the script is failing)
           SQL> create table ad_timestamps_bkp as select * from ad_timestamps;
                     Table created.
             SQL> create table  ad_appl_tops_bkp as select * from  ad_appl_tops;
                      Table created.
          SQL>  select distinct substr(attribute, 1, instr(attribute, '*') - 1)
                         from ad_timestamps
                          where type in ('INSTANTIATED_CURRENT_VIEW_SNAPSHOT',                           'BACKFILLED_BUGS_IN_CURRENT_VIEW_SNAPSHOT');
          SQL> delete ad_timestamps where type in ('INSTANTIATED_CURRENT_VIEW_SNAPSHOT',
                     'BACKFILLED_BUGS_IN_CURRENT_VIEW_SNAPSHOT');
          SQL>  delete ad_appl_tops where APPLICATIONS_SYSTEM_NAME='APDROD';
3. Run the autoconfig , it will complete successfully.
4. DMZ URL is not working
Solution: please confirm the port in DMZ and post clone steps for DMZ
          ii. Modify the HTTP port in the $CONTEXT_FILE of the external node and run autoconfig
               iii . Check SSL setup in the $CONTEXT_FILE and run autoconfig
              Issue:
External URL not up. Due to http port using wrong port#8002.
Processes in Instance: EBS_web_APDTST_OHS2
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+-----
EBS_web_APDTST                   | OHS                |   18624 | Alive    | 1412243724 |  1870960 |   0:24:01 | https:4445,https:10001,http:8002
Solution:
Manually modified port#8001 in httpd.conf since it is not changed by autoconfig.
[apapdtst@apdbadmz01 EBS_web_APDTST]$ pwd
/d02/app/APDT122/fs1/FMW_Home/webtier/instances/EBS_web_APDTST_OHS2/config/OHS/EBS_web_APDTST
[apapdtst@apdbadmz01 EBS_web_APDTST]$ grep "8001" *
httpd.conf:Listen 8001
[apapdtst@apdbadmz01 EBS_web_APDTST]$
Database Restore Issue:  
5. Database restore was failing because of space issue and not having apdficient space for mapping also.
Solution: As some of the mount point in APDTST has not enough space to restore all the datafiles from APDROD’s .
Remap mount points from source to target accordingly ,if any mount point still lacking space after mapping and adjusting all datafiles then triggered restore and move restored datafile to free mount point.
   ii. Once restore complete rename the datafiles from old to new location as below.
Rename datafile (ALTER DATABASE RENAME FILE '/old/location' TO '/new/location'; )