Oracle Application DBA

On EBS 12.2 when applying a patch using adop.sh in the prepare phase it fails with the error:
ERROR at line 1:
ORA-20001: Error: while calling ad_zd.cleanup(normal).ORA-20007: Ancestors of
Current RUN Edition are not retired
ORA-06512: at line 7
The issue can be reproduced at will with the following steps:
1. Apply patch using adop.sh.
CAUSE
The following data from customer environment shows V_20120409_0455 not retired. The other edition, ORA$BASE is correctly retired.
select apps.ad_zd.get_edition('OLD') "OLD",
  apps.ad_zd.get_edition('RUN') "RUN",
  apps.ad_zd.get_edition('PATCH') "PATCH"
from dual;
EDITION_NAME PARENT_EDITION_NAME USA
------------------------------ ------------------------------ ---
ORA$BASE YES
V_20120409_0455 ORA$BASE YES
V_20120521_0936 V_20120409_0455 YES
column CURRENT_EDITION format a30
column EDITION_TYPE format a15
select apps.ad_zd.get_edition "CURRENT_EDITION",
  apps.ad_zd.get_edition_type "EDITION_TYPE"
from dual;
OLD RUN PATCH
-------------------- -------------------- --------------------
V_20120409_0455 V_20120521_0936
Previous patch application did not get retired.
SOLUTION
1. Please run the following command to retire the edition:
SQL> exec sys.ad_zd_sys.retire_old_editions;
After this the following sql should not return any row for V_20120409_0455:
 select *
 from dba_tab_privs
 where table_name in (select edition_name from dba_editions);
2. Once the V_20120409_0455 is retired successfully, one will be able to do the cleanup and proceed further with the adop cycles.

Issue:
=============================
Inside evalADPATCHStatus()...
=============================
message_status: ERROR
Application of patch did not go through successfully.
*******FATAL ERROR*******
PROGRAM : (/u11/home/apdba/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Sat Feb  6 05:13:21 2016
FUNCTION: main::execADPATCH [ Level 1 ]
ERRORMSG: Application of patch did not go through successfully.
Cause:
For some reason , even if we source PATCH fs, custom tops were pointing to RUN fs.
[apdba@apdba-blog ~]$ . .bash_profile
  E-Business Suite Environment Information
  ----------------------------------------
  RUN File System           : /u11/home/apdba/fs2/EBSapps/appl
  PATCH File System         : /u11/home/apdba/fs1/EBSapps/appl
  Non-Editioned File System : /u11/home/apdba/fs_ne
  DB Host: edd01db01-vip.apdba.com  Service/SID: apdba
  E-Business Suite Environment Setting
  ------------------------------------
  - Enter [R/r] for sourcing Run File System Environment file, or
  - Enter [P/p] for sourcing Patch File System Environment file, or
  - Enter anything else to exit
  Please choose the environment file you wish to source [R/P]:P
  Sourcing the PATCH File System ...
[apdba@apdba-blog ~]$
[apdba@apdba-blog ~]$
[apdba@apdba-blog ~]$ echo $BIO_TOP
/u11/home/apdba/fs2/EBSapps/appl/bio/12.0.0
[apdba@apdba-blog ~]$
Fix:
Source PATCH fs and run autoconfig.
PDHINT2 Status:
[apdba@apdba-blog ~]$ adop -status
Enter the APPS password:
==============================================================
ADOP (C.Delta.6)
Session Id: 20
Command: status
Output: /u11/home/apdba/fs_ne/EBSapps/log/status_20160206_063201/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
apdba-blog    master     PREPARE         COMPLETED       2016/02/06 05:10:58  2016/02/06 06:27:21  1:16:23
                           APPLY           ACTIVE          2016/02/06 06:29:49  2016/02/06 06:31:05  0:01:16
                           FINALIZE        NOT STARTED
                           CUTOVER         NOT STARTED
                           CLEANUP         NOT STARTED

apdba-blog01    slave      PREPARE         COMPLETED       2016/02/06 05:12:34  2016/02/06 06:27:28  1:14:54
                           APPLY           ACTIVE          2016/02/06 06:31:22  2016/02/06 06:31:24  0:00:02
                           FINALIZE        NOT STARTED
                           CUTOVER         NOT STARTED
                           CLEANUP         NOT STARTED
File System Synchronization Type: Light
adop exiting with status = 0 (Success)

[ERROR] Failed to execute SQL statement :
  declare
  result varchar2(10);
  begin
  ad_zd_adop.cutover;
  exception when others then
  update ad_adop_sessions set status='F'
  where adop_session_id=26;
  commit;
  raise_application_error(-20001,'Error while calling ad_zd_adop.cutover.' || sqlerrm);
  end;

  [ERROR] Error Message :
  [ERROR] ORA-20001: Error while calling ad_zd_adop.cutover.ORA-01422: exact fetch returns more than requested number of rows
  [ERROR] ORA-06512: at "APPS.AD_ZD_LOG", line 78
  [ERROR] ORA-06512: at "APPS.AD_ZD", line 17
  [ERROR] ORA-06512: at "APPS.AD_ZD", line 61
  [ERROR] ORA-01422: exact fetch returns more than requested number of rows
  [ERROR] ORA-06512: at line 9 (DBD ERROR: OCIStmtExecute)
  [UNEXPECTED]Error occurred while performing database cutover
  [UNEXPECTED]Cutover phase completed with errors/warnings. Please check logfiles
Solution:
Please use the following workaround.
1. Bounce the Database completely, ensure that all RAC nodes are down.
2. Update ADOP Repository with the below SQLs :
Take a backup of table.
  1) update ad_adop_sessions set cutover_status='3' where adop_session_id=<session id> and node_name=<host name>;
  2) update ad_adop_sessions set abort_status='X' where adop_session_id=<session id>;
3. After commit, restart Cutover using the same command.

we have submitted several requests for the program “APDBA Item Interface” which are running for close to 20mins for a batch of 500 records each. During Mock & in UAT the same programs run in less than 5mins for a batch of 500 records.
Per ADDM recommendation (and AWR report proves it) we should focus on following SQL statement:
   Recommendation 1: SQL Tuning
   Estimated benefit is 3.28 active sessions, 69.89% of total activity.
   --------------------------------------------------------------------
   Action
      Run SQL Tuning Advisor on the SELECT statement with SQL_ID
      "839f9mtcmu5yn".
      Related Object
         SQL statement with SQL_ID 839f9mtcmu5yn.
         SELECT COUNT(*) FROM MTL_SYSTEM_ITEMS_B MSI WHERE INVENTORY_ITEM_ID
         IN ( SELECT I.INVENTORY_ITEM_ID FROM MTL_PARAMETERS MP,
         MTL_SYSTEM_ITEMS_INTERFACE I, MTL_DESCR_ELEMENT_VALUES MDEV WHERE
         I.PROCESS_FLAG = :B3 AND I.SET_PROCESS_ID = :B2 AND
         I.TRANSACTION_TYPE = :B1 AND I.ITEM_CATALOG_GROUP_ID =
         MSI.ITEM_CATALOG_GROUP_ID AND I.ORGANIZATION_ID = MP.ORGANIZATION_ID
         AND I.ORGANIZATION_ID = MP.MASTER_ORGANIZATION_ID AND
         I.INVENTORY_ITEM_ID=MDEV.INVENTORY_ITEM_ID AND I.ORGANIZATION_ID
         <>MSI.ORGANIZATION_ID )
   Rationale
      The SQL spent 95% of its database time on CPU, I/O and Cluster waits.
      This part of database time may be improved by the SQL Tuning Advisor.
   Rationale
      Database time for this SQL was divided as follows: 100% for SQL
      execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
      execution.
   Rationale
      SQL statement with SQL_ID "839f9mtcmu5yn" was executed 1139 times and
      had an average elapsed time of 2.3 seconds.
   Rationale
      Top level calls to execute the PL/SQL statement with SQL_ID
      "3hmqs17s6sgn1" are responsible for 100% of the database time spent on
      the SELECT statement with SQL_ID "839f9mtcmu5yn".
      Related Object
         SQL statement with SQL_ID 3hmqs17s6sgn1.
         BEGIN BIO_INV_ITEMPDH_PKG.main(:errbuf,:rc,:A0,:A1); END;
~~~

As has been advised we have run SQL tuning advisor for SQL ID 839f9mtcmu5yn.
There is a bunch of recommendations have been produced but we recommend to try to apply the SQL profile in-first:
~~~
  Recommendation (estimated benefit: 99.76%)
  ------------------------------------------
  - Consider accepting the recommended SQL profile.
    execute dbms_sqltune.accept_sql_profile(task_name => 'SQL_tun_20160202',
            task_owner => 'SYS', replace => TRUE);
  Validation results
  ------------------
  The SQL profile was tested by executing both its plan and th
e original plan
  and measuring their respective execution statistics. A plan
may have been
  only partially executed if the other could be run to completio
n in less time.
                           Original Plan  With SQL Profile  %
 Improved
                           -------------  ----------------  --------
--
  Completion Status:            COMPLETE          COMPLETE
  Elapsed Time (s):               3.476156           .030605      99.11
 %
  CPU Time (s):                    3.458475           .028995      99.16 %
  User I/O Time (s):                  0           .00138
4
  Buffer Gets:                         636679              1463      99.77 %
  Physical Read Requests:             0
6
  Physical Write Requests:            0                 0
  Physical Read Bytes:                0             57344
  Physical Write Bytes:               0                 0
  Rows Processed:                     1
  1
  Fetches:                                 1                 1
  Executions:                           1                 1
  Notes
  -----
  1. Statistics for the original plan were averaged over 1 executions.
  2. Statistics for the SQL profile plan were averaged over 10 executions.
 
  AWR Report.
  From AWR report:
~~~
Elapsed:   120.08 (mins) =  7,204.8 (sec)
DB Time:   563.57 (mins) = 33,814.2 (sec)
NUM_CPUS 72
--
Available DB time = NUM_CPU's * (elapsed_time between snapshots) * 60 (sec) 
 =  72 * 120.08 * 60 = 518,745.6 (sec)
Thus actual DB time (33,814.2 sec) is just a little part of Available DB time (518,745.6 sec), it means that not much time was spent on DB side for observed period.
Solution:
SQL> select name from v$database;
NAME
---------
EBSPRD
SQL> sho user
USER is "SYS"
SQL> execute dbms_sqltune.accept_sql_profile(task_name => 'SQL_tun_20160202', task_owner => 'SYS', replace => TRUE);
PL/SQL procedure successfully completed.
SQL>  select NAME,CREATED,TYPE,status from dba_sql_profiles;
NAME
------------------------------
CREATED
----------------------------------
TYPE    STATUS
------- --------
SYS_SQLPROF_0152a14edd900000
02-FEB-16 04.27.26.000000 AM
MANUAL  ENABLED
SYS_SQLPROF_01487ff300e00000
29-JAN-16 10.04.01.000000 AM
MANUAL  ENABLED
NAME
------------------------------
CREATED
-----------------------------------
TYPE    STATUS
------- --------
SYS_SQLPROF_0134d7c544630001
29-JAN-16 10.04.01.000000 AM
MANUAL  ENABLED
SYS_SQLPROF_0134d7b014e40000
29-JAN-16 10.04.02.000000 AM
NAME
------------------------------
CREATED
-----------------------------------
TYPE    STATUS
------- --------
MANUAL  ENABLED
Now the threads are completing in 2 minutes.

The error message from the discovery process shows this is the cause of the issue as we can see in the message extracted from the log file:
Caused by: oracle.apps.fnd.oam.em.sdk.targetData.discovery.AppsDiscoveryException: Duplicate entries exists in FND_OAM_CONTEXT_FILES Tables in Apps Database.
Please clean the table before running Discovery again.
Solution
1. Backup the fnd_oam_context_files table :
sqlplus applsys/pwd
create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;
2. truncate table fnd_oam_context_files;
3. Run autoconfig in the DB tiers to populate fnd_oam_context_files table with the database information;
4. Run autoconfig in each Apps tier to populate fnd_oam_context_files with the application tier information;
5. From the EBS agent home, run:
./emctl upload
This will upload the information from the EBS system to Grid Control
6. Re-run the Discovery process.

When attempting to apply patch p18123067_R12.FND.C_R12_GENERIC.zip,
ERROR
-----------------------
CLONE2014-03-14_02-33-17PM.error
WARNING : Mar 14, 2014 2:33:21 PM - ERROR - CLONE-20263   Invalid Oracle Inventory.
WARNING : Mar 14, 2014 2:33:21 PM - CAUSE - CLONE-20263   There are two possible causes for an invalid OraInventory "/u01/PROD/oraInventory".
1. Inventory was empty.
2. It did not have valid inventory.xml file.
WARNING : Mar 14, 2014 2:33:21 PM - ACTION - CLONE-20263   Provide a valid Oracle Inventory.
The inventory directory does not exist.
1. /etc/oraInct.loc ==> /u01/PAT/oraInventory (customer's OLD PAT Centralized Inventory Directory)
2. "/u01/PROD/oraInventory" Des Not exist on the instance.
Sol
1. change the setting for /etc/oraInventory.loc to an existing directory
2. run adop phase=fs_clone force=yes OR adop phase=fs_clone

dba_network_acls
oraprd@edp01dbadm01:/erp_admin/app/erp/SCALE $ cat ACL2.sql
#Run the following procedure to create ACL for the above #listed URL's

BEGIN
dbms_network_acl_admin.assign_acl (
acl => '/sys/acls/OracleEBS.xml',
host => '*'
);
END;
/
commit;

PREPARE in EBSPAT has failed with following error in logs
*******FATAL ERROR*******
PROGRAM : (/u11/home/ebspat/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Tue Jan 19 11:09:53 2016
FUNCTION: main::migrateCloneComponentApply [ Level 1 ]
ERRORMSG: /u11/home/ebspat/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl did not go through successfully.
Enrolling this machine with the domain directory at /u11/home/ebspat/fs2/FMW_Home/user_projects/domains/EBS_domain_ebspat ...
ERROR: Unable to enrol the domain home '/u11/home/ebspat/fs2/FMW_Home/user_projects/domains/EBS_domain_ebspat' with the Node Manager
ERROR:  Unable to register the host ebsx01appd06 with domain EBS_domain_ebspat
ERROR:  Can't proceed further.
*************************
Sol: 
adnodemgrctl.sh stop
adnodemgrctl.sh start
run adop phase=prepare
There is no issue but just failed communication with node manager , restarting node manager and running prepare should work.

[applint2@ebsx01appd10 ~]$ adop phase=prepare
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials...
Initializing...
    Run Edition context  : /u15/home/ebsint2/fs1/inst/apps/ebsint2_ebsx01appd10/appl/admin/ebsint2_ebsx01appd10.xml
    Patch edition context: /u15/home/ebsint2/fs2/inst/apps/ebsint2_ebsx01appd10/appl/admin/ebsint2_ebsx01appd10.xml
    Patch file system freespace: 422.31 GB
Validating system setup...
    Node registry is valid.
Logfile location /u15/home/ebsint2/fs1/inst/apps/ebsint2_ebsx01appd10/logs/appl/rgf/TXK/verifyssh.log
xml output = /u15/home/ebsint2/fs1/inst/apps/ebsint2_ebsx01appd10/logs/appl/rgf/TXK/out.xml
    Remote execution is operational.
    [ERROR]     Failed to execute SQL statement :
   declare
    l_msg varchar2(4000);
  begin
    ad_zd_adop.adop_database_validations(l_msg);
    dbms_output.put_line(l_msg);
  end;
    [ERROR]     Error Message :
    [ERROR]     ORA-04063: package body "APPS.AD_ZD_PREP" has errors
    [ERROR]     ORA-06508: PL/SQL: could not find program unit being called: "APPS.AD_ZD_PREP"
    [ERROR]     ORA-06512: at "APPS.AD_ZD_ADOP", line 2443
    [ERROR]     ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)
    [UNEXPECTED]Error occurred while performing database validations
[STATEMENT] Please run adopscanlog utility, using the command
"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 1 (Fail)
Sol:
Tried to compile the package "AD_ZD_PREP" but failed with the following error
SQL> alter package AD_ZD_PREP compile body;
Warning: Package Body altered with compilation errors.
SQL> sho errors
Errors for PACKAGE BODY AD_ZD_PREP:
LINE/COL ERROR
-------- -----------------------------------------------------------------
419/3    PL/SQL: SQL Statement ignored
419/19   PL/SQL: ORA-00942: table or view does not exist
423/5    PL/SQL: SQL Statement ignored
423/21   PL/SQL: ORA-00942: table or view does not exist
457/5    PL/SQL: SQL Statement ignored
457/21   PL/SQL: ORA-00942: table or view does not exist
SQL>
SQL> desc DBMS_OBJECTS_APPS_UTILS
ERROR:
ORA-04043: object DBMS_OBJECTS_APPS_UTILS does not exist
Issue was grants on the sys object "xdb$moveSchemaTab" to apps
SQL> grant select,update,delete,insert on sys.xdb$moveSchemaTab to apps;
Grant succeeded.
Now compiled the package "AD_ZD_PREP"
SQL> alter package apps.AD_ZD_PREP compile body;
Package body altered.
SQL>
run [applint2@ebsx01appd10 ~]$ adop phase=prepare
ADOP cycle completed without any further issues, cause for the above issue could be failed run of adgrants.sql

[ERROR] Failed to execute SQL statement :
select AD_ZD_ADOP.GET_INVALID_NODES() from dual
[ERROR] Error Message :
[ERROR] Failed to execute SQL statement :
select AD_ZD_ADOP.GET_INVALID_NODES() from dual
[ERROR] Error Message :
[UNEXPECTED]Nodes "-1" are listed in ADOP_VALID_NODES table but not in FND_NODES table.
[UNEXPECTED]To correct this, run AutoConfig on nodes "-1"
[UNEXPECTED]Error while checking if this is a multi node instance
SQL> ALTER PACKAGE APPS.AD_ZD_ADOP COMPILE BODY;
Warning: Package Body altered with compilation errors.
SQL> SHOW ERROR
Errors for PACKAGE BODY APPS.AD_ZD_ADOP:
LINE/COL ERROR
-------- -----------------------------------------------------------------
505/13 PL/SQL: SQL Statement ignored
507/86 PL/SQL: ORA-00904: "SESSION_TYPE": invalid identifier
511/13 PL/SQL: SQL Statement ignored
Solution:
1. Verify that your ad_adop_session_patches.xdf file is version 120.6.12020000.4
grep Header $AD_TOP/patch/115/xdf/ad_adop_session_patches.xdf
2. If the version is correct, then run this command
adjava -mx512m -nojit oracle.apps.fnd.odf2.FndXdfCmp applsys APPS apps APPS thin <host>:<PORT>:<SID>
table $AD_TOP/patch/115/xdf/ad_adop_session_patches.xdf $FND_TOP/patch/115/xdf/xsl changedb=y
Please put in the actual host (fully qualified), port, and SID
NOTE: If you would like to test the command first, please change "changedb=y" to "changedb=n".
3. Please retry compiling AD_ZD_ADOP (or retry running ADZDADOPB.pls).
4. If the compilation works, retry the patch.

adwork001.log
SQL Exception : java.sql.SQLException: ORA-06550: line 1, column 14:
PLS-00201: identifier 'JDR_PATHS=' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
ORA-06512: at "APPS.AD_ZD_SEED", line 1310
ORA-06512: at line 1
Solution:
1. EXEC AD_ZD_SEED.UPGRADE('JDR_PATHS');
2. adop phase=apply patches=20330477 restart=yes flags=autoskip

adop prepare phase failed
ssh is not enabled for the following nodes
    - - - - - - - - - - - - - - - - - - - - - -
    pdhx01appd06
    [UNEXPECTED]ssh is not enabled.
    [UNEXPECTED]Hint: These are some available options:
    [UNEXPECTED]- If you would like to enable ssh, execute the following command:
     perl /u11/home/pdhuat/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh \
          -contextfile=/u11/home/pdhuat/fs2/inst/apps/pdhuat_pdhx01appd05/appl/admin/pdhuat_pdhx01appd05.xml \
          -hosts=pdhx01appd05,pdhx01appd06
Solution:
1. Password is expired.
[applpdhu@pdhx01appd05 ~]$ ssh pdhx01appd06
[applpdhu@pdhx01appd06 ~]$ ssh pdhx01appd05
Your account has expired; please contact your system administrator
Connection closed by 10.2.217.143
2. set the password of applmgr user on both nodes.
[applpdhu@pdhx01appd05 clone]$ ssh pdhx01appd06
Last login: Fri Jan  8 13:56:17 2016 from pdhx01appd05.hostname.com
3. run again adop phase=prepare on primary node

adop phase=prepare failed.
WARNING:     Exception occurred: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
Error occurred while executing "perl /u11/home/pdhint2/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
[UNEXPECTED]Error while runPendingConfigClone sub-routine is called
cd $INST_TOP/admin/log/clone
view FSCloneApplyAppsTier_01082343.log
ERROR: Could not clone the Run context file due to the following exception:
null
ERROR: Failed to clone Run Context file to refresh Patch context file
 adConfigChanges.log
 view adzdshowts.out
 view adzdshowlog.out
 view adzdshowlog_cleanup.out
adopscanlog -latest=yes
view txkADOPValidations.error
view txkADOPValidations.log
view remote_execution_result_20160106_102803.xml
cd TXK_EVAL_prepare_Wed_Jan_6_10_30_04_2016
view CtxSynchronizer.log
view txkADOPEvalSrvStatus_Wed_Jan_6_10_30_04_2016.log
view txkADOPEvalSrvStatus_Wed_Jan_6_10_31_43_2016.log
diff /u11/home/pdhint2/fs1/inst/apps/pdhint2_pdhx01appd07/appl/admin/pdhint2_pdhx01appd07.xml_8jan16 /u11/home/pdhint2/fs1/inst/apps/pdhint2_pdhx01appd07/appl/admin/pdhint2_pdhx01appd07.xml
ls -lrth /u11/home/pdhint2/fs1/EBSapps/appl/ad/12.0.0/admin/template/adxmlctx.tmp
ls -lrth /u11/home/pdhint2/fs2/EBSapps/appl/ad/12.0.0/admin/template/adxmlctx.tmp
view /u11/home/pdhint2/fs2/EBSapps/appl/ad/12.0.0/admin/template/adxmlctx.tmp
ls -lrth /u11/home/pdhint2/fs2/EBSapps/appl/ad/12.0.0/admin/template/adxmlctx.tmp*
diff /u11/home/pdhint2/fs1/EBSapps/appl/ad/12.0.0/admin/template/adxmlctx.tmp /u11/home/pdhint2/fs1/EBSapps/appl/ad/12.0.0/admin/template/custom/adxmlctx.tmp
ls -lrth $AD_TOP/admin/template/custom/adxmlctx.tmp
grep -i Header /u11/home/pdhint2/fs1/EBSapps/appl/ad/12.0.0/admin/template/adxmlctx.tmp
/u11/home/pdhint2/fs1/inst/apps/pdhint2_pdhx01appd07/admin/log/
cd /u11/home/pdhint2/fs1/inst/apps/pdhint2_pdhx01appd07/admin/log/clone/FSCloneApplyAppsTier_01071727.log
[applpdhi@pdhx01appd07 clone]$ grep -i Header /u11/home/pdhint2/fs1/EBSapps/appl/ad/12.0.0/admin/template/custom/adxmlctx.tmp
<!-- $Header: adxmlctx.tmp 120.401.12020000.47 2014/11/27 09:06:56 jmajumde ship $ -->
Solution:
Issue was with missing custom entries in XML file.
[applpdhi@pdhx01appd07 ~]$ diff /u11/home/pdhint2/fs1/inst/apps/pdhint2_pdhx01appd07/appl/admin/pdhint2_pdhx01appd07.xml_8jan16 /u11/home/pdhint2/fs1/inst/apps/pdhint2_pdhx01appd07/appl/admin/pdhint2_pdhx01appd07.xml
>       <c_bio oa_var="c_bio" scope="CUSTOM" oa_type="PROD_TOP" oa_enabled="FALSE">/u11/home/pdhint2/fs1/EBSapps/appl/bio/12.0.0</c_bio>
>       <c_bio_au oa_var="c_bio_au" scope="CUSTOM" oa_type="PROD_TOP" oa_enabled="FALSE">/u11/home/pdhint2/fs1/EBSapps/appl/bio_au/12.0.0</c_bio_au>
>       <c_bio_ego oa_var="c_bio_ego" scope="CUSTOM" oa_type="PROD_TOP" oa_enabled="FALSE">/u11/home/pdhint2/fs1/EBSapps/appl/bio_ego/12.0.0</c_bio_ego>
>       <c_bio_fnd oa_var="c_bio_fnd" scope="CUSTOM" oa_type="PROD_TOP" oa_enabled="FALSE">/u11/home/pdhint2/fs1/EBSapps/appl/bio_fnd/12.0.0</c_bio_fnd>

Please open the below Load Balancer ticket from VSM, below are the details for EBSPRD/EBSUAT.
Load balancer EBSPRD VIP : ebsprd.apdba.com:
• Redirect http to https      
http://ebsprd.apdba.com redirect to https://ebsprd.apdba.com
• https traffic route
https://ebsprd.apdba.com send traffic to below SSL pass through
https://apdbablog01.apdba.com:4443
https://apdbablog02.apdba.com:4443
https://apdbablog03.apdba.com:4443
https://apdbablog04.apdba.com:4443
• Note: (Above port 4443 is derived from port pool 00.
If this changes please change this to "s_webssl_port" value from the context file)
Load balancer EBSUAT VIP : ebsuat.apdba.com:
• Redirect http to https      
http://ebsuat.apdba.com redirect to https://ebsuat.apdba.com
• https traffic route
https://ebsuat.apdba.com send traffic to below SSL pass through
https://apdbaappd02.apdba.com:4443
https://apdbaappd07.apdba.com:4443
https://apdbaappd08.apdba.com:4443
https://apdbaappd09.apdba.com:4443
• Note: (Above port 4443 is derived from port pool 00.
If this changes please change this to "s_webssl_port" value from the context file)
Copy the wallet/jks certificates
Source the RUN file system
. /<u11/home/ebsprd>/EBSapps.env run
cat $CONTEXT_FILE|grep s_web_ssl_directory
cd <s_web_ssl_directory>/
Example :
cd  /u11/home/ebsprd/fs_ne/inst/ebsprd_apdbablog01/certs
mv Apache Apache.orig
cp -r /patch/erp/SCALE/OHS_SSO/ebscerts Apache
Do the Same on other nodes, Execute 1 & 2 on all the nodes
Validate the certificates, export the below PATH
. /<u11/home/ebsprd>/EBSapps.env run
export PATH=$FMW_HOME/webtier/bin:$FMW_HOME/oracle_common/bin:$PATH
Source the below ENV
. $EBS_DOMAIN_HOME/bin/setDomainEnv.sh
cat $CONTEXT_FILE|grep s_web_ssl_directory
cd <s_web_ssl_directory>/Apache/
Example :
cd  </u11/home/ebsprd/fs_ne/inst/ebsprd_apdbablog01/certs/Apache>
java utils.ValidateCertChain -jks bgwildccert ebsidentity.jks
orapki wallet display -wallet . -pwd bg321456
Update b64InternetCertificate.txt
. /<u11/home/ebsprd>/EBSapps.env run
Backup the file
cp $ORACLE_HOME/sysman/config/b64InternetCertificate.txt $ORACLE_HOME/sysman/config/b64InternetCertificate.txt.orig
ll $ORACLE_HOME/sysman/config/b64InternetCertificate.txt*
Copy the certs to b64InternetCertificate.txt
cat $CONTEXT_FILE|grep s_web_ssl_directory
cd <s_web_ssl_directory>/Apache/
Example :
cd </u11/home/ebsprd/fs_ne/inst/ebsprd_apdbablog01/certs/Apache>
cat RootCA.cer >> $ORACLE_HOME/sysman/config/b64InternetCertificate.txt
cat RootIntCA.cer >> $ORACLE_HOME/sysman/config/b64InternetCertificate.txt
cat BIIBWildcard.cer >> $ORACLE_HOME/sysman/config/b64InternetCertificate.txt
Modify HTTP wallet on all the nodes
. /<u11/home/ebsprd>/EBSapps.env run
Node # 1
cat $CONTEXT_FILE|grep s_ohs_instance_loc
cd $IAS_ORACLE_HOME/instances/<EBS_web_pdhdev_OHS1>/config/OHS/<EBS_web_pdhdev>/keystores/
Example :
cd </u11/home/ebsprd/fs2/FMW_Home/webtier/instances/EBS_web_pdhdev_OHS/config/OHS/EBS_web_pdhdev/keystores>
mv default default.orig
cp -r /patch/erp/SCALE/OHS_SSO/ebscerts  default
Do the Same on Node # 2 /3
Do the Same on other nodes, Execute on all the nodes
Modify HTTP OPMN wallet
. /<u11/home/ebsprd>/EBSapps.env run
Node # 1
cat $CONTEXT_FILE|grep s_ohs_instance_loc
Example :
</u11/home/ebsprd/fs2/FMW_Home/webtier/instances/EBS_web_pdhint2_OHS1>
cd $IAS_ORACLE_HOME/instances/<EBS_web_pdhdev_OHS1>/config/OPMN/opmn>
Example:
cd </u11/home/ebsprd/fs2/FMW_Home/webtier/instances/EBS_web_pdhdev_OHS1/config/OPMN/opmn>
mv wallet wallet.orig
cp -r /patch/erp/SCALE/OHS_SSO/ebscerts  wallet
Do the Same on Node # 2 /3
Do the Same on other nodes, Execute on all the nodes
Modify Fusion Middleware Control Console
backup directory in case you wish to use them again in the future
. /<u11/home/ebsprd>/EBSapps.env run
mv $EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/<s_ohs_component>/wallet $EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/<s_ohs_component>/wallet.orig
Node # 1
Example : mv $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS1>/<EBS_web_pdhint2>/wallet $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS1>/<EBS_web_pdhint2>/wallet.orig
Node # 2
Example : mv $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS2>/<EBS_web_pdhint2>/wallet $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS2>/<EBS_web_pdhint2>/wallet.orig
mv  $EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/wallet $EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/wallet.orig
Node # 1
Example : mv  $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS1>/wallet $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS1>/wallet.orig
Node # 2
Example : mv  $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS2>/wallet $EBS_DOMAIN_HOME/opmn/<EBS_web_pdhint2_OHS2>/wallet.orig
mv $FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_component>/proxy-wallet
$FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_component>/proxy-wallet.orig
Node # 1
Example :  mv $FMW_HOME/webtier/instances/<EBS_web_pdhint2_OHS1>/config/OHS/<EBS_web_pdhint2>/proxy-wallet
$FMW_HOME/webtier/instances/<EBS_web_pdhint2_OHS1>/config/OHS/<EBS_web_pdhint2>/proxy-wallet.orig
Node # 2
Example : mv $FMW_HOME/webtier/instances/<EBS_web_pdhint2_OHS2>/config/OHS/<EBS_web_pdhint2>/proxy-wallet
$FMW_HOME/webtier/instances/<EBS_web_pdhint2_OHS2>/config/OHS/<EBS_web_pdhint2>/proxy-wallet.orig
Do the Same on other nodes, Execute on all the nodes
Copy the certificates to wallet locations
. /<u11/home/ebsprd>/EBSapps.env run
cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/<s_ohs_component>/wallet
Node # 1
Example : cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $EBS_DOMAIN_HOME/opmn/EBS_web_pdhint2_OHS1/EBS_web_pdhint2/wallet
Node # 2
Example : cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $EBS_DOMAIN_HOME/opmn/EBS_web_pdhint2_OHS2/EBS_web_pdhint2/wallet
cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/wallet
Node # 1
Example : cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $EBS_DOMAIN_HOME/opmn/EBS_web_pdhint2_OHS1/wallet
Node # 2
Example : cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $EBS_DOMAIN_HOME/opmn/EBS_web_pdhint2_OHS2/wallet
cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_component>/proxy-wallet
Node # 1
Example : cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $FMW_HOME/webtier/instances/EBS_web_pdhint2_OHS1/config/OHS/EBS_web_pdhint2/proxy-wallet
Node # 2
Example : cp -r /patch/erp/SCALE/OHS_SSO/ebscerts $FMW_HOME/webtier/instances/EBS_web_pdhint2_OHS2/config/OHS/EBS_web_pdhint2/proxy-wallet
Update the JDK cacerts file
. /<u11/home/ebsprd>/EBSapps.env run
cat $CONTEXT_FILE|grep s_fmw_jdk
cd <s_fmw_jdktop>/jre/lib/security
eg: cd /u11/home/ebsprd/fs2/EBSapps/comn/util/jdk64/jre/lib/security
cp  cacerts cacerts.orig
chmod u+w cacerts
cp /patch/erp/SCALE/OHS_SSO/ebscerts/RootCA.cer .
cp /patch/erp/SCALE/OHS_SSO/ebscerts/RootIntCA.cer .
cp /patch/erp/SCALE/OHS_SSO/ebscerts/BIIBWildcard.cer .
keytool -import -alias rootcaebs  -file RootCA.cer  -keystore cacerts -storepass changeit
keytool -import -alias rootintcaebs -file RootIntCA.cer -keystore cacerts -storepass changeit
keytool -import -alias BIIBWildcard -file BIIBWildcard.cer -keystore cacerts -storepass changeit
chmod u-w cacerts
32 bit JDK
cd $OA_JRE_TOP/lib/security
chmod u+w cacerts
cp /patch/erp/SCALE/OHS_SSO/ebscerts/RootCA.cer .
cp /patch/erp/SCALE/OHS_SSO/ebscerts/RootIntCA.cer .
cp /patch/erp/SCALE/OHS_SSO/ebscerts/BIIBWildcard.cer .
keytool -import -alias rootcaebs  -file RootCA.cer  -keystore cacerts -storepass changeit
keytool -import -alias rootintcaebs -file RootIntCA.cer -keystore cacerts -storepass changeit
keytool -import -alias BIIBWildcard -file BIIBWildcard.cer -keystore cacerts -storepass changeit
chmod u-w cacerts
Update the WLS JKS File
. /<u11/home/ebsprd>/EBSapps.env run
cd $FMW_HOME/wlserver_10.3/server/lib/
cp DemoTrust.jks DemoTrust.jks.orig
cp DemoIdentity.jks DemoIdentity.jks.orig
cp cacerts cacerts.orig
cp /patch/erp/SCALE/OHS_SSO/ebscerts/RootCA.cer .
cp /patch/erp/SCALE/OHS_SSO/ebscerts/RootIntCA.cer .
cp /patch/erp/SCALE/OHS_SSO/ebscerts/BIIBWildcard.cer .
keytool -import -alias rootcaebs  -file RootCA.cer  -keystore DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase
keytool -import -alias rootintcaebs -file RootIntCA.cer -keystore DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase
keytool -import -alias BIIBWildcard -file BIIBWildcard.cer -keystore DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase
keytool -import -alias rootcaebs  -file RootCA.cer  -keystore DemoTrust.jks -storepass DemoTrustKeyStorePassPhrase
keytool -import -alias rootintcaebs -file RootIntCA.cer -keystore DemoTrust.jks -storepass DemoTrustKeyStorePassPhrase
keytool -import -alias BIIBWildcard -file BIIBWildcard.cer -keystore DemoTrust.jks -storepass DemoTrustKeyStorePassPhrase
keytool -import -alias rootcaebs  -file RootCA.cer  -keystore cacerts -storepass changeit
keytool -import -alias rootintcaebs -file RootIntCA.cer -keystore cacerts -storepass changeit
keytool -import -alias BIIBWildcard -file BIIBWildcard.cer -keystore cacerts -storepass changeit
Validate/perform the below related steps on all nodes
. /<u11/home/ebsprd>/EBSapps.env run
cat $CONTEXT_FILE|grep s_web_ssl_directory
$IAS_ORACLE_HOME/instances/<EBS_web_pdhdev_OHS1>/config/OHS/<EBS_web_pdhdev>/keystores/default
$IAS_ORACLE_HOME/instances/<EBS_web_pdhdev_OHS1>/config/OPMN/opmn/wallet
$EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/<s_ohs_component>/wallet
$EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/wallet
$FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_component>/proxy-wallet
Validate/perform/Redo the steps for Instance location on all nodes
cat $CONTEXT_FILE|grep s_web_ssl_directory
$IAS_ORACLE_HOME/instances/<EBS_web_pdhdev_OHS1>/config/OHS/<EBS_web_pdhdev>/keystores/default
$IAS_ORACLE_HOME/instances/<EBS_web_pdhdev_OHS1>/config/OPMN/opmn/wallet
$EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/<s_ohs_component>/wallet
$EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/wallet
$FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_component>/proxy-wallet
Update the Context File and Config Files
You need to validate your SSL.conf value for the SSL port used
Example :
cat  /u11/home/ebsprd/fs2/FMW_Home/webtier/instances/EBS_web_pdhint2_OHS1/config/OHS/EBS_web_pdhint2/ssl.conf|grep Listen
. /<u11/home/ebsprd>/EBSapps.env run
cp $CONTEXT_FILE $CONTEXT_FILE.b4ssl
Edit the context file
Depending on the port pool & Load balancer entry update the below
Context Variables Vales
s_url_protocol  https
s_local_url_protocol   https
s_webentryurlprotocol   https
s_webssl_port     4503  ( dervied from Port Pool)
s_https_listen_parameter 4503  ( dervied from Port Pool)
s_active_webport        443
s_webentryhost    <ebspatdev>( Load balancer Value)
s_webentrydomain         apdba.com
s_enable_sslterminator   #
s_login_page      https://<ebspatdev>.apdba.com/OA_HTML/AppsLogin
s_external_url  https://<ebspatdev>.apdba.com
Sync the CONTEXT_FILE
. /<u11/home/ebsprd>/EBSapps.env run
perl $AD_TOP/bin/adSyncContext.pl contextfile=$CONTEXT_FILE
Enter the APPS user password:
Enter the WebLogic AdminServer password:
Add a redirect rule
. /<u11/home/ebsprd>/EBSapps.env run
Node # 1
cat $CONTEXT_FILE|grep s_ohs_instance_loc
Example:
cd /u11/home/ebsprd/fs2/FMW_Home/webtier/instances/<EBS_web_pdhint2_OHS1>/config/OHS/<EBS_web_pdhint2>
Edit custom.conf, Update the below values
Add the Load balanced https url below
vi  custom.conf
RewriteCond %<HTTPS> !=on
RewriteRule ^/(.*) https://<pdhint2>.apdba.com/$1 [R,L]
Stop application All the  nodes
Run adautocfg.sh on All the nodes
. /<u11/home/ebsprd>/EBSapps.env run
cd $ADMIN_SCRIPTS_HOME
adautocfg.sh
Restart the Application Tier Services/ Start application after regenerating jar
ADD the above modified file in adop_sync.drv
Backup the file
. /<u11/home/ebsprd>/EBSapps.env run
cp $APPL_TOP_NE/ad/custom/adop_sync.drv  $APPL_TOP_NE/ad/custom/adop_sync.drv.bssl
Edit the adop_sync.drv, add the below entries
vi $APPL_TOP_NE/ad/custom/adop_sync.drv
#SSL SECTION - START
## Required for SSL setup migration from RUN to PATCH file-system.
## Please alter the commands in the event that rsync is not available or the platform does not support the example syntax.
#
##10.1.2 b64InternetCertificate.txt
rsync -zr %s_current_base%/EBSapps/10.1.2/sysman/config/b64InternetCertificate.txt %s_other_base%/EBSapps/10.1.2/sysman/config/b64InternetCertificate.txt
#
##Oracle HTTP Server Wallet - cwallet.sso
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default
#
##OPMN Wallet - cwallet.sso
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet
#
##Fusion Middleware Control Wallets - cwallet.sso
rsync -zr %s_current_base%/FMW_Home/user_projects/domains/EBS_domain_%s_dbSid%/opmn/%s_ohs_instance%/%s_ohs_component%/wallet %s_other_base%/FMW_Home/user_projects/domains/EBS_domain_%s_dbSid%/opmn/%s_ohs_instance%/%s_ohs_component%/wallet
#
rsync -zr %s_current_base%/FMW_Home/user_projects/domains/EBS_domain_%s_dbSid%/opmn/%s_ohs_instance%/wallet %s_other_base%/FMW_Home/user_projects/domains/EBS_domain_%s_dbSid%/opmn/%s_ohs_instance%/wallet
#
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet
#WLS JKS
rsync -zr %s_current_base%/FMW_Home/wlserver_10.3/server/lib/*.jks %s_other_base%/FMW_Home/wlserver_10.3/server/lib/*.jks
#JDK keystore
rsync -zr --include=jdk* --include=jdk*/jre --include=jdk*/jre/lib --include=jdk*/jre/lib/security --include=cacerts --exclude=* %s_current_base%/EBSapps/comn/util/ %s_other_base%/EBSapps/comn/util/
#SSL SECTION – END
Regenerate JAR files
Stop application All the  nodes
adadmin
Chose > Generate Applications Files menu
Chose > Generate product JAR files
Exit
Restart the Application Tier Services
Load wallet into the Database node
Log into all the DB nodes as oracle user
cd $ORACLE_HOME/appsutil
mv wallet wallet.old
cp -r /erp_admin/app/ssl/ebscerts wallet
 Validate the SSL EBS URL
https://ebsprd.apdba.com
SSO Steps
Remove EBS Source AccessGate ( Required for cloned env)
Start the WebLogic Administration Console on the primary node
Execute the below command
Start the Admin console on run system
EBSapps.env run
cd $ADMIN_SCRIPTS_HOME
./adadminsrvctl.sh start
Start the Admin console on patch system
EBSapps.env patch
cd $ADMIN_SCRIPTS_HOME
./adadminsrvctl.sh start forcepatchfs
Delete the Deployed EBS AccessGate
In the WebLogic Administration Console on the run file system,
Navigate to EBS_domain_sid > Deployments,
Stop the Oracle E-Business Suite AccessGate application named "accessgate".
Then delete the Oracle E-Business Suite AccessGate application named "accessgate".
In the WebLogic Administration Console on the patch file system,
Navigate to EBS_domain_sid > Deployments.
Stop the Oracle E-Business Suite AccessGate application named "accessgate".
Then delete the Oracle E-Business Suite AccessGate application named "accessgate".
STOP the PATCH console
cd $ADMIN_SCRIPTS_HOME
./adadminsrvctl.sh start forcepatchfs
De-register the EBS instance from OID for TEST to TEST clone.
Note:
*** : DO NOT execute this step   without the APPNAME parameter
cd <EBS_BASE_HOME>
Source RUN   EBSapps.env run
$FND_TOP/bin/txkrun.pl  \
-script=SetSSOReg  \
-deregisteroid=yes  \
-ldapport=3060  \
-ldaphost=ssooiddev.apdba.com  \
-oidadminuserpass=bg321456   \
-appspass=appss2015  \
-appname=<ebsint1>
Enter the valid
apps password
orcladmin password
appname/ svcname to $TWO_TASK
Note **: Please ignore any error proceed to next step
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
echo $CONTEXT_FILE
Backup  the  echo $CONTEXT_FILE
cp < CONTEXT_FILE path >  <backup location path>
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
$FND_TOP/bin/txkrun.pl   \
-script=SetSSOReg  \
-removereferences=Yes \
-appspass=appss2015
Enter the apps password when prompted
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
sqlplus apps/pass@["two_task]
delete from fnd_user_preferences WHERE user_name='#INTERNAL' and module_name='LDAP_SYNCH';
delete from fnd_user_preferences WHERE user_name='#INTERNAL' and module_name='OID_CONF';
commit;
exit;
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
 echo $FILE_EDITION
Substitute right apps/instance password below
txkrun.pl -script=SetSSOReg \
-registeroid=yes \
-ldapport=3060  \
-ldaphost=ssooiddev.apdba.com  \
-oidadminuser=cn=orcladmin  \
-oidadminuserpass=bg321456   \
-instpass=appss2015  \
-appspass=appss2015  \
-provisiontype=3
Validate the registration
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
 echo $FILE_EDITION
sqlplus apps/pass@["two_task]
execute fnd_oid_plug.setPlugin;
select fnd_sso_registration.getdefaultrealm from dual;
exit;
Ensure this patch is already applied
R12.TXK.C  Patch  # 19767816
Validate if the patch is applied
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
echo $FILE_EDITION
commands to apply this patch
select * from ad_appl_tops;
Use the appl_top_id (85) for the current NAME (ebspat) in the below query
select bug_number from ad_bugs where bug_number='19767816';
SQL> select appl_top_id from ad_appl_tops where name='<ebsint1>';
APPL_TOP_ID
-----------
        104
select ad_patch.is_patch_applied('R12',104,19767816) from dual;
select ad_patch.is_patch_applied('R12',104,19197270) from dual;
select ad_patch.is_patch_applied('R12',104,19330775) from dual;
All are applied in ebsint1
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
echo $FILE_EDITION
Run the below command to install
 Validate the webgate install
cd $FMW_HOME
ls -lrt Oracle_OAMWebGate1
If Oracle_OAMWebGate1 (webgate) exists  SKIP this step
If WebGate not found Install:
txkrun.pl -script=SetOAMReg  \
-installWebgate=yes  \
-webgatestagedir=/patch/erp/SCALE/SSO
Log into the EBS middle tier with applmgr, execute these steps  only on Internal WEB nodes
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
sqlplus apps/pass@["two_task]
show parameter %UTL%
exit
Make sure utl_file_dir is present
echo $FILE_EDITION
The script will prompt for the following passwords:
Enter the APPS Schema password.
Enter the WebLogic AdminServer password.
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
sqlplus apps/pass@["two_task]
show parameter %UTL%
exit
Make sure utl_file_dir is present
Execute these steps  only on all Internal WEB nodes
echo $FILE_EDITION
The script will prompt for the following passwords:
Enter the APPS Schema password.
Enter the WebLogic AdminServer password.
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-create-oaea_resources  \
-contextfile=$CONTEXT_FILE  \
-deployApps=accessgate  \
-managedsrvname=oaea_server1 \
-managedsrvport=6801 \
-SSOServerURL=https://ssoimdev.apdba.com  \
-logfile=/tmp/deployeag_dep.log
Based on the port pool replace the value for managedsrvport parameter
For Port Pool 00 the value of  managedsrvport=6801 replace accordingly
For Port Pool 10 the value of  managedsrvport=6811 replace accordingly
On Additional nodes, please add/change the parameter “managedsrvname”. For additional nodes add the parameter  & use the same port
-managedsrvname=oaea_server(n) \
-managedsrvport=6801 \
Execute the following on the second node # 2 :
source EBSapps.env run
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-create-oaea_resources  \
-managedsrvname=oaea_server2 \
-managedsrvport=6801 \
-contextfile=$CONTEXT_FILE  \
-deployApps=accessgate  \
-SSOServerURL=https://ssoimdev.apdba.com  \
-logfile=/tmp/deployeag_dep.log
Execute the following on the second node # 3 :
Based on the port pool replace the value for managedsrvport parameter
For Port Pool 00 the value of  managedsrvport=6801 replace accordingly
For Port Pool 10 the value of  managedsrvport=6811 replace accordingly
On Additional nodes, please add/change the parameter “managedsrvname”. For additional nodes add the parameter  & use the same port
-managedsrvname=oaea_server(n) \
-managedsrvport=6801 \
source EBSapps.env run
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-create-oaea_resources  \
-managedsrvname=oaea_server3 \
-managedsrvport=6801 \
-contextfile=$CONTEXT_FILE  \
-deployApps=accessgate  \
-SSOServerURL=https://ssoimdev.apdba.com  \
-logfile=/tmp/deployeag_dep.log
Execute the following on the second node # 4 :
Based on the port pool replace the value for managedsrvport parameter
For Port Pool 00 the value of  managedsrvport=6801 replace accordingly
For Port Pool 10 the value of  managedsrvport=6811 replace accordingly
On Additional nodes, please add/change the parameter “managedsrvname”. For additional nodes add the parameter  & use the same port
-managedsrvname=oaea_server(n) \
-managedsrvport=6801 \
source EBSapps.env run
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
ebs-create-oaea_resources  \
-managedsrvname=oaea_server4 \
-managedsrvport=6801 \
-contextfile=$CONTEXT_FILE  \
-deployApps=accessgate  \
-SSOServerURL=https://ssoimdev.apdba.com  \
-logfile=/tmp/deployeag_dep.log
Execute these steps  only on all Internal WEB nodes
 ( Change the hostname to server name & port to the OAEA server port used above)
perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-configoption=addMS \
-accessgate=<host>.<domain>:<port>
Example :
Login to web node apdbaappd02
. EBSapps.env run
perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-configoption=addMS \
-accessgate=apdbaappd02.apdba.com:6811
Login to web node apdbaappd07
. EBSapps.env run
perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-configoption=addMS \
-accessgate=apdbaappd07.apdba.com:6811
Login to web node apdbaappd08
. EBSapps.env run
perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-configoption=addMS \
-accessgate=apdbaappd08.apdba.com:6811
Login to web node apdbaappd09
. EBSapps.env run
perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-configoption=addMS \
-accessgate=apdbaappd09.apdba.com:6811
Validate the mod_ohs_wl.conf for correct entry on all the web nodes.
. EBSapps.env run
cat $CONTEXT_FILE|grep ohs_instance_loc
cd cd /u17/home/ebsuat/fs2/FMW_Home/webtier/instances/EBS_web_ebsuat_OHS1/config/OHS/EBS_web_ebsuat/
validate mod_wl_ohs.conf for
<Location /accessgate>
SetHandler weblogic-handler
WebLogicCluster apdbaappd02.apdba.com:6801
WLTempDir ${ORACLE_INSTANCE}/tmp
</Location>
Make sure pointing to active internal web node only.
Login into Weblogic console of RUN file system
Example:
http://<apdbaappd07>.apdba.com:7071/console/login/LoginForm.jsp
Increase the Data Source parameters
Login to weblogic console as admin
Go to the EBS weblogic  Console Tree panel on the left hand side and
Navigate to
EBS_domain_sid > Services > Data Sources, and verify that a data source "OAEADatasource" is available
Select ""Services"-->JDBC-->Datasources---> OAEADatasource >Connection pool > Advanced )
Select >  Lock and Edit
Modify
• Test Connections On Reserve - Check the box
• Inactive Connection Timeout to 10 seconds
Save  & Activate the changes
Verify successful deployment
logon to EBS WebLogic Administration Console on RUN, for example:
http://ebshost.example.com:7001/console
In the WebLogic Administration Console, navigate to EBS_domain_sid > Environment > Servers, and verify that a managed server "oaea_server1" is available.
Navigate to EBS_domain_sid > Deployments, and verify that the Oracle E-Business Suite AccessGate application named "accessgate" is deployed.
Register EBS with OAM
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
echo $FILE_EDITION
Execute the following command to register Oracle E-Business Suite with Oracle Access Manager. The below command needs to be executed on all the nodes.
Execute these steps  only on all Internal WEB nodes
txkrun.pl -script=SetOAMReg  \
-registeroam=yes  \
-oamHost=http://ssoadmdev.apdba.com:7003 \
-oamUserName=oamadmin  \
-ldapUrl=ldap://ssooiddev.apdba.com:3060  \
-oidUserName=cn=orcladmin  \
-skipConfirm=yes  \
-ldapSearchBase="ou=all users,dc=corp,dc=apdba,dc=com"  \
-ldapGroupSearchBase="cn=Groups, dc=corp,dc=apdba,dc=com" \
-authScheme=KerberosScheme \
-authSchemeMode=reference
Execute these steps  only on all Internal WEB nodes
Enable EBS System Profile Options for SSO
1. Login to EBS instance as user with system administrator
2. System administrator - Profiles - system
3. Query for the below profile and make changes on the site level
4. Change following profile options:
Application SSO LDAP SYNC  Enabled
Application SSO Auto Link Enabled
Applications SSO Enable OID Identity Add Event Disabled
Application SSO Login Types   Both
Application SSO Type   SSWA W/SSO
Application Logout URL http://inet.apdbaidec.com
Applications SSO User Creation and Updation Allowed Disabled
Configure Secure Sockets Layer (SSL
If EBS is on SSL , if not ignore this step
Then
When using WebLogic Server Release 10.3.4 and above and enabling SSL,  this is generic EBS SSL steps, we need to able to extend this to AccessGate server also
Ensure that the following are enabled in the WebLogic Server Administration Console:
WebLogic Plug-In
Client Cert Proxy
To verify this:
Navigate to 'Environments' > 'Servers' > 'AdminServer(Admin)'
Access the 'General' tab
Expand the 'Advanced' section and check the checkboxes for:
WebLogic Plug-In Enabled
Client Cert Proxy Enabled
Navigate to 'Environments' > 'Servers' > oaea_server’s'
Access the 'General' tab
Expand the 'Advanced' section and check the checkboxes for:
WebLogic Plug-In Enabled
Client Cert Proxy Enabled
Restart all the Managed servers once
cd $ADMIN_SCRIPTS_HOME
adstpall.sh
adstrtal.sh
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
echo $FILE_EDITION
sqlplus apps/pass@["two_task]
• Run to clear USER_GUID from FND_USER table ( NOT FOR BULK updates)
Below command is for individual user , for BULK update follow the next step
sqlplus apps @$FND_TOP/patch/115/sql/fndssouu.sql <USER_NAME>
• Backup the tabled before BULK changes
1. Login to EBS database as apps user
2. Run the below command to back fnd_user and fnd_oracle_userid table
create table fnd_user_backup as select * from fnd_user;
create table FND_ORACLE_USERID_backup as select * from FND_ORACLE_USERID;
select query to cross check the user count is same on the base table and in backup table
select count(*) from  fnd_user;
select count(*) from  fnd_user_backup;
select count(*) from  FND_ORACLE_USERID;
select count(*) from  FND_ORACLE_USERID_backup;
Update the USER_GUID to NULL  for BULK change
Log into the EBS middle tier with applmgr
cd <EBS_BASE_HOME>
Execute the below command
EBSapps.env run
echo $FILE_EDITION
sqlplus apps/pass@["two_task]
update fnd_user set USER_GUID=null;
commit;
exit
Validate SSO and SSL configuration and release the instance
Validate SSO URL login
http://<hostname.domain.com:port>
Perform the post clone step for Workflow Status Diagram Not Showing Up in R12.2 (Doc ID 2024395.1)
• Bring up the WLS Admin console for the EBS instance ->
http://host:port/console and login as the weblogic user.
• Click on Lock & Edit
• Under the Domain Structure, navigate to EBS Domain -> Environment -> Servers.
• Click on all oacore_server1/oacore_server2/oacore_server3/oacore_server4, and click on the SSL tab.
• Click on Advanced to expand that section.
• Look for the 'Use JSSE SSL' parameter and enable this by placing a check in the box next to it.
• Click on Activate Changes.
• Restart the managed server...
#Perform fs_clone
EBSapps.env run
Sync context on all nodes
perl $AD_TOP/bin/adSyncContext.pl contextfile=$CONTEXT_FILE
echo $FILE_EDITION
Run the fs_clone force=yes
adop phase=fs_clone force=yes
End of Test Steps

Cloning Reference:-
Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1) – For clone procedure
Sharing the Application Tier File System in Oracle E-Business Suite Release 12.2 (Doc ID 1375769.1) – For Shared Appl top procedure
Issues Refrence:
http://balaoracledba.com/category/oracleappsr12/issues/ - For known issues and support to oracle docs
http://oradbapp.blogspot.in/2015/05/ebs-122-unable-to-start-up-managed.html- For a known issue in 12.2.4
12.2: error while fs_clone of patching(WLS 17495356) cycle (Doc ID 1634239.1)
12.2: fs_clone fails with "ERROR: Number of servers are not in sync between Run and Patch Context" (Doc ID 1582473.1)
Fs_clone Fails with Status 2 (Doc ID 1960769.1)
FS Clone Fails Saying Port In Use: Free The Listed Ports And Retry The Adop Operation (Doc ID 1901109.1)
Clone Oracle Applications Environment
This section explains the detailed steps to clone an Oracle applications 12i (12.2.4) installation using a 11gR2 (11.2.0.4) database to another Oracle 12i (12.2.4) 11gR2(11.2.0.4) environment.
The main steps for cloning Oracle Applications Environment are:
Pre Clone Steps.
Clone Database Tier.
Clone Application Tier.
Post Clone Steps.
Pre Clone Steps
On Source Instance Please perform below activities.
Nodes Task Description Command
Check if instance can be cloned This can be done anytime before the clone
App Source Environment and choose RUN  . /u11/home/ebsdev/EBSapps.env
App Find master node if multi-node system
Master node is RUN FS for instance ls -ltr $NE_BASE/EBSapps/log/status*/adzdshowstatus.out | tail -n 1
check for master node in the file
App Check ADOP status complete on master-node CLEANUP         COMPLETED
Or
PREPARE         NOT STARTED “adop -status” should show all completed,
otherwise use
“adop phase=finalize/cutover/cleanup/abort/fs_clone” to complete
Run Preclone on App-tier and copy This can be done anytime before the clone after ADOP cleanup
App On Master_node RUN FS
Run adpreclone.pl appsTier Following files exist in $RUN_BASE/ /EBSapps/comn/clone/FMW
   FMW_Home.jar
   WLS/EBSdomain.jar
   WLS/plan/moveplan.xml
   OHS/ohsarchive.jar
   OHS/moveplan.xml $INST_TOP/admin/scripts/adadminserver.sh start
$INST_TOP/admin/scripts/adpreclone.pl appsTier
App Copy $RUN_BASE/EBSapps to target Copy the tar/zip file to target instance $RUN_BASE tar & zip the binaries
tar –cf – EBSapps | gzip > EBSapps.tar.gz
DB Run Preclone on DB-tier and copy Subbu
DB Run adpreclone.pl dbTier Subbu $OH/appsutil/scripts/$CTX/adpreclone.pl dbTier
DB Backup DB Subbu
DB Backup RDBMS home Subbu
Clone Database Tier
Clone oracle home & Database
Clone DB & Oracle Homes on target DB tier. This task is performed by DBA team (Subbu)
Enable Auto Config on DB Tier
For RAC node run below steps on both nodes.
NODE 1
1. Create appsutil.zip on source app server
apdbappd01:appldev1$ $ADPERLPRG $AD_TOP/bin/admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at /u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/log/MakeAppsUtil_04131514.log
output located at /u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip
MakeAppsUtil completed successfully.
2. Transfer appsutil.zip file to target DB server
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $ scp mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00
cd $ORACLE_HOME
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $ scp mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $
3. Unzip appsutil.zip file
cd $ORACLE_HOME
unzip -o appsutil.zip
4. Purge Old nodes information
Login as apps user
$ sqlplus apps/apps
SQL> exec fnd_conc_clone.setup_clean
5. Install JRE
Install jre under $ORACLE_HOME/appsutil
cd /bio/ebs/str/db/11.2.0.4/appsutil
cat /ora_nfs/jre/jre-7u11-linux-x64.tar.gz | gzip -d | tar xf - &
mv jre1.7.0_11 jre
6. Create Context file
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ adbldxml.pl appsuser=apps appspass=appss2015
Starting context file generation for db tier..
Using JVM from /bio/ebs/str/db/11.2.0.4/appsutil/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/bio/ebs/str/db/11.2.0.4/appsutil/log/adbldxml_04231351.log
Enter Database Service Name: ebsstr1
Enter the value for Display Variable: apdbadm01:0.0
Context File  /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
 already exists.
Overwriting Context File may cause loss of existing settings, hence
 backing it up as: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml.bak
 The context file has been created at:
/bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Update context file with target db port number
vi /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
update db port from 1521 to 1731 & save the file
Ensure that Util file dir has /usr/tmp
7. Run Auto Config
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ perl adconfig.pl
Enter the full file path to the Context file: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Enter the APPS user password:
The log file for this session is located at: /bio/ebs/str/db/11.2.0.4/appsutil/log/ebsstr1_apdbadm01/04231358/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /bio/ebs/str/db/11.2.0.4
        Classpath                   : :/bio/ebs/str/db/11.2.0.4/jdbc/lib/ojdbc6.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java/xmlparserv2.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java:/bio/ebs/str/db/11.2.0.4/jlib/netcfg.jar:/bio/ebs/str/db/11.2.0.4/jlib/ldapjclnt11.jar
        Using Context file          : /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Context Value Management will now update the Context file
        Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.
NODE 2:
1. Transfer appsutil.zip file to target DB server
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $ scp` mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00
cd $ORACLE_HOME
orapat@apdbadm01:/bio/ebs/pat/db/11.2.0.4 $
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $ scp mpareek@10.2.216.226:/u09/home/ebsdev1/fs1/inst/apps/ebsdev1_apdbappd01/admin/out/appsutil.zip ./
The authenticity of host '10.2.216.226 (10.2.216.226)' can't be established.
RSA key fingerprint is da:3a:e1:bf:98:8a:0c:a5:31:78:d3:2e:06:db:66:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.2.216.226' (RSA) to the list of known hosts.
Password:
appsutil.zip                                                                                                                          100% 3459KB   3.4MB/s   00:00`
orapat@apdbadm02:/bio/ebs/pat/db/11.2.0.4/network/admin/ebspat2_apdbadm02 $
2. Unzip appsutil.zip file
cd $ORACLE_HOME
unzip -o appsutil.zip
3. Purge Old nodes information
Login as apps user
$ sqlplus apps/apps
SQL> exec fnd_conc_clone.setup_clean
4. Install JRE
Install jre under $ORACLE_HOME/appsutil
cd /bio/ebs/str/db/11.2.0.4/appsutil
cat /ora_nfs/jre/jre-7u11-linux-x64.tar.gz | gzip -d | tar xf - &
mv jre1.7.0_11 jre
5. Create Context file
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ adbldxml.pl appsuser=apps appspass=appss2015
Starting context file generation for db tier..
Using JVM from /bio/ebs/str/db/11.2.0.4/appsutil/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/bio/ebs/str/db/11.2.0.4/appsutil/log/adbldxml_04231351.log
Enter Database Service Name: ebsstr1
Enter the value for Display Variable: apdbadm01:0.0
Context File  /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
 already exists.
Overwriting Context File may cause loss of existing settings, hence
 backing it up as: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml.bak
 The context file has been created at:
/bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Update context file with target db port number
vi /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
update db port from 1521 to 1731 & save the file
Ensure that Util file dir has /usr/tmp
6. Run Auto Config
orastr@apdbadm01:/bio/ebs/str/db/11.2.0.4/appsutil/bin $ perl adconfig.pl
Enter the full file path to the Context file: /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Enter the APPS user password:
The log file for this session is located at: /bio/ebs/str/db/11.2.0.4/appsutil/log/ebsstr1_apdbadm01/04231358/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /bio/ebs/str/db/11.2.0.4
        Classpath                   : :/bio/ebs/str/db/11.2.0.4/jdbc/lib/ojdbc6.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java/xmlparserv2.jar:/bio/ebs/str/db/11.2.0.4/appsutil/java:/bio/ebs/str/db/11.2.0.4/jlib/netcfg.jar:/bio/ebs/str/db/11.2.0.4/jlib/ldapjclnt11.jar
        Using Context file          : /bio/ebs/str/db/11.2.0.4/appsutil/ebsstr1_apdbadm01.xml
Context Value Management will now update the Context file
        Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.
Clone Application Tier – Primary node apdbablog01
Create RUN File System
1. Untar/zip EBSApps dump file on target server
                      Copied fs2(Run Base) from source to /u11/home/ebsprd/fs2/
tar -cf - EBSapps | gzip > EBSapps.tar.gz
cp -pr EBSapps.tar.gz /patch/erp/bala/ebsprd_files/
cp -pr /patch/erp/bala/ebsprd_files/EBSapps.tar.gz ./
tar –xvf EBSapps.tar.gz
To determine which file system (fs1 or fs2) is currently the Run Edition File System, run the command:
echo $RUN_BASE
Note: If the Source Run Edition File System is under fs1, you should configure the Target Run Edition File System under fs1 as well.
Similarly, if the Source Run Edition File System is under fs2, you should configure the Target Run Edition File System under fs2.
2. Create RUN file system
cd /u11/home/ebspat/fs1/EBSapps/comn/clone/bin
[applpat@apdbablog03 bin]$ perl adcfgclone.pl appsTier
                     Copyright (c) 2011 Oracle Corporation
                        Redwood Shores, California, USA
                        Oracle E-Business Suite Rapid Clone
                                 Version 12.2
                      adcfgclone Version 120.63.12020000.39
Enter the APPS password :
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper -encryptpwd /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../FMW/tempinfoApps.txt
Enter the Weblogic AdminServer password :
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../FMW/tempinfo.txt
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../FMW/EBSDataSource
Do you want to add a node (yes/no) [no] :
Running:
/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -cp /u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/xmlparserv2.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojdbc6.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs1/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.context.CloneContext -e /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../context/apps/CTXORIG.xml -validate -pairsfile /tmp/adpairsfile_11610.lst -stage /u11/home/ebspat/fs1/EBSapps/comn/clone  2> /tmp/adcfgclone_11610.err; echo $? > /tmp/adcfgclone_11610.res
Log file located at /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/CloneContext_0414180413.log
Target System File Edition type [run] :
Provide the values required for creation of the new APPL_TOP Context file.
Target System Hostname (virtual or normal) [apdbablog03] :
Target System Database SID : ebspat
Target System Database Server Node [apdbablog03] : edp01-scan
Target System Database Domain Name [apdba.com] :
Target System Base Directory : /u11/home/ebsprd
Target System Base Directory set to /u11/home/ebsprd
Target System Current File System Base set to /u11/home/ebspat/fs1
Target System Other File System Base set to /u11/home/ebspat/fs2
Target System Fusion Middleware Home set to /u11/home/ebspat/fs1/FMW_Home
Target System Web Oracle Home set to /u11/home/ebspat/fs1/FMW_Home/webtier
Target System Appl TOP set to /u11/home/ebspat/fs1/EBSapps/appl
Target System COMMON TOP set to /u11/home/ebspat/fs1/EBSapps/comn
Target System Instance Home Directory [/u11/home/ebspat] :
Target System Instance Top set to /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03
Do you want to preserve the Display [apdbappd01:0.0] (y/n)  : n
Target System Display [apdbablog03:0.0] :
Target System Root Service [enabled] :
Target System Web Administration [enabled] :
Target System Web Entry Point Services [enabled] :
Target System Web Application Services [enabled] :
Target System Batch Processing Services [enabled] :
Target System Other Services [disabled] :
Do you want the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 50
Checking the port pool 50
done: Port Pool 50 is free
Report file located at /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/admin/out/portpool.lst
Complete port information available at /u11/home/ebspat/fs1/inst/apps/ebsprd_apdbablog03/admin/out/portpool.lst
RC-40201: Unable to connect to Database ebspat.
Enter the Database listener port [1571] : 1725
UTL_FILE_DIR on database tier consists of the following directories.
1. /bio/ebs/pat/tmp
2. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 2 – select /usr/tmp
Creating the new APPL_TOP Context file from :
  /u11/home/ebspat/fs1/EBSapps/comn/clone/context/apps/adxmlctx.tmp
The new APPL_TOP context file has been created :
  /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml
Log file located at /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/CloneContext_0414180413.log
Check Clone Context logfile /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/CloneContext_0414180413.log for details.
Running Rapid Clone with command:
Running:
perl /u11/home/ebspat/fs1/EBSapps/comn/clone/bin/adclone.pl java=/u11/home/ebspat/fs1/EBSapps/comn/clone/bin/../jre mode=apply stage=/u11/home/ebspat/fs1/EBSapps/comn/clone component=appsTier method=CUSTOM appctxtg=/u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml showProgress contextValidated=true
Faced an Error:-
############################ ALERT ###################################
The oraInst.loc file is not present under /etc. Without this file cloning cannot proceed further
Please run the oraInstFileGenerator.sh located under /tmp with ROOT privileges
This script will create the oraInst.loc file under /etc
ERROR: Failed to execute /u11/home/ebsprd/fs2/EBSapps/comn/clone/bin/adclone.pl
 Please check logfile.
Reason for this error:
This is the first time we are cloning in this machine so we need to run the orainstFileGenerator.sh
Raising a ticket to UNIX team to run this as root user.
Fixed the issue in this server:
/u11/home/oraInventory
Manually create this dir
I renamed the existing oraInventory to oraInventory.obsolotete and mkdir oraInventory to create a new directory. Then restarted the process
Running the adcfgclone.pl again:
Again faced an error:
No write permissions to the Inventory pointed by:
        /etc
Rapid Clone needs to be able to write to the inventory
Please run the script /tmp/orainstRoot.sh as root
Restarted adcfg clone:
Success!!
Issue:3
adcfgclone in Run FS
Enter value for 1: Enter value for 2: Enter value for 3: ERROR:
ORA-12505: TNS:listener does not currently know of SID given in connect
descriptor
Fix:
1) Verified the Autocofig erroed in DB second nodesync
2) Ran the Autoconfig in DB nodes and ran the adcfgclone seems completed successfully
Post Clone Steps
Source ENV and run the cmclean in sqlplus
sqlplus apps/apps_pw
SQL>@cmclean.sql
SQL>commit;
SQL>exit;
Run the script to keep the jobs on hold
sqlplus apps/apps_pw
SQL> create table dba_conc_req_hold as select * from fnd_concurrent_requests;
SQL> update fnd_Concurrent_requests set hold_flag='Y' where PHASE_CODE='P' and hold_flag='N' and request_id in (select request_id from dba_conc_req_hold);
SQL> commit;
SQL>exit;
Change apps & sysadmin password
Source the Run FS
$FNDCPASS apps/appss2015 0 Y system/<target_system_pw> SYSTEM APPLSYS <target_apps_pw>
$FNDCPASS apps/<target_apps_pw> 0 Y system/<target_system_pw> USER   SYSADMIN  <target_sysadmin_pw>
verify the log and make sure there is no errors in FND log
Update wf tables
Source the Run FS
sqlplus apps/apps_pw
SQL>create table wf_notifications20130709 as select * from wf_notifications;
SQL>update wf_notifications set mail_status='SENT' where mail_status is not null;
SQL> Update wf_resources
Set text=’ FND_RESP|FND|FNDWF_ADMIN_WEB|STANDARD’
Where name =’ WF_ADMIN_ROLE’  ;
SQL>commit;
SQL>exit;
Run adcmclean.sql script
Source the Run FS
Configure Load Balancer – Configured by Load Balancer team
Source the Run FS primary node
Update auto config context file and run auto config(Please take a backup of currecnt context file)
<externURL oa_var="s_external_url">http://ebsprd.apdba.com</externURL>
<login_page oa_var="s_login_page">http://epsprd.apdba.com/OA_HTML/AppsLogin</login_page>
<activewebport oa_var="s_active_webport" oa_type="DUP_PORT" base="8000" step="1" range="-1" label="Active Web Port">80</activewebport>
<webentryhost oa_var="s_webentryhost">ebsprd</webentryhost>
<APPL_TOP_NAME oa_var="s_atName">ebsprd</APPL_TOP_NAME>
Cd $INST_TOP/admin/scripts
./adautocfg.sh appspass=<pw>
Start application services
cd $INST_TOP/admin/scr*
./adstrtall.sh apps/<pw>
Starting Application for the first time.
You are running admanagedsrvctl.sh version 120.14.12020000.10
Starting oacore_server1...
admanagedsrvctl.sh: exiting with status 1
admanagedsrvctl.sh: check the logfile /u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog01/logs/appl/admin/log/adoacorectl.txt for more infmation ...
.end std out.
.end err out.
****************************************************
adstrtal.sh: Exiting with status 1
Fix:
http://oradbapp.blogspot.in/2015/05/ebs-122-unable-to-start-up-managed.html
It is observed that Apps Password is changed as part of execution of adcfgclone.pl on DBTier and this new password change is somehow not reflected when managed servers are created during execution of rapidclone on Apps Tier.
Now this is fixed manually by performing below steps.
As ADMIN Server is up and runnning,
1. Login to EBS Weblogic console -> Perform Lock & Edit -> Under Domain Structure -> Services -> Datasources -> Select "EBSDataSource"
2. Navigate to Configuration -> Connection pool  -> Update with new password in password fields.
3. Then Save changes.
4. Perform the above steps of other Datasource OAADatasource.
Fixed:-
Local URL: http://ebsprd.apdba.com/OA_HTML/AppsLocalLogin.jsp
Login & validate
Log into application
Run  a concurrent request and view output
Checked and verified Primary node Run FS application
Create Patch File System
Note:- Do not delete or move the orainventory.xml(central inventory) during patch file system clone
1. Export the environment variablesa as below
export CONFIG_JVM_ARGS="-Xms512m -Xmx1024m"
export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/tmp -Xms512m -Xmx1024m"
export IGNORE_FMW_DISK_WARNING=true
export TIMEDPROCESS_TIMEOUT=-1
2. Run preclone on Run FS and take backup of EBSapps. Move it to Patch files system.
3. Run adcfgclone.pl in Patch FS
cd /u11/home/ebspat/fs2/EBSapps/comn/clone/bin
[applpat@apdbablog03 bin]$ ./adcfgclone.pl appsTier
                     Copyright (c) 2011 Oracle Corporation
                        Redwood Shores, California, USA
                        Oracle E-Business Suite Rapid Clone
                                Version 12.2
                     adcfgclone Version 120.63.12020000.39
Enter the APPS password :
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper -encryptpwd /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../FMW/tempinfoApps.txt
Enter the Weblogic AdminServer password :
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../FMW/tempinfo.txt
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.clone.util.OPWrapper /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../FMW/EBSDataSource
Do you want to add a node (yes/no) [no] :
Running:
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -cp /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/xmlparserv2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojdbc6.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/emCfg.jar oracle.apps.ad.context.CloneContext -e /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../context/apps/CTXORIG.xml -validate -pairsfile /tmp/adpairsfile_27612.lst -stage /u11/home/ebspat/fs2/EBSapps/comn/clone  2> /tmp/adcfgclone_27612.err; echo $? > /tmp/adcfgclone_27612.res
Log file located at /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/CloneContext_0419034910.log
Target System File Edition type [run] : patch (we should enter patch)
Enter the full path of Run File System Context file : /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml
Provide the values required for creation of the new APPL_TOP Context file.
Target System Fusion Middleware Home set to /u11/home/ebspat/fs2/FMW_Home
Target System Web Oracle Home set to /u11/home/ebspat/fs2/FMW_Home/webtier
Target System Appl TOP set to /u11/home/ebspat/fs2/EBSapps/appl
Target System COMMON TOP set to /u11/home/ebspat/fs2/EBSapps/comn
Target System Instance Top set to /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03
Target System Port Pool [0-99] : 60(in FS1 the port pool is 50 so here we should enter 60)
Checking the port pool 56
done: Port Pool 56 is free
Report file located at /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/admin/out/portpool.lst
Complete port information available at /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/admin/out/portpool.lst
UTL_FILE_DIR on database tier consists of the following directories.
1. /bio/ebs/pat/tmp
2. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1] : 2 – select /usr/tmp
Creating the new APPL_TOP Context file from :
  /u11/home/ebspat/fs2/EBSapps/comn/clone/context/apps/adxmlctx.tmp
The new APPL_TOP context file has been created :
/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml
Log file located at /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/CloneContext_0419034910.log
Check Clone Context logfile /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/CloneContext_0419034910.log for details.
Running Rapid Clone with command:
Running:
perl /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/adclone.pl java=/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre mode=apply stage=/u11/home/ebspat/fs2/EBSapps/comn/clone component=appsTier method=CUSTOM appctxtg=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml showProgress contextValidated=true
FMW Pre-requisite check log file location : /u11/home/ebspat/fs2/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Running: /u11/home/ebspat/fs2/EBSapps/comn/clone/FMW/t2pjdk/bin/java -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/engine.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereq.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraPrereqChecks.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstaller.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/OraInstallerNet.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/srvm.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/Scripts/ext/jlib/ojdl-log4j.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/xmlparserv2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/prereq/webtier/oui/jlib/share.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java oracle.apps.ad.clone.util.FMWOracleHomePreReqCheck -prereqCheckFMW -e /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml -stage /u11/home/ebspat/fs2/EBSapps/comn/clone -log /u11/home/ebspat/fs2/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Beginning application tier Apply - Sun Apr 19 03:52:15 2015
/u11/home/ebspat/fs2/EBSapps/comn/clone/bin/../jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=true -Doracle.installer.oui_loc=/oui -classpath /u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/xmlparserv2.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojdbc6.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/java:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/oui/OraInstaller.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/oui/ewt3.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/oui/share.jar:/u11/home/ebspat/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/ojmisc.jar:/u11/home/ebspat/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u11/home/ebspat/fs2/EBSapps/comn/clone/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.ApplyAppsTier -e /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml -stage /u11/home/ebspat/fs2/EBSapps/comn/clone    -showProgress -nopromptmsg
Log file located at /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/admin/log/clone/ApplyAppsTier_04190352.log
100% completed
Completed Apply...
1. Run con clean in DB nodes:
1) Concclean
2) Auto config
2. Run auto config in all application nodes – Run fs
1) Run Auto config
3. Check and make sure that admin server runs only in Primary server and web is disabled in CC nodes.
select NODE_NAME,SUPPORT_CP,SUPPORT_WEB,SUPPORT_ADMIN from fnd_nodes;
If managed server oacore_server2 exists in primary node then,
1) perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -accessgate=apdbablog11.apdba.com:6891
2) perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -ekanban=apdbablog11.apdba.com:6891
3) perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -yms=apdbablog11.apdba.com:6891
After you run these three make sure there are no entries of host apdbablog11.apdba.com  in mod_wl_ohs.conf under /u13/home/ebsdev/fs2/FMW_Home/webtier/instances/EBS_web_ebsdev_OHS1/config/OHS/EBS_web_ebsdev/
Then run below to remove oaea_server2 managed server and confirm the server is removed.
4) perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-managedserver -contextfile=$CONTEXT_FILE -managedsrvname=oaea_server2 -servicetype=accessgate -logfile=/tmp/oaeaserver2.log
FS_CLONE:
Source the Run env
Primary RUN FS amdmin server should run
[applprd@apdbablog01 ~]$ adop phase=fs_clone force=yes
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials...
Initializing...
    Run Edition context  : /u11/home/ebsprd/fs1/inst/apps/ebsprd_apdbablog01/appl/admin/ebsprd_apdbablog01.xml
    Patch edition context: /u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog01/appl/admin/ebsprd_apdbablog01.xml
    Patch file system freespace: 341.25 GB
Validating system setup...
    Node registry is valid.
Generating log report...
    Output: /u11/home/ebsprd/fs_ne/EBSapps/log/adop/43/fs_clone_20150819_022924/ebsprd_apdbablog01/adzdshowlog.out
adop phase=fs_clone - Completed Successfully
adop exiting with status = 0 (Success)
Issue 1:
-----------------------------
ERROR: The following required ports are in use:
-----------------------------
7451 : Managed Server Port 2
7651 : Managed Server Port 1
7851 : Managed Server Port 3
7251 : Managed Server Port 4
Corrective Action: Free the listed ports and retry the adop operation.
FIX :
1) Down all the services in Primary Run fs and restrated the clone
2) Backup all the Context file Change the below server port numbers ( old 7451 : New 7457) make sure your given port number must be
oacorestatus, formsstatus, oafmstatus,forms-c4wsstatus
once the port number changed in all the context files run auto config in all the nodes and start the services(make sure the old port number is not in any context files )
Issue 2:
/u11/home/ebsprd/fs_ne/EBSapps/log/adop/43/fs_clone_20150818_221556/ebsprd_apdbablog01/TXK_SYNC_create/fsclone_stage/FSCloneStageAppsTier_08182219.log:
--------------------------------------------------------------------------------------------------------------------------------------------------------
Lines #(23-27):
patchMSList size: 34
ERROR: Number of servers are not in sync between Run and Patch Context.
ERROR: Managed Servers would be in-sync at apply phase
Lines #(24-28):
patchMSList size: 34
ERROR: Number of servers are not in sync between Run and Patch Context.
ERROR: Managed Servers would be in-sync at apply phase
Fix:
Run the below command in all the nodes to sync the Servers with patch FS and run the auto config in all the Run FS(Please change tha node name in the command)
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/u11/home/ebsprd/fs1/inst/apps/ebsprd_apdbablog05/appl/admin/ebsprd_apdbablog05.xml logfile=/tmp/patchctxupload.log
Issue 3:
Lines #(73-77):
diffSrvName : oaea_server2
runMSSrvPortsVal : oaea_server1:6851
ERROR: Run fs Context variable s_oaea_server_ports value cannot be NULL for oaea_server2
ERROR:  Derived Patch managed server oaea_server2 port : NULL
ERROR: Failed to clone Run Context file to refresh Patch context file
Lines #(74-78):
runMSSrvPortsVal : oaea_server1:6851
ERROR: Run fs Context variable s_oaea_server_ports value cannot be NULL for oaea_server2
ERROR:  Derived Patch managed server oaea_server2 port : NULL
ERROR: Failed to clone Run Context file to refresh Patch context file
Lines #(75-79):
ERROR: Run fs Context variable s_oaea_server_ports value cannot be NULL for oaea_server2
ERROR:  Derived Patch managed server oaea_server2 port : NULL
ERROR: Failed to clone Run Context file to refresh Patch context file
FIX: please do to the following
1) Stop the services in all the Run FS
2) cd /u11/home/ebsprd/fs1/FMW_Home/user_projects/domains/EBS_domain_ebsprd/servers/
3) mv oaea_server2 oaea_server2_old
4) run the autoconfig in all the nodes
5) run the FS_CLONE again
Configure Shared APPL_TOP
Source RUN_FS of Primary application server
1. The value for the context variable "s_atName" must be same across all the application tier node context filesEnsure
2. that value of the AutoConfig variable " s_shared_file_system" is set to "true" ( without the quote) on the primary application tier node.
vi $CONTEXT_FILE
Change variable "s_shared_file_system" is set to "true"
Configure Node 2 - apdbablog02 Web
3. Start weblogic on primary application server if it is not already running and run Preclone on Primary Run FS
Before executing these steps, ensure the AdminServer on both the Run Edition File System and the Patch Edition File System is running. This is required for adcfgclone.pl to properly re-register the new node on the Target System
Source Run file system
Start weblogic administration server
./adadminsrvctl.sh start
./adpreclone.pl appsTier
4. Start weblogic on primary application Patch FS server if it is not already running and run Preclone on Primary Patch FS
Source Patch file system
Start weblogic administration server
./adadminsrvctl.sh start forcepatchfs
./adpreclone.pl appsTier
stop weblogic admin server
./adadminsrvctl.sh stop forcepatchfs
5. Create context files for RUN file system second application node.
[applpat@apdbablog02 scripts]$ cd /u11/home/ebspat/
[applpat@apdbablog02 ebspat]$ ls
EBSapps.env  fs1  fs2  fs_ne  sqlnet.log
[applpat@apdbablog02 ebspat]$ mkdir -p pairsfiles/run
[applpat@apdbablog02 ebspat]$ mkdir -p pairsfiles/patch
[applpat@apdbablog02 ebspat]$
[applpat@apdbablog02 run]$ cd /u11/home/ebspat/pairsfiles/run
[applpat@apdbablog02 run]$ cp /u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03_run.txt ./
1.Ensure the correct perl utility is in the $PATH. You must use the perl utility installed in the Fusion Middleware Oracle Home.
 For example:
$ export PATH=/u11/home/ebsprd/fs1/FMW_Home/webtier/perl/bin:$PATH
Edit the Pairs run and patch file
Enter the new node information to be added in pairsfile of run and patch
Run File system
S_batch status should be enabled only in the concurrent nodes in order to configure PCP.
Adminserver runs only in the primary node
Export path: on Secondary node
export PATH=/u11/home/ebsprd/fs2/FMW_Home/webtier/perl/bin:$PATH
For Run FS
2.Ensure the WebLogic Administration Server from the run file system is running on the primary application tier node.
cd $INST_TOP/admin/scripts
./adadminsrvctl.sh start
3.Execute the Clone Context Utility to configure the run file system. For example:
$ cd /u11/home/ebsprd/fs1/EBSapps/comn/clone/bin
$ /u11/home/ebsprd/fs1/FMW_Home/webtier/perl/bin/perl ./adclonectx.pl \
 addnode contextfile=/u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog01/appl/admin/ebsprd_apdbablog01.xml \
 pairsfile=/u11/home/ebspat/pairsfiles/run/ebspat_apdbablog02_run.txt \
 outfile=/u11/home/ebspat/fs1/inst/apps/ebspat_apdbablog02/appl/admin/ebsprd_apdbablog02.xml
Now node added succesffully for Run fs
Create context files for PATCH file system second application node.
export PATH=/u11/home/ebspat/fs2/FMW_Home/webtier/perl/bin:$PATH
[applpat@apdbablog03 admin]$ cd /u11/home/ebspat/pairsfiles/patch
[applpat@apdbablog03 patch]$ cp /u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03_patch.txt ./
[applpat@apdbablog03 scripts]$ ./adadminsrvctl.sh start forcepatchfs
You are running adadminsrvctl.sh version 120.10.12020000.9
Enter the WebLogic Admin password:
Enter the APPS Schema password:
[applpat@apdbablog04 secure]$ cd /u11/home/ebspat/fs2/EBSapps/comn/clone/bin/                             [applpat@apdbablog04 bin]$ /u11/home/ebspat/fs2/FMW_Home/webtier/perl/bin/perl ./adclonectx.pl \
addnode contextfile=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog03/appl/admin/ebspat_apdbablog03.xml \ pairsfile=/u11/home/ebspat/pairsfiles/patch/ebspat_apdbablog03_patch.txt \ outfile=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog04/appl/admin/ebspat_apdbablog04.xml
Node added successfully
[applprd@apdbablog02 ebsprd]$ perl /u11/home/ebsprd/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh                  -contextfile=/u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog02/appl/admin/ebsprd_apdbablog02.xml -hosts=apdbablog01,apdbablog02,apdbablog03,apdbablog04,apdbablog05,apdbablog06
Enter SSH User password for the OS user applprd:
Logfile location /u11/home/ebsprd/fs2/inst/apps/ebsprd_apdbablog02/logs/appl/rgf/TXK/enablessh.log
SSH equivalence setup successful.
6. Run auto config on second RUN file system
cd $INST_TOP/admin/scripts
./adautocfg.sh appspass=<pw>
7. Run auto config on second patch file system
cd $AD_TOP/bin
./adconfig.sh contextfile=<patchcontextfile.xml> -syncctx
Eg:- ./adconfig.sh contextfile=/u11/home/ebspat/fs2/inst/apps/ebspat_apdbablog02/appl/admin/ebspat_apdbablog02.xml -syncctx
Configure PCP
1. Start application services in all nodes
cd $INST_TOP/admin/scripts
./adstrtal.sh apps/<pw>
2. Stop adcmctl in both concurrent nodes.
3. Take all CM that are active and set workshift as Standard - 1 and set primary and secondary servers(Refer source CMs)
4. Make sure the nodes are defined as below
5. Assign primary/secondary nodes for each manager
6. Restart the managers and validate
cd $INST_TOP/admin/scripts
./adcmctl.sh stop apps/<pw>
./adcmctl.sh start apps/<pw>
4. Start all application services now. Check if concurrent process runs only in 05 and 06 nodes.