12.2.3 Cloning Procedure
Pre-Clone Phase
Preparing Target
Environment
Application Tier:
1. The environment and configuration files
(.env, .xml, tnsnames, httpd.conf, mod_wl_ohs.conf, apps.conf, default.env etc)
(.env, .xml, tnsnames, httpd.conf, mod_wl_ohs.conf, apps.conf, default.env etc)
should be backed up. The
backup should be placed under
$HOME/apdba/clone/backup/ directory only.
$HOME/apdba/clone/backup/ directory only.
You can use the script
under $HOME/apdba/clone/ backup_config.sh to take the backups.
2. Shutdown the application and remove the
files system (both fs1/fs2) and rename the oraInventory.
cd $ADMIN_SCRIPTS_HOME
adstpall.sh apps/<pwd>
Note: This will ask for weblogic password as well.
Database Tier:
1. Backup the parameter file under
$ORACLE_HOME/dbs/{pfile|spfile}.ora under $HOME/apdba/clone/backup
directory.
2. Also, create pfile from spfile as drop
database will remove the spfile.
create pfile from
spfile;
3. Restart the TARGET database in “restrict”
mode and drop the database using the commands below.
sqlplus “/as sysdba”
shutdownn immediate
startup mount restrict;
select name from
v$database;
drop database;
4. Ensure all the data files are
deleted.
Preparing Source
Environment
The Pre-Clone Phase
describes steps needed to prepare the source and target instances for the
cloning process.
Database Node Pre-Clone:
1.Login to the apdba primary node of the
Production server (orap01.apdbablogspot.com) as datab ase owner.
2. Navigate to
$ORACLE_HOME/appsutil/scripts/<CONTEXT_NAME>/
cd
$ORACLE_HOME/appsutil/scripts/APPROD_apdb01
3. Execute adpreclone.pl for the patch file
system and validate the log files.
[oraprod@apdb01 APPROD_apdb01]$
perl adpreclone.pl dbTier
Application Node
Pre-clone:
1. Login to the apdbamary node of the
Production server (orap01.apdbablogspot.com) as application user and source the
run file system.
2. Navigate to $ADMIN_SCRIPTS_HOME and
execute adpreclone.pl.
perl adpreclone.pl
appsTier
3. Once the pre-clone is completed in run
FS, source the patch FS of the primary node and navigate to
$ADMIN_SCRIPTS_HOME.
4. Start the AdminServer of the patch file
system by executing the below command.
adadminsrvctl.sh start
forcepatchfs
5. Execute adpreclone.pl for the patch file
system and validate the log files.
perl adpreclone.pl
appsTier
Database Backup:
1. Database is backed up every day via RMAN
that is scheduled through crontab. The same backup can be used for cloning
purpose.
Note: The RMAN Hot
Backup Script is to be used only when cloning the target instance from
Production Hot Backup. As all the non-production databases are in no-archivelog
mode and not undergoing any backups.
Script Name : /home/orAPPROD/scripts/rman/APPROD_rman_lvl0.sh
Backup location : /backupdb/APPROD/RMAN/{control_backup|level0|archive}
Usage : / APPROD_rman_lvl0.sh <GLOBAL_NAME>
<INSTANCE>
./rman_backup_hot.sh APPROD
1
===================
[orapddev@apdbasrv01 level0]$ cat rman_hot_backup.sh
rman target / log=/backupdb/APDPROD/RMAN/level0/APDPROD_bkp_043016.log <<EOF
run
{
allocate channel c1 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c2 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c3 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c4 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c5 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
BACKUP AS COMPRESSED BACKUPSET INCREMENTAL LEVEL 0 TAG 'APDPROD_hotbackup_for_APDDEV' format '/backupdb/APDPROD/RMAN/level0/%d_NEW_%Y%M%D_%t_%s' database plus archivelog;
copy CURRENT CONTROLFILE TO '/backupdb/APDPROD/RMAN/level0/%d_NEW_cntrlbk_%Y%M%D_%t_%s';
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
}
EXIT;
EOF
[orapddev@apdbasrv01 level0]$
===================
[orapddev@apdbasrv01 level0]$ cat rman_hot_backup.sh
rman target / log=/backupdb/APDPROD/RMAN/level0/APDPROD_bkp_043016.log <<EOF
run
{
allocate channel c1 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c2 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c3 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c4 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
allocate channel c5 type disk FORMAT '/backupdb/APDPROD/RMAN/level0/%U';
BACKUP AS COMPRESSED BACKUPSET INCREMENTAL LEVEL 0 TAG 'APDPROD_hotbackup_for_APDDEV' format '/backupdb/APDPROD/RMAN/level0/%d_NEW_%Y%M%D_%t_%s' database plus archivelog;
copy CURRENT CONTROLFILE TO '/backupdb/APDPROD/RMAN/level0/%d_NEW_cntrlbk_%Y%M%D_%t_%s';
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
}
EXIT;
EOF
[orapddev@apdbasrv01 level0]$
=================================
2. The backup location is not shared and
hence do copy(SCP) of the backup pieces from /backupdb/APPROD/RMAN/level0/ and
/backupdb/APPROD/RMAN/archive/ to the target server under “/backupdb/APPROD/RMAN/level0”.
Note: The backup from
production server (both archives and data file backup) should be placed under
highlighted location of the target server to ease up the duplicate
process.
Application
Backup:
1. Backup the Application file system using
the script,
Script Name : /home/apAPPROD/apdba/clone/scripts/application_backup.sh
Backup location : /l01/Backup/APPROD/{APPROD.dt}/
Usage : sh application_backup.sh
<GLOBAL_NAME> <LOCATION>
sh application_backup.sh
APPROD /l01/Backup/APPROD
2. Copy the backup files from the source
(Production) application server to the source (non-production) application
server.
For example, login to
the non-production application server as application owner and execute the
below statement to scp the run file system alone.
scp -pr
<application_owner>@<server name>:/l01/Backup/APPROD/{APPROD.dt}/fs2.tar.gz
.
(or)
scp -pr
<application_owner>@<server name>:/l01/Backup/APPROD/{APPROD.dt}/fs1.tar.gz
.
3. You can identify the run/patch file
system of the 12.2, environments by querying TWO_TASK variable at OS level.
If run file system is
sourced, the output of TWO_TASK will be APPROD
If patch file system is
sourced, the output of TWO_TASK will be APPROD_patch.
(or)
You can use the
following command to confirm that the environment variable FILE_EDITION points
to the run edition file system: echo $FILE_EDITION and this should return the
value as “run”
Cloning Phase
Database Tier
1. Check if any opatch has been recently
applied in Production and are not available in Target. If required, install the
missing opatch in target environments and this exercise is performed to avoid
copy of RDBMS home from source to target.
2. Preliminary Checks on the target server :
a. Check if the db_file_name_convert and
log_file_name_convert parameters are set in the pfile/spfile of the target
environment. If this is not set, please update the pfile (created as part of
prepare phase) and restart the database to nomount state.
db_file_name_convert=’<SOURECE>,
<TARGET>’
log_file_name_convert=’<SOURCE>,<TARGET>’
For APDTST:
db_file_name_convert='/d02/app/oradata/APPROD/','/d02/oracle/oradata/APDTST/','/d03/app/oradata/APPROD/',
'/d05/oracle/oradata/APDTST/','/d04/app/oradata/APPROD/','/d09/oracle/oradata/APDTST/',
'/d05/app/oradata/APPROD/','/d10/oracle/oradata/APDTST/','/d06/app/oradata/APPROD/','/d11/oracle/oradata/APDTST/'
log_file_name_convert='/d03/app/oradata/APPROD/','/d02/oracle/oradata/APDTST/','/d02/app/oradata/APPROD/',
'/d05/oracle/oradata/APDTST/','/d05/app/oradata/APPROD/','/d10/oracle/oradata/APDTST/'
For APDDEV:
db_file_name_convert='/d02/app/oradata/APPROD/','/d03/app/oradata/APDDEV/','/d03/app/oradata/APPROD/',
'/d04/app/oradata/APDDEV/','/d04/app/oradata/APPROD/','/d06/app/oradata/APDDEV/','/d05/app/oradata/APPROD/',
'/d07/app/oradata/APDDEV/','/d06/app/oradata/APPROD/','/d08/app/oradata/APDDEV/'
log_file_name_convert='/d03/app/oradata/APPROD/','/d03/app/oradata/APDDEV/','/d02/app/oradata/APPROD/',
'/d06/app/oradata/APDDEV/','/d05/app/oradata/APPROD/','/d08/app/oradata/APDDEV/'
b. Login to the non-production server as
database owner (here orapddev is used as an example) and navigate to $HOME/
apdba/scripts.
c. View the file rman_apddev_dup.rman to
validate the timestamp to be used for point-in-time recovery. Below is only an
example and rest of the non-production database owner have their own duplicate
script with rman_<sid>_dup.rman
[orapddev@bosmvsrv01
scripts]$ cat rman_apddev_dup.rman
connect auxiliary /
run
{
allocate auxiliary
channel c20 device type disk;
allocate auxiliary
channel c21 device type disk;
allocate auxiliary
channel c22 device type disk;
allocate auxiliary
channel c23 device type disk;
allocate auxiliary
channel c24 device type disk;
set until time
"to_date('04-OCT-2014 03:20:00','DD-MON-YYYY HH24:MI:SS')";
DUPLICATE DATABASE TO apddev
BACKUP LOCATION '/backupdb/APPROD/RMAN/level0/' NOFILENAMECHECK;
}
Note:
? In the above script, we set the until time clause for the
incomplete recovery which needs to be the completion time of the Production
backup.
? The target database name which needs to be cloned
(duplicated).
? For convenience, both are highlighted. As mentioned in the
prepare phase, all the backup pieces should be placed under /backupdb/APPROD/RMAN/level0/
d. Execute the below script as target
database owner from a vnc,
rman log=apddev_clone_<enter_date>.log
e. Once you se e the RMAN> prompt in vnc,
execute the file which was edited/created earliest “rman_<sid>_dup.rman”
RMAN>@rman_apddev_dup.rman
f. This will run without any issues and
opens the database. You can monitor the restore in the rman log file or the
alert log file of the target.
g. Restart the listener with the example as
below:
lsnrctl stop apddev
lsnrctl start apddev
3. Post duplicate checks on the target
server.
a. Ensure no datafiles are in recovery list
and database is opened cleanly.
select * from
v$recover_file;
select distinct status
from dba_data_files;
b. Ensure that the database is not in
archivelog mode by disabling archiving
select instance_name,
host_name, version, startup_time, status, archiver from v$instance;
c. Verify and recreate Temp Tablespace as
below (IF required).
select tablespace_name,file_name,bytes/1000/1000/1000
from dba_temp_files;
To add the temp files,
please use the command below or script $HOME/apdba/scripts/temp.sql
FOR APDTST:
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp01.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp02.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp03.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp04.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp05.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp06.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp07.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d11/oracle/oradata/APDTST/temp08.dbf' size 3000M reuse;
FOR APDDEV:
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp01.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp02.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp03.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp04.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp05.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp06.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp07.dbf' size 3000M reuse;
alter tablespace temp
add tempfile '/d08/app/oradata/APDDEV/temp08.dbf' size 3000M reuse;
d. Verify the creation date of the database.
select name,created from
v$database;
Note: Perform the Step#4
below is the creation date is “NOT” as of current date.
4. Once the database is up and running
without any issues, backup the control file and recreate the control file to
ensure the correct database creation date is reflected in v$database. (This
step to be performed only if the creation date of the environment is same as
production (07/17/2014)
a. Execute “alter database backup
controlfile to trace;”
b. Execute “shutdown immediate;”
c. Navigate to the background_dump_dest
location and remove the unwanted lines in the trace except the NORESETLOGS
section of the trace file and create a create_control.sql script.
d. Execute “startup nomount”
e. Execute “@create_control.sql”
script
f. Execute “alter database open resetlogs;”
5. Login to the target database and cleanup
the fnd_nodes tables.
sqlplus apps/<password>
exec
FND_CONC_CLONE.setup_clean;
select * from
apps.fnd_nodes;
6. Navigate to
$ORACLE_HOME/appsutil/scripts/<context_name>/ and execute adautocfg.sh
Application Tier
1. Login to the non-production Application
node as application owner and untar the files that were scp’d from Production.
2. Extract the run file system from the tar
backup which was copied to the non-production server as part of prepare phase
section.
nohup cat fs2.tar.gz |
gunzip| tar xf - &
(or)
nohup cat fs1.tar.gz |
gunzip| tar xf - &
3. Once the untar is completed for the
“to-be” run file system of the target environment, remove the FMW_HOME
directory as this will be created as part of rapidclone.
4. Verify if the inventory is pointing to
the correct location in /etc/oraInst.loc. Modify the inventory if needed to
refer to the right location and specific to the environment.
5. Configure the run file system for the
target environment. Navigate to $OAD_TOP/clone/bin [example: /d01/app/APDDEV/<run_fs>/EBSapps/comn/clone/bin]
and execute as below and answer the prompts with respect to the environments.
Below prompts are for reference only.
Env name Port Pool
OHS Port
(run/patch) (run/patch)
APDTST Internal External Internal External
run - 0, patch -1 run -2,
patch -3 8000 8001
APDDEV run - 95,patch -96 N/A 8095 N/A
perl adcfgclone.pl
appsTier
[apapddev@bosmvdmz01 ~]$
cd /d01/app/APDDEV/fs1/EBSapps/comn/clone/bin
[apapddev@apdbablog02
bin]$ perl adcfgclone.pl appsTier
Enter the APPS password:
Enter the Weblogic
AdminServer password: “Enter non-production weblogic password: welcome12”
Do you want to add a
node (yes/no) [no]:
Target System File
Edition type [run]:
Target System Hostname
(virtual or normal) [apdbablog02]:
Target System Database
SID: APDDEV
Target System Database
Server Node [apdbablog02]: bosmvsrv01
Target System Database
Domain Name [apdbablogspot.com]:
Target System Base
Directory: /d01/app/APDDEV
Target System Base
Directory set to /d01/app/APDDEV
Target System Current
File System Base set to /d01/app/APDDEV/fs1
Target System Other File
System Base set to /d01/app/APDDEV/fs2
Target System Fusion
Middleware Home set to /d01/app/APDDEV/fs1/FMW_Home
Target System Web Oracle
Home set to /d01/app/APDDEV/fs1/FMW_Home/webtier
Target System Appl TOP set
to /d01/app/APDDEV/fs1/EBSapps/appl
Target System COMMON TOP
set to /d01/app/APDDEV/fs1/EBSapps/comn
Target System Instance
Home Directory [/d01/app/APDDEV]:
Target System Instance
Top set to /d01/app/APDDEV/fs1/inst/apps/APDTST_bosmvdmz01
Do you want to preserve
the Display [orap01:0.0] (y/n): n
Target System Display [apdbablog02:0.0]:
Target System Root
Service [enabled] : enabled
Target System Web Entry
Point Services [enabled]: enabled
Target System Web
Application Services [enabled]: enabled
Target System Batch
Processing Services [enabled]: enabled
Target System Other
Services [disabled]: enabled
Do you want the target
system to have the same port values as the source system (y/n) [y]? : n
Target System Port Pool
[0-99]: 95
Checking the port pool
95
done: Port Pool 95 is
free
Report file located at
/d01/app/APDDEV/fs1/inst/apps/APDDEV_apdbablog02/admin/out/portpool.lst
Complete port
information available at /d01/app/APDDEV/fs1/inst/apps/APDDEV_apdbablog02/admin/out/portpool.lst
UTL_FILE_DIR on database
tier consists of the following directories.
1. /d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/temp
2. /usr/tmp
3. /usr/tmp/APDBA
4. /d01/app/APDDEV/CUSTOM/xxsuf/out
5. /usr/tmp
6. /d01/app/oracle/APDTST122/product/11.2.0/appsutil/outbound/APDTST_bosmvsrv01
7. /usr/tmp
8. /usr/tmp
Choose a value which
will be set as APPLPTMP value on the target node [1] :
Backing up /d01/app/APDDEV/fs1/inst/apps/APDTST_bosmvdmz01/appl/admin/APDTST_bosmvdmz01.xml
to /d01/app/APDDEV/fs1/inst/apps/APDTST_bosmvdmz01/appl/admin/APDTST_bosmvdmz01.xml1.bak
Creating the new
APPL_TOP Context file from :
/d01/app/APDDEV/fs1/EBSapps/comn/clone/context/apps/adxmlctx.tmp
The new APPL_TOP context
file has been created :
/d01/app/APDDEV/fs1/inst/apps/APDTST_bosmvdmz01/appl/admin/APDTST_bosmvdmz01.xml
Log file located at
/d01/app/APDDEV/fs1/EBSapps/comn/clone/bin/CloneContext_0819120049.log
Check Clone Context
logfile /d01/app/APDDEV/fs1/EBSapps/comn/clone/bin/CloneContext_0819120049.log
for details.
/d01/app/APDDEV/fs1/EBSapps/comn/clone/bin/../jre/bin/java
-Xmx600M -DCONTEXT_VALIDATED=true -Doracle.installer.oui_loc=/oui
-classpath /d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/xmlparserv2.jar:/d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/ojdbc6.jar:
/d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/java:/d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/oui/OraInstaller.jar:
/d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/oui/ewt3.jar:/d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/oui/share.jar:
/d01/app/APDDEV/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/ojmisc.jar:
/d01/app/APDDEV/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/d01/app/APDDEV/fs1/EBSapps/comn/clone/jlib/obfuscatepassword.jar
oracle.apps.ad.clone.ApplyAppsTier
-e /d01/app/APDDEV/fs1/inst/apps/APDTST_bosmvdmz01/appl/admin/APDTST_bosmvdmz01.xml
-stage
/d01/app/APDDEV/fs1/EBSapps/comn/clone
-showProgress -nopromptmsg
Log file located at
/d01/app/APDDEV/fs1/inst/apps/APDTST_bosmvdmz01/admin/log/clone/ApplyAppsTier_08191201.log
|
100% completed
Completed Apply...
Do you want to startup
the Application Services for TEST? (y/n) [n] :
Services not started
6. Update the override address by logging in
as APPS user in the target database.
select parameter_value
from fnd_svc_comp_param_vals where parameter_id = (select parameter_id from
fnd_svc_comp_params_tl where display_name = 'Test Address');
update
fnd_svc_comp_param_vals set parameter_value = ' PPinyochon@apdbaolk.com' where
parameter_id = (select parameter_id from fnd_svc_comp_params_tl where
display_name ='Test Address');
7. Validate and Change APPLSYS/SYSADMIN
passwords using FNDCPASS
FNDCPASS
apps/<pwd> 0 Y system/<pwd> SYSTEM APPLSYS <REFER SHEET>
FNDCPASS
apps/<pwd> 0 Y system/manager USER SYSADMIN <REFER SHEET>
8. Bring up AdminsServer alone and modify
the apps password in the WebLogic console against the EBSDatasource.
a. Login to the weblogic console as weblogic
credentials
b. Click on Lock&Edit in the Change
center
c. Expand “Services” under Domain Structure.
d. Click on Data Sources and click on
EBSDataSource HomePage
e. On the “Settings for EBSDataSource” page,
select the Connection Pool tab
f. Update the new apps password and Save the
changes. Click Activate Changes in Change Center.
g. Shut down the services, run autoconfig
and restart your services.
cd $ADMIN_SCRIPTS_HOME
adstrtal.sh
apps/<apps password>
Provide Weblogic
Password:
10. Perform the SSL setup in the $CONTEXT_FILE
and run autoconfig
Context Variable APDDEV Context Value APDTST Context Value
s_webentryurlprotocol https https
s_webentryhost oracledev oracletest
s_webentrydomain apdbaolk.com apdbaolk.com
s_active_webport 443 443
s_login_page https://oracledev.apdbaolk.com/OA_HTML/AppsLogin
https://oracletest.apdbaolk.com/OA_HTML/AppsLogin
s_external_url https://oracledev.apdbaolk.com
https://oracletest.apdbaolk.com
11. Configure the Patch file system for the
target environment by following the below steps.
a. Run pre-clone on the run file
system
cd $ADMIN_SCRIPTS_HOME
perl adpreclone.pl
appsTier
b. Shut down the run file system services,
adstpall.sh
apps/<password>
Provide Weblogic
Password:
c. Copy the EBSapps directory from the run
file system to the patch file system. Ensure the patch file system base
directory structure exists apdbaor to copying the files from run fs.
d. Unset the source environment variables
(logout and login) before configuring the patch fs.this is to ensure you are
not connected to the run fs environment.
e. Navigate to $OAD_TOP/clone/bin [example:
/d01/app/APDDEV/<patch_fs>/EBSapps/comn/clone/bin] and execute as below
and answer the prompts with respect to the environments. Below prompts are for
reference only
perl adcfgclone.pl
appsTier
[apapddev@bosmvdmz01 ~]$
cd /d01/app/APDDEV/fs2/EBSapps/comn/clone/bin
[apapddev@apdbablog02
bin]$ perl adcfgclone.pl appsTier
Target System File
Edition type [run] : patch
Enter the full path of
Run File System Context file : /d01/app/APDDEV/fs1/inst/apps/APDDEV_apdbablog02/appl/admin/APDDEV_apdbablog02.xml
Provide the values
required for creation of the new APPL_TOP Context file.
Target System Fusion
Middleware Home set to /d01/app/APDDEV/fs2/FMW_Home
Target System Web Oracle
Home set to /d01/app/APDDEV/fs2/FMW_Home/webtier
Target System Appl TOP
set to /d01/app/APDDEV/fs2/EBSapps/appl
Target System COMMON TOP
set to /d01/app/APDDEV/fs2/EBSapps/comn
Target System Instance
Top set to /d01/app/APDDEV/fs2/inst/apps/APDDEV_apdbablog02
Target System Port Pool
[0-99] : 96
Checking the port pool
96
done: Port Pool 1 is
free
Report file located at
/d01/app/APDDEV/fs2/inst/apps/APDDEV_apdbablog02/admin/out/portpool.lst
Complete port
information available at /d01/app/APDDEV/fs2/inst/apps/APDDEV_apdbablog02/admin/out/portpool.lst
UTL_FILE_DIR on database
tier consists of the following directories.
1. /d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/temp
2. /usr/tmp
3. /usr/tmp/APDBA
4. /d01/app/APDDEV/CUSTOM/xxsuf/out
5. /usr/tmp
6. /d01/app/oracle/APDTST122/product/11.2.0/appsutil/outbound/APDTST_bosmvsrv01
7. /usr/tmp
8. /usr/tmp
Choose a value which
will be set as APPLPTMP value on the target node [1] :
Creating the new
APPL_TOP Context file from :
/d01/app/APDDEV/fs2/EBSapps/comn/clone/context/apps/adxmlctx.tmp
The new APPL_TOP context
file has been created :
...
...
/ 100% completed
Completed Apply...
Wed Oct 23 06:03:53 2013
Looking for incomplete
CLONE record in ad_adop_session_patches table
The CLONE record status
is no rows selected
Updating incomplete
CLONE record to COMPLETED
f. Verify if Oracle Inventory contains the
following Oracle Home entries:
<FMW_HOME>/oracle_common
<FMW_HOME>/webtier
<FMW_HOME>/Oracle_EBS-app1
g. Verify the SSL setup and perform step# 8
in the patch file system $CONTEXT_FILE if required.
12. Once the patch fs setup is completed
successfully, source the run file system and start the services.
[apapddev@apdbablog02
~]$ . .bash_profile
1. Run File System
2. Patch File System
Enter your Choice
1
E-Business Suite
Environment Information
----------------------------------------
RUN File System
: /d01/app/APDDEV/fs1/EBSapps/appl
PATCH File System
: /d01/app/APDDEV/fs2/EBSapps/appl
Non-Editioned
File System : /d01/app/APDDEV/fs_ne
13. DMZ setup/Clone specific for APDTST.
DMZ configuration is
setup in APDTST environment and the document to perform the DMZ setup is
mentioned in the DBA handbook under section “Other technical References” and
also attached here for your reference.
Post-Clone Phase
Database Tier
1. Login to the database server as database
owner and validate the utl_file_dir. The parameter values should be the same as
in below table.
For APDTST:
utl_file_dir /d02/app/APDBA/fs_ne/inst/APDTST_apdbablog02/temp,
/d02/app/APDBA/CUSTOM/xxsuf/out/hr/icims/inbound,
/d02/app/APDBA/CUSTOM/xxsuf/out/hr/icims/outbound,
/d02/app/APDBA/CUSTOM/xxsuf/12.0.0/out,
/d02/app/APDBA/CUSTOM/xxsuf/out,/d02/app/APDBA/CUSTOM/xxsuf/out/kbi,
/d02/app/APDBA/CUSTOM/payroll/nacha,
/d02/app/APDBA/CUSTOM/payroll/pospay,
/d02/app/APDBA/CUSTOM/payroll/401k, /d02/app/APDBA/CUSTOM/ap/pospay,
/d01/app/oracle/APDTST122/product/11.2.0/appsutil/outbound/APDTST_bosmvsrv01,/usr/tmp
For APDDEV:
utl_file_dir /d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/temp,
/d01/app/APDDEV/CUSTOM/xxsuf/out/hr/icims/inbound,
/d01/app/APDDEV/CUSTOM/xxsuf/out/hr/icims/outbound,
/d01/app/APDDEV/CUSTOM/xxsuf/12.0.0/out,
/d01/app/APDDEV/CUSTOM/xxsuf/out,/d01/app/APDDEV/CUSTOM/xxsuf/out/kbi,
/d01/app/APDDEV/CUSTOM/payroll/nacha,
/d01/app/APDDEV/CUSTOM/payroll/pospay,
/d01/app/APDDEV/CUSTOM/payroll/401k, /d01/app/APDDEV/CUSTOM/ap/pospay,
/d01/app/oracle/APDDEV/product/11.2.0/appsutil/outbound/APDDEV_bosmvsrv01,/usr/tmp
2. Modify the SYS/SYSTEM passwords along
with the SMTP setups.
Change SYS/System
Password [Refer connectivity sheet for
passwords]
alter user system
identified by xxxxx;
alter user sys
identified by xxxxxx;
Modify smtp_out_server
paraemter in the parameter file alter
system set smtp_out_server='webmail.apdbablogspot.com' scope=both;
Create or Replace
DIRECTORY mail_attachments select *
from v$database
col name format a25
col OS_PATH format a75
set linesize 300
SELECT d.obj#, u.name,
o.name, d.os_path
FROM
sys.user$ u, sys.obj$ o, sys.dir$ d
WHERE
u.user# = o.owner# AND o.obj# = d.obj#;
Take a note of the OBJ#
for the NAME_1=MAIL_ATTACHMENTS and NAME=SYS.
select * from
sys.dir$ d
where OBJ# =
'&obj_id'; -- where obj_id is the OBJ# noted above.
For APDTST
update sys.dir$ d
set os_path = '/d02/app/APDBA/fs_ne/inst/APDTST_apdbablog02/temp'
where obj# =
'&obj_id';
commit;
For APDDEV
update sys.dir$ d
set os_path = '/d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/temp'
where obj# =
'&obj_id';
commit;
Application Tier
1. Validate the CONTEXT_FILE values with the
backup file and below table.
S. No Parameter Values
1 Modify the CONTEXT_FILE values under oa_customized tag
c_XXSUF_TOP
c_XXAS_TOP
c_MARKVIEW_TOP FOR APDDEV
/d01/app/APDDEV/CUSTOM/xxsuf/12.0.0
/d01/app/APDDEV/CUSTOM/xxas/1.0.1
The value should be set
as similar to $SF_TOP
For APDTST
/d02/app/APDBA/CUSTOM/xxsuf/12.0.0
/d02/app/APDBA/CUSTOM/xxas/1.0.1
The value should be set
as similar to $SF_TOP
APPLPTMP and APPLTMP value to be
verified if not modified For APDTST:
/d02/app/APDBA/fs_ne/inst/APDTST_apdbablog02/temp
For APDDEV:
/d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/temp
If these variables are
modified, please run autoconfig. These values should match the first entry of
utl_file_dir
2 Update default.env under $ORA_CONFIG_HOME/10.1.2/forms/server
The default.env in should have the
custom top entries pointed correctly and _TOP_TOP variables can be removed Considering fs1 as run file system. (APDDEV)
SF_TOP=/d01/app/APDDEV/fs1/EBSapps/appl/sf/12.0.0
XXSUF_TOP=/d01/app/APDDEV/CUSTOM/xxsuf/12.0.0
c_MARKVIEW_TOP=/d01/app/APDDEV/fs1/EBSapps/appl/sf/12.0.0
c_XXSUF_TOP=/d01/app/APDDEV/CUSTOM/xxsuf/12.0.0
c_XXAS_TOP=/d01/app/APDDEV/CUSTOM/xxas/1.0.1
Considering fs2 as run
file system. (APDTST)
SF_TOP=/d02/app/APDBA/fs2/EBSapps/appl/sf/12.0.0
XXSUF_TOP=/d02/app/APDBA/CUSTOM/xxsuf/12.0.0
c_MARKVIEW_TOP=/d02/app/APDBA/fs2/EBSapps/appl/sf/12.0.0
c_XXSUF_TOP=/d02/app/APDBA/CUSTOM/xxsuf/12.0.0
c_XXAS_TOP=/d02/app/APDBA/CUSTOM/xxas/1.0.1
3 Disable JDBC Authentication
Navigate to $FND_SECURE and perform the
following FOR APDTST :
java
oracle.apps.fnd.security.AdminAppServer apps/<apps_pwd> AUTHENTICATION
OFF DBC=APDTST.dbc
FOR APDDEV :
java
oracle.apps.fnd.security.AdminAppServer apps/<apps_pwd> AUTHENTICATION
OFF DBC=APDDEV.dbc
2. For the integration of 170 MarkView, remove
170 MarkView job scheduled for now and we will start it later. From
SQLPLUS as “markview” user, execute the following commands:
sqlplus
markview/markview
SQL> select name from
v$database;
COL SCHEMA_USER FORMAT
A25
COL WHAT FORMAT A70
WORD_WRAP
COL BROKEN FORMAT A12
SET LINESIZE 200
SET PAGESIZE 79
SELECT
JOB,
SCHEMA_USER,
TO_CHAR(LAST_DATE,'DD-MON-YY
HH24:MI:SS') "LAST DATE",
TO_CHAR(NEXT_DATE,'DD-MON-YY
HH24:MI:SS') "NEXT DATE",
BROKEN,
WHAT
FROM USER_JOBS;
Make a note of all job
number and execute the following SQL statement to break and remove each job:
exec
sys.dbms_job.broken(&job_no,TRUE);
3. Verify to put the custom notification
concurrent job on HOLD before run post clone for Apps Tier. Execute
the following scripts. As APPS users, connect via SQL*Plus:
sqlplus apps/xxxxxxx
SQL> select name from
v$database;
• Cancel “XXSUF: Requests on Hold” concurrent job
col conc_prog format a50
select cr.request_id,
cr.phase_code, cr.status_code,
(SELECT SUBSTR (
cp.concurrent_program_name || ' - ' || cpl.user_concurrent_program_name, 1, 40
)
FROM
FND_CONCURRENT_PROGRAMS cp,
FND_CONCURRENT_PROGRAMS_VL cpl
WHERE
cp.application_id = cr.program_application_id
AND
cp.concurrent_program_id = cr.concurrent_program_id
AND
cpl.application_id = cr.program_application_id
AND
cpl.concurrent_program_id = cr.concurrent_program_id ) conc_prog,
Cr.hold_flag Hold_Flag
from
FND_CONCURRENT_REQUESTS cr,
fnd_concurrent_programs
cp
where
cr.concurrent_program_id
= cp.concurrent_program_id
and
cp.CONCURRENT_PROGRAM_NAME like '%SUF_REQUESTS_ON_HOLD%'
and cr.phase_code = 'P'
-- P = Pending
and cr.status_code = 'I'
-- I = Inactive;
Take a note of the
request_id from the result above, and then run the following script to place
the job on hold.
update
FND_CONCURRENT_REQUESTS
set hold_flag = 'Y'
where request_id =
'&request_id';
commit;
4. Check Profile setting for
“CONC_GSM_ENABLED”, it should be “Y” since GSM is configured on the source.
It should return the value as per the table below.
set linesize 500
set pagesize 132
column SHORT_NAME format
A25 wrap
column NAME format A40
wrap
column LEVEL_VAL format
999999999
column VALUE format A30
wrap
select
p.profile_option_name SHORT_NAME,
n.user_profile_option_name NAME,
decode(v.level_id,
10001, 'Site',
10002, 'Application',
10003, 'Responsibility',
10004, 'User',
'UnDef') LEVEL_SET,
v.level_value LEVEL_VAL,
v.profile_option_value VALUE
from fnd_profile_options
p,
fnd_profile_option_values v,
fnd_profile_options_tl
n
where
p.profile_option_id = v.profile_option_id (+)
and
p.profile_option_name = n.profile_option_name
and
upper(n.user_profile_option_name) like upper('Concurrent:GSM Enabled')
and n.language = 'US'
and
p.profile_option_name = 'CONC_GSM_ENABLED';
SHORT_NAME NAME LEVEL_SET LEVEL_VAL VALUE
CONC_GSM_ENABLED Concurrent:GSM Enabled Site 0 Y
5. Check Profile setting for “SITENAME”, it
should be “N” since GSM is not fully configured.
set linesize 500
set pagesize 132
column SHORT_NAME format
A25 wrap
column NAME format A40
wrap
column LEVEL_VAL format
999999999
column VALUE format A30
wrap
select
p.profile_option_name SHORT_NAME,
n.user_profile_option_name NAME,
decode(v.level_id,
10001, 'Site',
10002, 'Application',
10003, 'Responsibility',
10004, 'User',
'UnDef') LEVEL_SET,
v.level_value LEVEL_VAL,
v.profile_option_value VALUE
from fnd_profile_options
p,
fnd_profile_option_values v,
fnd_profile_options_tl n
where
p.profile_option_id = v.profile_option_id (+)
and p.profile_option_name
= n.profile_option_name
and
upper(n.user_profile_option_name) like upper('Site Name')
and n.language = 'US'
and
p.profile_option_name = 'SITENAME';
If needed, you can run
the following statement to update it:
For APDTST:
update fnd_profile_option_values
set profile_option_value = 'APDTST (Refreshed from Production backup on
06/10/2014) ' – Change date as required (format mm/dd/yyyy).
where profile_option_id
in
(select
p.profile_option_id
from fnd_profile_options
p,
fnd_profile_option_values v,
fnd_profile_options_tl n
where
p.profile_option_id = v.profile_option_id (+)
and
p.profile_option_name = n.profile_option_name
and
upper(n.user_profile_option_name) like upper('Site Name')
and n.language = 'US'
and
p.profile_option_name = 'SITENAME');
For APDDEV:
update
fnd_profile_option_values set profile_option_value = 'APDDEV (Refreshed from
Production backup on 06/10/2014) ' – Change date as required (format
mm/dd/yyyy).
where profile_option_id
in
(select
p.profile_option_id
from fnd_profile_options
p,
fnd_profile_option_values v,
fnd_profile_options_tl n
where
p.profile_option_id = v.profile_option_id (+)
and
p.profile_option_name = n.profile_option_name
and
upper(n.user_profile_option_name) like upper('Site Name')
and n.language = 'US'
and
p.profile_option_name = 'SITENAME');
6. Change DISPLAY_NAME for WF_SYSTEMS.
From APPS user, run the following statement:
select * from
WF_SYSTEMS;
Make a note of GUID
value:
FOR APDTST:
update WF_SYSTEMS
set display_name = 'APDTST'
WHERE GUID =
'&enter_GUID_value_from_above';
COMMIT;
FOR APDDEV:
update WF_SYSTEMS
set display_name = 'APDDEV'
WHERE GUID =
'&enter_GUID_value_from_above';
COMMIT;
7. Update database link with the new
database global name in the table WF_AGENTS.ADDRESS.
select * from WF_AGENTS
select global_name from
global_name ;
FOR APDTST:
update wf_agents set
address = replace(address,'APPROD.WORLD','APDTST.WORLD');
FOR APDDEV:
update wf_agents set
address = replace(address,'APPROD.WORLD','APDDEV.WORLD');
COMMIT;
8. Update the APPLPTMP value for the below
profile option.
select
PROFILE_OPTION_VALUE from fnd_profile_option_values where profile_option_id in
(8789,3504,5206,3108,3620,5349,4551);
update
fnd_profile_option_values set
PROFILE_OPTION_VALUE='&ENTER_VALUE_of_APPLPTMP' where profile_option_id in
(8789,3504,5206,3108,3620,5349,4551);
select
PROFILE_OPTION_VALUE from fnd_profile_option_values where profile_option_id in
(8789,3504,5206,3108,3620,5349,4551);
9. Change concurrent request logfile_name
and output_file. Update concurrent requests log and output files and then
purge them
For APDTST
update
FND_CONCURRENT_REQUESTS set logfile_name = replace(logfile_name,'/d01/app/APPROD/fs_ne/inst/APPROD_orap01/log','/d02/app/APDBA/fs_ne/inst/APDTST_apdbablog02/logs/appl/conc/log');
update
FND_CONCURRENT_REQUESTS set outfile_name = replace(outfile_name,'/d01/app/APPROD/fs_ne/inst/APPROD_orap01/out','/d02/app/APDBA/fs_ne/inst/APDTST_apdbablog02/logs/appl/conc/out');
update
FND_CONCURRENT_REQUESTS set logfile_node_name = replace(logfile_node_name,'ORAP01','APDBABLOG02');
update
FND_CONCURRENT_REQUESTS set outfile_node_name =
replace(outfile_node_name,'ORAP01','APDBABLOG02');
COMMIT;
For APDDEV
update
FND_CONCURRENT_REQUESTS set logfile_name = replace(logfile_name,'/d01/app/APPROD/fs_ne/inst/APPROD_orap01/log','/d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/logs/appl/conc/log');
update
FND_CONCURRENT_REQUESTS set outfile_name = replace(outfile_name,'/d01/app/APPROD/fs_ne/inst/APPROD_orap01/out','/d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/logs/appl/conc/out');
update
FND_CONCURRENT_REQUESTS set logfile_node_name =
replace(logfile_node_name,'ORAP01','APDBABLOG02');
update
FND_CONCURRENT_REQUESTS set outfile_node_name =
replace(outfile_node_name,'ORAP01','APDBABLOG02');
COMMIT;
10. Update the TAX libraries in the database.
update
pay_action_parameters set parameter_value= '&PAY_TOP/vendor/quantum/data'
where parameter_name like 'TAX_DATA%';
update
pay_action_parameters set parameter_value= '&PAY_TOP/vendor/quantum/lib'
where parameter_name like 'TAX_LIB%';
11. Check text_value for tables
WF_NOTIFICATION_ATTRIBUTES and WF_ITEM_ATTRIBUTE_VALUES
select * from
WF_NOTIFICATION_ATTRIBUTES where text_value like 'http:%'
update
WF_NOTIFICATION_ATTRIBUTES set text_value =
replace(text_value,'orap01','&enter_application_node_name'); -- application
node name should be without domain (bosmvsrv01)
update
WF_NOTIFICATION_ATTRIBUTES set text_value = replace(text_value,'10600','8090');
select * from
WF_ITEM_ATTRIBUTE_VALUES where text_value like 'http:%'
update WF_ITEM_ATTRIBUTE_VALUES
set text_value =
replace(text_value,'orap01','&enter_application_node_name');
update
WF_ITEM_ATTRIBUTE_VALUES set text_value = replace(text_value,'10600','8090');
commit;
12. Sign-on to Oracle Applications and reduce
Standard Concurrent Manager Work shift processes to 5. Go to Concurrent
-> Manager -> Define -> Query for Manager=Standard Manager -> click
Work Shift. Reduce the processes to 5.
13. Modify Login message web page.
a. Sign-on as SYSADMIN
b. Choose Application Developer responsibility
c. Go to Application > Messages, see
below for what message to change
d. To change messages:
e. Message for the bottom of the screen,
query for "FND_SSO_COPYRIGHT_TEXT" (the default message is Copyright
(c) 2004, Oracle. All rights reserved.).
For APDTST
Copyright (c) 2006,
Oracle. All rights reserved. Connected to APDTST environment.
For APDDEV
Copyright (c) 2006,
Oracle. All rights reserved. Connected to APDDEV environment.
f. To change message for Login button
message on the Logon page, change text values for “FND_SSO_LOGIN”. Change
text to:
Login APDTST / Login APDDEV
g. To change message for welcome Login
message, “FND_SSO_WELCOME”. Change text to:
Login APDTST / Login APDDEV
h. To change OA Framework E-Business
Homepage Branding. As SYSADMIN, choose responsibility Application
Developer -> Application -> Function, query for
“FWK_HOMEPAGE_BRAND”.
Change User Function
Name to: Oracle E-Business Suite 12.2.3 – APDTST Environment
(or)
Change User Function
Name to: Oracle E-Business Suite 12.2.3 – APDDEV Environment
14. Modify custom concurrent programs below to
change output directory to point to APDTST$APPLTMP directory. Sign-on to APDTST/APDDEV
as “SYSADMIN” user, navigate to Concurrent -> Program -> Define.
Query for each concurrent program below and click on Parameter button to
change the output file location directory:
For APDTST
a. XXSUF: Extract VSP File (change the
output directory path to ' $APPLPTMP')
b. XXSUF: Extract AP Positive Pay File -
Citizens Bank, change p_file_path parameter - /d02/app/APDBA/CUSTOM/xxsuf/12.0.0/out
c. XXSUF: Extract AP Positive Pay File -
SunTrust Bank, change p_file_path parameter - /d02/app/APDBA/CUSTOM/xxsuf/12.0.0/out
d. XXSUF: Extract PR Positive Pay File -
Citizens Bank, change “Default Value” parameter - /d02/app/APDBA/CUSTOM/xxsuf/12.0.0/out
e. XXSUF: Extract PR Positive Pay File -
SunTrust Bank, change “Default Value” parameter - /d02/app/APDBA/CUSTOM/xxsuf/12.0.0/out
f. XXSUF: Extract Delta Dental File, change
p_file_dir parameter – $APPLPTMP
g. XXSUF: Move Payroll Check File, change
DESTINATION_DIRECTORY parameter - /d02/app/APDBA/CUSTOM/payroll/2014/
For APDDEV
a. XXSUF: Extract VSP File (change the
output directory path to ' $APPLPTMP')
b. XXSUF: Extract AP Positive Pay File -
Citizens Bank, change p_file_path parameter - /d01/app/APDDEV/CUSTOM/xxsuf/12.0.0/out
c. XXSUF: Extract AP Positive Pay File -
SunTrust Bank, change p_file_path parameter - /d01/app/APDDEV/CUSTOM/xxsuf/12.0.0/out
d. XXSUF: Extract PR Positive Pay File -
Citizens Bank, change “Default Value” parameter - /d01/app/APDDEV/CUSTOM/xxsuf/12.0.0/out
e. XXSUF: Extract PR Positive Pay File -
SunTrust Bank, change “Default Vaue”parameter - /d01/app/APDDEV/CUSTOM/xxsuf/12.0.0/out
f. XXSUF: Extract Delta Dental File, change
p_file_dir parameter – $APPLPTMP
g. XXSUF: Move Payroll Check File, change
DESTINATION_DIRECTORY parameter -
/d01/app/APDDEV/CUSTOM/payroll/2014/
15. Cancel the following concurrent jobs as
they are not needed to be run in the test environment:
apdbaolk OLM Workflow
Reminder
Send iCal Events
apdba: VSP extract
apdba: Delta Dental
Extract
apdba: FSA Extract
XXSUF: WF Pay-When-Paid
Notification
16. Change profile option for following:
For APDTST
apdba_401K_OUTBOUND_PATH
= /d02/app/APDBA/CUSTOM/xxsuf/12.0.0/out
apdba_PAYROLL_POSPAY_PATH
= /d02/app/APDBA/CUSTOM/xxsuf/12.0.0/out
For APDDEV
apdba_401K_OUTBOUND_PATH
= /d01/app/APDDEV/CUSTOM/xxsuf/12.0.0/out
apdba_PAYROLL_POSPAY_PATH
= /d01/app/APDDEV/CUSTOM/xxsuf/12.0.0/out
17. Use XML Publisher's Administration
interface to assign a temporary directory for the site level.
(Navigation: XML
Publisher Administrator > Home > Administration > Configuration >
General Properties > Temporary Directory).
For APDTST :
/d02/app/APDBA/fs_ne/inst/APDTST_apdbablog02/temp
For APDDEV:
/d01/app/APDDEV/fs_ne/inst/APDDEV_apdbablog02/temp
18. Validate the $PAY_TOP quantum sync values
are present and uncommented in adop_sync.drv.
Navigate to
$APPL_TOP_NE/ad/custom and validate the highlighted entries exist in the adop_sync.drv
file. If not, please add the same and save it.
##Begin of Section for
Quantum Synchronization##
###Sync up Quantum
Directories##
rsync -lzr
%s_current_base%/EBSapps/appl/pay/12.0.0/vendor
%s_other_base%/EBSapps/appl/pay/12.0.0
###Update to Action
Parameter Level Quantum Directories Overrides##
%s_tools_oh%/bin/sqlplus
%s_apps_user%/%s_appsPwd%
@%s_current_base%/EBSapps/appl/pay/12.0.0/patch/115/sql/pyusupdpa.sql
%s_current_base% %s_other_base%
###End for Quantum
Synchronization##
#End Customization
19. Update Profile settiing for Applaud
Software
a. XXAS: iCal Database Name ? APDTST/APDDEV
b. XXAS: iCal Override Email Address ?
MMaggiacomo@apdbablogspot.com
20. Modify the Profile option for EUL schema
ICX: Discoverer Default
End User Layer Schema Prefix to EUL4
Closure and Sanity Check
The environment must be
released to the user community with reference to the ticket number.
Instance Name
Discoverer Login
Internal URL login
Validate the quantum
sync parameters in adop_sync.drv
External URL login (if
applicable)
Workflow mailer Status
Active User concurrent
program concurrent program
Subscribe to:
Post Comments
(
Atom
)
No comments :
Post a Comment
Note: only a member of this blog may post a comment.