Oracle Application DBA

On Oracle Applications 12.2.4,
when attempting to open the status diagram,
the following error occurs:
[1953]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Exception in getHtml: java.lang.IllegalStateException: Cipher not initialized
[1954]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Complete Exception stack is:
[1957]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:java.lang.IllegalStateException: Cipher not initialized at javax.crypto.Cipher.checkCipherState(Cipher.java:1672)
The issue can be reproduced at will with the following steps:
1. Workflow Administrator Web Applications
2. Administrator Workflow
3. Status Monitor
CAUSE
The Workflow code wasn't able to open the URL for the workflow diagram due to SSL setting was missing.
The debug information in NewWorkflowError.docx shows the following error when trying to connect to the url:
[1505]:STATEMENT:[fnd.wf.WFBaseUtil]:Done openConnection [1505]:STATEMENT:[fnd.wf.WFBaseUtil]:String representation of urlConn: weblogic.net.http.SOAPHttpsURLConnection:https://<serverName>:<portNumber>/forms/frmservlet?appletmode=nonforms&HTMLpageTitle=&HTMLpreApplet=&code=oracle.apps.fnd.wf.Monitor&width=800&height=400&archive=/OA_JAVA/oracle/apps/fnd/jar/wfmon.jar,/OA_JAVA/oracle/apps/fnd/jar/fndewt.jar,/OA_JAVA/oracle/apps/fnd/jar/fndswing.jar,/OA_JAVA/oracle/apps/fnd/jar/fndbalishare.jar,/OA_JAVA/oracle/apps/fnd/jar/fndctx.jar...
1505]:STATEMENT:[fnd.wf.WFBaseUtil]:Done setUseCaches
[1953]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Exception in getHtml: java.lang.IllegalStateException: Cipher not initialized
[1954]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:Complete Exception stack is:
[1957]:STATEMENT:[fnd.wf.monitor.webui.GraphMonitorCO]:java.lang.IllegalStateException: Cipher not initialized at javax.crypto.Cipher.checkCipherState(Cipher.java:1672)
SOLUTION
To implement the solution, please execute the following steps:
1. Bring up the WLS Admin console for the EBS instance ->
http://host:port/console and login as the weblogic user.
2. Click on Lock & Edit.
3. Under the Domain Structure, navigate to EBS Domain -> Environment -> Servers.
4. Click on oacore_server1, and click on the SSL tab.
5. Click on Advanced to expand that section.
6. Look for the 'Use JSSE SSL' parameter and enable this by placing a check in the box next to it.
7. Click on Activate Changes.
8. Restart the managed server.
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oacore_server1
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server1
9. Retest the issue.
10. Migrate the solution as appropriate to other environments.

First How to check the status of Notification Mailer
a) Check workflow mailer service current status
sqlplus apps/
b) select running_processes
from fnd_concurrent_queues
where concurrent_queue_name = ‘WFMLRSVC’;
Number of running processes should be greater than 0
c) Find current mailer status
sqlplus apps/
select component_status
from fnd_svc_components
where component_id =
(select component_id
from fnd_svc_components
where component_name = ‘Workflow Notification Mailer’);
Possible values:
RUNNING
STARTING
STOPPED_ERROR
DEACTIVATED_USER
Now how to stop Notification Mailer from Backend
a) Stop notification mailer
sqlplus apps/
b) declare
p_retcode number;
p_errbuf varchar2(100);
m_mailerid fnd_svc_components.component_id%TYPE;
begin
— Find mailer Id
—————–
select component_id
into m_mailerid
from fnd_svc_components
where component_name = ‘Workflow Notification Mailer’;
————–
— Stop Mailer
————–
fnd_svc_component.stop_component(m_mailerid, p_retcode, p_errbuf);
commit;
end;
/
Now how to start Notification Mailer from Backend
a). Start notification mailer
sqlplus apps/
b) declare
p_retcode number;
p_errbuf varchar2(100);
m_mailerid fnd_svc_components.component_id%TYPE;
begin
— Find mailer Id
—————–
select component_id
into m_mailerid
from fnd_svc_components
where component_name = ‘Workflow Notification Mailer’;
————–
— Start Mailer
————–
fnd_svc_component.start_component(m_mailerid, p_retcode, p_errbuf);
commit;
end;
/

Outbound Notification Processing –
-Creation of Workflow notification raises oracle.apps.wf.notification.send event.
-This event is deferred immediately and placed on the WF_DEFERRED agent.
-Workflow Agent Listener on WF_DEFERRED queue process the deferred subscription.
-Subscription to this event calls the Generate function (WF_XML.Generate) to generate an XML representation of the notification.
-The event is then placed on the WF_NOTIFICATION_OUT agent.
-Workflow Notification Mailer dequeues the message from WF_NOTIFICATION_OUT.
-Before the e-mail notification is sent, Workflow Notification Mailer performs the following actions:
1. Resolves the notification recipient role to a single e-mail address, which itself can be a mail list.
2. Switches its database session to the recipient role’s preferred language and territory.
3. Generates the message and any optional attachments using the appropriate message template.
4. Sends the message by SMTP protocol.
Inbound Notification Processing
-This is followed If Inbound proccessing Checkbox is enabled for it in Workflow setup
-Email received into INBOX on IMAP Server
-Inbound Workflow Notification Mailer thread queries its IMAP e-mail account for incoming message.
-Use a Java-based e-mail parser to perform high level verification checks on the message.
-If the message is a valid notification response, create an XML message and place it on WF_NOTIFICATION_IN.
-Workflow Agent Listener runs on WF_NOTIFICATION_IN dequeues the message.
Workflow executes the appropriate response function to record the response and complete the notification.
Some important point about Workflow Notification Mailer issues
For workflow mailer to work variable AF_CLASSPATH and AFJVAPRG should be set correctly in adovars.env
-Override address feature can be used to test the mailer.By providing this address,mailer sent all the mail to this address only
-$FND_TOP/sql/Wfver.sql can be used to find the version of worflow.There are some more scripts in $FND_TOP/sql which can be used to debug the workflow items
Some useful queries about Workflow Notification Mailer issues
select count(*), MSG_PRIORITY, corr_id, msg_state from applsys.aq$wf_deferred
where corr_id like ‘%oracle.apps.wf.notification%’ group by MSG_PRIORITY, corr_id, msg_state;
select count(*), msg_state from applsys.aq$wf_notification_out group by msg_state
select status, mail_status, count(*) from wf_notifications group by status, mail_status;
select count(*), MSG_PRIORITY, corr_id, msg_state from applsys.aq$wf_error where corr_id like ‘APPS:oracle.apps.wf.notification%’ group by MSG_PRIORITY, corr_id, msg_state;
Log file location for Workflow Notification Mailer
Log files are located in $APPLCSF/$APPLLOG on the concurrent tier
Log file start with FNDCPGSC.  There are three files,One belongs to Notification mailer while other two belongs to Workflow Agent listener
ls -ltr FNDCPGSC*.txt
Troubleshooting Workflow Notification  Mailer issues
First of all here are the sanity check for Inbound and Outbound Processing
Verify Outbound Processing
1 Verify Agent Listeners are running
2 Go to the “View Details” screen for the Mailer you are configuring
3 Verify that Mailer is running
4 Click on the “Test Email” button
5 Enter a role using the LOV
6. The test address will override that role’s email address
7. Click Send, write down the Notification Id just in case
8. Verify your email account receives this email
Verify Inbound Processing
1 Send an e-mail to the Workflow Mailer reply to address
2 Subject “Hello World”
3 Body “Hello World”
4. The mailer should process this email and send back an FYI “unsolicited email” message to your email account
Now lets take some particular issues on Workflow Notification Mailer
A)  A particular user is not receiving the mail from Workflow mailer
We need to  check the user preferences
declare
recipient_role VARCHAR2(100);
display_name VARCHAR2(200);
email VARCHAR2(1000);
notification_pref VARCHAR2(100);
installed VARCHAR2(1);
language VARCHAR2(100);
territory VARCHAR2(100);
orig_system VARCHAR2(100);
orig_system_id number;
begin
recipient_role := ‘&username’;
WF_DIRECTORY.GetRoleInfoMail(recipient_role, display_name, email,
notification_pref,
language, territory,
orig_system, orig_system_id, installed);
dbms_output.put_line(‘display_name=’|| display_name);
dbms_output.put_line(’email=’|| email);
dbms_output.put_line(‘language=’|| language);
dbms_output.put_line(‘territory=’|| territory);
dbms_output.put_line(‘notification_pref=’|| notification_pref);
end;
/
The notification preferences determine if the notification will be emailed or not. When the notification preference is set to “Do not send me mail” then notifications are not emailed to that user.
The script in step#1 also returns the notification preference value. QUERY means “Do not send me mail”.
How to change the user’s notification preference to a value other than “Do not send me mail” ?
Navigation path:
a. Log into the application as the user with this problem
b. Preferences (at the TOP of the main page on your right hand side).
c. Notifications Section> Email Style field
B)  Outbound Processing Not working, No mails are being send from Notification Mailer
i)  Make sure Java mailer and WF Agent Listeners are up and running:
Navigation path:
a. Workflow Administrator web Applications Responsibility
b. Workflow Manager
c. Click on the Service Components icon
The following components should be up and running:
Workflow Notification Mailer
Workflow Deferred Agent Listener
ii) Are messages in WF_DEFERRED and WF_NOTIFICATION_OUT processed?
a. Execute the following queries:
select substr(wfd.corrid,1,40) corrid,
decode(wfd.state,
0, ‘0 = Ready’,
1, ‘1 = Delayed’,
2, ‘2 = Retained’,
3, ‘3 = Exception’,
to_char(substr(wfd.state,1,12))) State,
count(*) COUNT
from applsys.wf_deferred wfd
group by wfd.corrid, wfd.state;
b.
select substr(wfd.corrid,1,40) corrid,
decode(wfd.state,
0, ‘0 = Ready’,
1, ‘1 = Delayed’,
2, ‘2 = Retained’,
3, ‘3 = Exception’,
to_char(substr(wfd.state,1,12))) State,
count(*) COUNT
from applsys.wf_notification_out wfd
group by wfd.corrid, wfd.state;
c. Re-execute the same queries after 15-20 minutes of first execution and compare the number of messages in READY state/status. The number messages in READY state/status should decrease.
iii) Check the SMTP server
Here is the check
telnet <SMTP SERVER> 25 Enter
Trying 192.avc.ass….
Connected to <SMTP SERVER>.
Escape character is ‘^]’.
220 <SMTP SERVER> ESMTP Sendmail 8.13.8/8.13.8; Fri, 19 Mar 2016 15:4
0:07 -0400
EHLO <SMTP SERVER> Enter
250-<SMTP SERVER> Hello <SMTP SERVER> [192.127.240.8], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
250-VERB
250-8BITMIME
250-SIZE
250-ETRN
250-AUTH DIGEST-MD5 CRAM-MD5
250-DELIVERBY
250 HELP
MAIL FROM:workflow-<ORACLE_SID>@<SMTP SERVER> Enter From address
250 2.1.0 workflow-<ORACLE_SID>@<SMTP SERVER>… Sender ok
RCPT TO:abc@xyz.com Enter To address
250 2.1.5 abc@xyz.com… Recipient ok
DATA Enter
354 Enter mail, end with “.” on a line by itself
Test message from <SMTP server> Enter the message
. enter a . by itself on a new line
250 2.0.0 o2JJe7Ip016348 Message accepted for delivery
Quit Enter
221 2.0.0 <SMTP SERVER> closing connection
Connection closed by foreign host.
iv)
Run $FND_TOP/sql/wfmlrdbg.sql for a notification id and check the status of the message in WF_DEFERRED and WF_NOTIFICATION_OUT queue.
PROCESSED in WF_DEFERRED – The message is enqueued to WF_NOTIFICATION_OUT
PROCESSED in WF_NOTIFICATION_OUT – The message is sent as e-mail
READY in WF_DEFERRED – Check if Deferred Agent Listener is running
READY in WF_NOTIFICATION_OUT – Check if Notification Mailer is running
v)  Try rebuilding the WF_NOTIFICATION_OUT queue if necessary
The wfntfqup.sql script rebuilds the WF_NOTIFICATION_OUT queue by dropping and recreating
that queue, removing pending notification messages from the WF_DEFERRED queue, and
repopulating the WF_NOTIFICATION_OUT queue from the Oracle Workflow Notification System
tables.
Stop Notification Mailer, rebuild Mailer Queue using $FND_TOP/patch/115/sql/wfntfqup.sql.
C) Inbound Processing not working
i)  Check the IMAP server
$AFJVAPRG -classpath $AF_CLASSPATH -Dprotocol=imap -Dserver=<IMAP Server> -Dport=143 \
-Ddebug=Y \
-Daccount=workflow-<ORACLE_SID> -Dpassword=<workflow password> -Dconnect_timeout=120 \
-Ddbcfile=$FND_SECURE/<ORACLE_SID>.dbc -Dfolder=Inbox oracle.apps.fnd.wf.mailer.Mailer
ii) Check if expunge to inbox is checked or Not
iii) Login to the IMAP mailbox   and cleanup any spam email if present. It is recommended to regularly cleanup the Process and discard folder
iv) Create tag  for the autoreply and delivery failure notification to avoid the continuous sending of the mails
v) Ensure  the Mailer inbound processing has been enabled by setting the mailer parameter Inbound thread count=1 and ensure that Notification mailer is up and running.

Expense Report Approval Workflow shows Error ORA-04061 existing state of package body APPS.AP_WEB_EXPENSE_WF has been invalidated (Doc ID 1303929.1)
If there are no invalid objects related to Internet Expenses, then proceed to step #4.
4.    Stop and Restart the Workflow Services.
5.    Re-test your issue.

Below are some of the important Operating system level environment variables  related to R12.2.
$ echo $FILE_EDITION
<shows which file edition you are using, run or patch>
$ echo $RUN_BASE
<shows absolute path to run file system>
$ echo $PATCH_BASE
<shows absolute path to patch file system>
$ echo $NE_BASE
<shows absolute path to non-edition file system>
$ echo $APPL_TOP_NE
<non-editioned appl_top path. Equivalent to $NE_BASE/EBSapps/appl>
$ echo $LOG_HOME
<Application Instance Specific Log Directory>
$ echo $ADOP_LOG_HOME
<Online patching Specific Log Directory. Equivalent to $NE_BASE/EBSapps/log/adop>
$ echo $IAS_ORACLE_HOME
<FMW Web Tier Home Directory>
$ echo $ORACLE_HOME
< 10.1.2 ORACLE_HOME>
$ echo $CONTEXT_FILE
<Source for information populating template files (autoconfig)>
$ echo $EBS_DOMAIN_HOME
<WLS Deployment of Oracle E-Business Suite 12.2 Domain (instance specific)>
$ echo $ADMIN_SCRIPTS_HOME
<Shell scripts to control processes associated to the Applications Instance>
$ echo $EBS_ORACLE_HOME
<Oracle E-Business Suite 12.2 FMW Deployment directory>
$ echo $RW
<10.1.2 reports directory>
$ echo $HOSTNAME
<hostname without domain name>
$ echo $APPS_VERSION
<to get the EBS version>
$ ps -ef | grep "NodeManager"
$ ps -ef | grep "weblogic.Name=AdminServer"
$ ps -ef | grep "weblogic.Name=forms-c4ws_server"
$ ps -ef | grep "weblogic.Name=forms_server"
$ ps -ef | grep "weblogic.Name=oafm_server"
$ ps -ef | grep "weblogic.Name=oacore_server"

Both Primary (Run) and Secondary Edition (Patch) file systems will each contain these logs:
Main installation log:
$INST_TOP/logs/<MMDDHHMM>.log
FMW and OHS TechStack Installation/Patching logs:
$APPL_TOP/admin/$CONTEXT_NAME/log/installAppl.log
$APPL_TOP/admin/$CONTEXT_NAME/log/ebsfmwlayout.log
$APPL_TOP/admin/$CONTEXT_NAME/log/NetServiceHandler.log
$APPL_TOP/admin/$CONTEXT_NAME/log/adcvmlog.xml
$APPL_TOP/admin/$CONTEXT_NAME/log/executeac.log
$APPL_TOP/admin/$CONTEXT_NAME/log/deploytech.log
$APPL_TOP/admin/$CONTEXT_NAME/logs/wls1036.log
$APPL_TOP/admin/$CONTEXT_NAME/logs/oracle.apps.fnd.txk.default0.log
$APPL_TOP/admin/$CONTEXT_NAME/logs/oracle.apps.fnd.txk.install0.log
<Global Inventory>/logs/<timestamp>.log
<Global Inventory>/logs/cloneActions<timestamp>.log
<Global Inventory>/logs/install<timestamp>.log
<Global Inventory>/logs/installActions<timestamp>.log
<Global Inventory>/logs/installProfile<timestamp>.log
<Global Inventory>/logs/installSummary<timestamp>.log
<Global Inventory>/logs/OPatch<timestamp>.log
<Global Inventory>/logs/oraInstall<timestamp>.log
<Global Inventory>/logs/silentInstall<timestamp>.log
Forms Oracle Home installation logs:
$APPL_TOP/admin/$CONTEXT_NAME/log/ApplyAppsTechStack.log
$INST_TOP/admin/log/ohclone.log
$INST_TOP/admin/log/ApplyAppsTechStack_<MMDDHHMM>.log
Service Control logs :
$INST_TOP/logs/appl/admin/log/adadminsrvctl.txt
$INST_TOP/logs/appl/admin/log/adalnctl.txt
$INST_TOP/logs/appl/admin/log/adapcctl.txt
$INST_TOP/logs/appl/admin/log/adcmctl.txt
$INST_TOP/logs/appl/admin/log/adforms-c4wsctl.txt
$INST_TOP/logs/appl/admin/log/adformsctl.txt
$INST_TOP/logs/appl/admin/log/adnodemgrctl.txt
$INST_TOP/logs/appl/admin/log/adoacorectl.txt
$INST_TOP/logs/appl/admin/log/adoafmctl.txt
$INST_TOP/logs/appl/admin/log/adopmnctl.txt
$INST_TOP/logs/appl/admin/log/adstpall.log
$INST_TOP/logs/appl/admin/log/adstrtal.log
$INST_TOP/logs/appl/admin/log/jtffmctl.txt

Please backup and then clear out the following log files:
$IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager/nmHome1/nodemanager.log
$IAS_ORACLE_HOME/instances/EBS_web_<SID>_OHS1/diagnostics/logs/OHS/EBS_web_<SID>/EBS_web_<SID>.log
$IAS_ORACLE_HOME/instances/EBS_web_<SID>_OHS1/diagnostics/logs/OHS/EBS_web_<SID>/access_log
$EBS_DOMAIN_HOME/servers/oacore_server*/logs/oacore_server1.log
$EBS_DOMAIN_HOME/servers/oacore_server*/logs/oacore_server1.out
$EBS_DOMAIN_HOME/servers/oacore_server*/logs/access.log
$EBS_DOMAIN_HOME/servers/oacore_server*/logs/oacore_server1-diagnostic.log
$EBS_DOMAIN_HOME/servers/AdminServer/logs/access.log
$EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer-diagnostic.log
$EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log
$EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.out
$EBS_DOMAIN_HOME/servers/AdminServer/logs/EBS_domain_*.log

This is a list of all the log files generated throughout the installation process :

Before the installation starts:

$TMP/<MMDDHHMM>/<MMDDHHMM>.log
$TMP/<MMDDHHMM>/<hostname>_<SID>.xml
$TMP/<MMDDHHMM>/<hostname>_<SID>_apps.xml
The Rapidwiz Configuration File is saved in 4 locations:-
$TMP/<MMDDHHMM>/conf_<SID>.txt
Primary $INST_TOP/conf_<SID>.txt
Secondary $INST_TOP/conf_<SID>.txt
<RDBMS ORACLE_HOME>/appsutil/conf_<SID>.txt
Preinstall System Check logs:
$TMP/<MMDDHHMM>/<MMDDHHMM>.log
$TMP/dbPreInstCheck.xxxxx.log
$TMP/wtprechk.xxxx/wtprechk.xxxx.log
<Global Inventory>/logs/installActions<timestamp>.log
Database Tier:
Main installation log:
RDBMS $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>.log
Oracle Home installation logs:
RDBMS $ORACLE_HOME/temp/$CONTEXT_NAME/logs/oracle.apps.fnd.txk.install0.log
RDBMS $ORACLE_HOME/temp/$CONTEXT_NAME/logs/oracle.apps.fnd.txk.default0.log
<Global Inventory>/logs/<timestamp>.log
<Global Inventory>/logs/cloneActions<timestamp>.log
<Global Inventory>/logs/install<timestamp>.log
<Global Inventory>/logs/installActions<timestamp>.log
<Global Inventory>/logs/installProfile<timestamp>.log
<Global Inventory>/logs/installSummary<timestamp>.log
<Global Inventory>/logs/OPatch<timestamp>.log
<Global Inventory>/logs/oraInstall<timestamp>.log
<Global Inventory>/logs/silentInstall<timestamp>.log
RMAN restore scripts (Install mode only):
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-rac1.rman
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-rac2.sql
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-single2.sql
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-rac21.sql
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-single21.sql
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-rac3.rman
RMAN restore logs (Install mode only):
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-rac1.log
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-rac2.log
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-rac3.log
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-single1.log
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-single2.log
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/restore-single3.log
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/LogOff.sql
<RDBMS Oracle Home>/appsutil/out/$CONTEXT_NAME/fndClean.sql
Database characterset conversion (Install mode only):
<ORACLE_HOME>/appsutil/log/<Context Name>/addbccnv.txt
AutoConfig context file:
<RDBMS ORACLE_HOME>/appsutil/<Context Name>.xml
AutoConfig logs:
<RDBMS ORACLE_HOME>/appsutil/log/<Context Name>/<MMDDHHMM>/adconfig.log
<RDBMS ORACLE_HOME>/appsutil/log/<Context Name>/<MMDDHHMM>/NetServiceHandler.log
<RDBMS ORACLE_HOME>/temp/$CONTEXT_NAME/RACAutoConfig_nnnnn.log
Listener log:
<RDBMS ORACLE_HOME>/appsutil/log/$CONTEXT_NAME/addlnctl.txt
Applications Tier(s):
Both Primary (Run) and Secondary Edition (Patch) file systems will each contain these logs:
Main installation log:
$INST_TOP/logs/<MMDDHHMM>.log
FMW and OHS TechStack Installation/Patching logs:
$APPL_TOP/admin/$CONTEXT_NAME/log/installAppl.log
$APPL_TOP/admin/$CONTEXT_NAME/log/ebsfmwlayout.log
$APPL_TOP/admin/$CONTEXT_NAME/log/NetServiceHandler.log
$APPL_TOP/admin/$CONTEXT_NAME/log/adcvmlog.xml
$APPL_TOP/admin/$CONTEXT_NAME/log/executeac.log
$APPL_TOP/admin/$CONTEXT_NAME/log/deploytech.log
$APPL_TOP/admin/$CONTEXT_NAME/logs/wls1036.log
$APPL_TOP/admin/$CONTEXT_NAME/logs/oracle.apps.fnd.txk.default0.log
$APPL_TOP/admin/$CONTEXT_NAME/logs/oracle.apps.fnd.txk.install0.log
<Global Inventory>/logs/<timestamp>.log
<Global Inventory>/logs/cloneActions<timestamp>.log
<Global Inventory>/logs/install<timestamp>.log
<Global Inventory>/logs/installActions<timestamp>.log
<Global Inventory>/logs/installProfile<timestamp>.log
<Global Inventory>/logs/installSummary<timestamp>.log
<Global Inventory>/logs/OPatch<timestamp>.log
<Global Inventory>/logs/oraInstall<timestamp>.log
<Global Inventory>/logs/silentInstall<timestamp>.log
Forms Oracle Home installation logs:
$APPL_TOP/admin/$CONTEXT_NAME/log/ApplyAppsTechStack.log
$INST_TOP/admin/log/ohclone.log
$INST_TOP/admin/log/ApplyAppsTechStack_<MMDDHHMM>.log
Service Control logs :
$INST_TOP/logs/appl/admin/log/adadminsrvctl.txt
$INST_TOP/logs/appl/admin/log/adalnctl.txt
$INST_TOP/logs/appl/admin/log/adapcctl.txt
$INST_TOP/logs/appl/admin/log/adcmctl.txt
$INST_TOP/logs/appl/admin/log/adforms-c4wsctl.txt
$INST_TOP/logs/appl/admin/log/adformsctl.txt
$INST_TOP/logs/appl/admin/log/adnodemgrctl.txt
$INST_TOP/logs/appl/admin/log/adoacorectl.txt
$INST_TOP/logs/appl/admin/log/adoafmctl.txt
$INST_TOP/logs/appl/admin/log/adopmnctl.txt
$INST_TOP/logs/appl/admin/log/adstpall.log
$INST_TOP/logs/appl/admin/log/adstrtal.log
$INST_TOP/logs/appl/admin/log/jtffmctl.txt

[applint2@ebsx01appd10 webcache]$ find / -name *access*.log 2>/dev/null
/u15/home/ebsint2/fs2/EBSapps/appl/admin/ebsint2/log/US_fndacatfullaccess_ldt.log
/u15/home/ebsint2/fs2/EBSapps/10.1.2/j2ee/DevSuite/log/default-web-access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/oacore_server2/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/forms_server2/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/oafm_server3/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/oaea_server1/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/forms-c4ws_server1/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/oaea_server2/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/forms_server1/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/oacore_server1/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/forms-c4ws_server2/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/oafm_server2/logs/access.log
/u15/home/ebsint2/fs2/FMW_Home/user_projects/domains/EBS_domain_ebsint2_1442270252416/servers/oacore_server3/logs/access.log

I saw the particular user mail status in that all status was shows expired.
So I submitted "Move Messages from Exception to Normal Queue of Workflow Agent" concurrent request. It will ask parameters then we need to give WF component name in my case I was given WF_NOTIFICATION_OUT.
How To Move Notifications From Exception Queue To Normal Queue?  ID 472204.1
FNDWF_MOVE_MSGS_EXCEP2NORMAL "The agent is not found" [ID 760983.1]
Error APP-FND-1030 Is Displayed When Trying To Submit "Move messages from Exception to Normal Queue of Workflow Agent" FNDWF_MOVE_MSGS_EXCEP2NORMAL [ID 1451820.1]
For Enabling De-Queue we need run below procedure,
exec dbms_aqadm.start_queue(queue_name=>'APPLSYS.AQ$_WF_NOTIFICATION_OUT_E',
dequeue=>TRUE,enqueue=>FALSE)

Solution:
Don't use "localhost" as the name of the SMPT server name, use the physical name or IP address instead.
Solution:
This is a known issue which also occurred in Dev environment, We have followed these steps to fix it. Hence please co-ordinate with DBA team to fix the ACL error followed by SMTP configuration in EBS
Set the smtp_out_server parameter value to smtpvip.apdbablog.com at the database level
     Run the following pl/sql block from db server (as sysdba) to clear ACL related errors
BEGIN
dbms_network_acl_admin.assign_acl (
acl => '/sys/acls/OracleEBS.xml',
host => 'smtpvip.apdbablog.com',
lower_port => 25,
upper_port => 25
);
END;
/
    Set the Notification Mailer values as following.
a.Applications System:apdbadev  -->   Service Components -->   Edit 
b.Outbound EMail Account (SMTP) = smtpvip.apdbaidec.com
c.Inbound EMail Account (IMAP) = imap.apdbaidec.com
d.Username Username for the Inbound Server = svc-applpdhd
e.Reply-To Address  = apdbadev@apdba.com
f.Checked "Inbound SSL Enabled"
Asked the User to run the conc prog and check the email in APDBA Email box (smb-outlook-erp@apdba.com)
=========================
Run Below command on Database Tier.
exec dbms_network_acl_admin.create_acl(acl=>'OracleEBS.xml',description=>'OracleEBS.xml',
principal=>'APPS',privilege=>'connect',is_grant=>true,start_date=>null,end_date=>null);
exec DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE(acl=>'OracleEBS.xml',principal=>
'APPS',is_grant=>true,privilege=>'connect',position=>1,start_date=>null,end_date=>null);
exec DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE(acl=>'OracleEBS.xml',principal=>
'APPS',is_grant=>true,privilege=>'resolve',position=>2,start_date=>null,end_date=>null);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'ebsint2',lower_port=>80,upper_port=>80);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'ebsx01appd10',lower_port=>8080,upper_port=>8080);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'ebsx01appd11',lower_port=>8080,upper_port=>8080);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'ebsx01appd12',lower_port=>8080,upper_port=>8080);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'ebsx01appd13',lower_port=>8080,upper_port=>8080);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'ebsx01appd14',lower_port=>8080,upper_port=>8080);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'ebsx01appd15',lower_port=>8080,upper_port=>8080);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'dev2osbadminvhn.apdbablog.com',lower_port=>80,upper_port=>8040);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'dev3eip.apdbablog.com',lower_port=>80,upper_port=>80);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'dev4eip.apdbablog.com',lower_port=>80,upper_port=>80);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'dev4soaeip.apdbablog.com',lower_port=>80,upper_port=>80);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'smtpvip.apdbablog.com',lower_port=>25,upper_port=>25);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'dev2eip.apdbablog.com',lower_port=>25,upper_port=>25);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'dev2osbadminvhn',lower_port=>80,upper_port=>80);
exec DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(acl=>'OracleEBS.xml',host=>
'*',lower_port=>null,upper_port=>null);