Fusion Middleware in E-Business Suite 12.2

Agenda 
Overview of Architecture & components
Installation and configuration management
Administration Tools
Managing FMW components and Troubleshooting
Best Practices for maximizing performance

Advantages of FMW 
 Oracle WebLogic Server delivers best performance in the Middleware
Technology Stack
 Real time mentoring and management of the application
 Consolidated Administration, Configuration and Deployment.
 Powerful Troubleshooting framework
 High Availability, scalability & Failover
 Robust Platform for integration with other Fusion applications like SOA,
    Business Intelligence etc
 Single integrated , modular, design tool to build a complete application
     improving productivity
FMW Terminologies 
 Java components : Deployed Java EE Applications in WLS domain.  
   Managed by Admin Interfaces            
 System Components : Manageable process not deployed as Java
Application.  Managed by OPMN.
 Domain : Logically related group of Java components
 Admin Server  : Central control entity for the configuration of entire
domain
 Managed Servers :  Host business applications, application components,
Web services, and their associated resources
 Node Manager  :  Perform common operations for Managed Servers,
regardless of its location with respect to its Administration Server.
 Oracle Instance :  Contains one or more system components, such as
Oracle Web Cache, Oracle HTTP Server, or Oracle Internet Directory.
     
Important Directories and related configuration files  
EBS ORACLE_HOME
Oracle_EBS-app1
      |- applications                               EAR/WAR deployments
      |- deployment_plans                    Default deployment plans
      |- shared_libs                                FMW shared libraries

Domain Directory
User_projects
      |- domains
         |- EBS_domain_<SID>               EBS 12.2 domain directory
            |- config                                   Domain specific configuration (inc. config.xml)
            |- servers                                 Server = JVM, specific configuration, logs etc

Domain Config Directory
  <Domain_home>
    |- config
         |- config.xml                                  Central configuration file
         |- fmwconfig                                  FMW related configuration
         |- nodemanager                            Node Manager password properties

Domain Server Directory
  <Domain_Home>
 |- servers
      |- AdminServer
      |- forms-c4ws_server1
      |- forms_server1
      |- oacore_server1
                  |- logs                               Server and application logs (same per server)
      |- oafm_server1

WebTier Instance Home
|- webtier
   |- instances
      |- EBS_web_<context>                         EBS OHS Home
         |- bin                                                     opmnctl
         |- config
                |- OHS
                        |- EBS_web_component     Apache conf files
                |- OPMN
                         |- opmn                                 opmn.xml
          |- diagnostics
               |- logs
                    |- OHS
                          |- EBS_web_component    Apache & OHS log files
                    |- OPMN
                          |- opmn                                 OPMN logs

Installation and configuration management 
12.2 Rapid Install FMW configuration scripts

 adrunfmw.sh / adrunfmw.cmd : Installs  FMW Techstack and Weblogic Server.
 The adProvisionEBS.pl :  Creates the Domain , Managed server and start services
     Executed 3 times in succession using the options below:
     1. ebs-execute-ebsfmwlayout: Deploy the EBS-specific file system in $FMW_HOME
     2. ebs-execute-ac: executes AutoConfig
     3. ebs-deploy-alltech: creates a new domain and a new OHS component
  The required parameter values for each option are retrieved from the context file.
  Log files
     Main Installation Log: $INST_TOP/logs/<MMDDHHMM>.log
     FMW Installation Logs: $APPL_TOP/admin/$CONTEXT_NAME/log/
           Refer: Troubleshooting Rapid Install for E-Business Suite Release 12.2 (Doc ID 1378579.1)

Changes in Autoconfig 
Service Groups
New Service Groups added:
   Web Administration service group  :
 Contains WebLogic Administration server , enabled only on one of the Application tier nodes.
Changes to existing Service Groups .
 Root Service Group :
 Now comprises Node Manager and not Oracle Process Manager (OPMN).
 Web Entry Point Services service :
 Contains HTTP Server managed by OPMN
 Web Application services :
 Consist of  Manager server applications oacore, oafm, forms and forms-c4ws.

FMW Service Control Scripts
 Located at $ADMIN_SCRIPTS_HOME.
 Various FMW Service Scripts are:
   - adopmnctl.sh <start/stop>
   - adnodemgrctl.sh <start/stop>
   - adadminsrvctl.sh <start/stop>
   - admanagedsrvctl.sh <start/stop> oacore_server[n] / oafm_server[n] /forms_server[n]/ forms-c4ws_server[n] .
  Where [n] indicates the corresponding Managed server instance.
 adadminsrvctl.sh and adnodemgrctl.sh call the adProvisionEBS.pl  script internally to manage all the EBS


Sample options available:
  ebs-create-node/ebs-delete-node
  ebs-execute-ac
  ebs-enable-servicegroup
  ebs-enable-service/ebs-disable-service
  ebs-deploy-tech/ebs-delete-tech
  ebs-create-managedserver/ebs-delete-managedserver
  ebs-start-nm/ebs-stop-nm
  ebs-nmstart-adminsrv/ ebs-nmstop-adminsrv
  ebs-nmstart-managedsrv/ ebs-nmstop-managedsrv
  ebs-start-midtierservices/ebs-stop-midtierservices
  ebs-deploy-app/ebs-deploy-app/ebs-redeploy-app

Autoconfig Context variables for FMW 
 Admin Server :
     s_wls_admin_host ,  s_wls_admin_user , s_wls_adminport , s_adminservername,  s_adminservertimeout , s_adminserverlog,
s_adminserverstatus, s_adminserverctrl
 Managed Servers:
 Managed Server Nodes: s_<msname>_nodes
     Managed Server Names: s_<msname>_managed_servers
     Ports: s_wls_<msname>port
     JVM parameters: s_<msname>_jvm_start_options
                                      [ Where msname is oacore, forms, oafm and forms-c4ws ]
 OHS configuration: s_ohs_instance_loc, s_ohs_instance, s_ohs_component
 Node managers:
      s_nmport, s_nmtype, s_nodemanagername, s_nodemanagertimeout, s_nodemanagerstatus, s_nodemanagerlog,
s_nodemanagerctrl,  s_nm_jvm_startup_properties
 Others: s_fmw_home, s_apps_deployment_dir
System Configuration Syncing 
 Not all configurations are managed by autoconfig in 12.2.
     Some are managed natively via FMW Control and WLS Admin Console.
 Couple of tools used to synchronize the configurations:
 The SyncContext tool : used for explicit synchronization of the context  variables with the WLS configuration parameters.
  This tool can be run on all application tier nodes as follows:
  $ perl <AD_TOP>/bin/adSyncContext.pl contextfile=<CONTEXT_FILE>
  The Node Manager and WebLogic Admin Server must be running during this execution.
 The adRegisterWLSListeners.pl script :  background process  , listens for changes to WLS configuration parameters in order
to facilitate synchronization with relevant context variables.
- Does not listen changes to the HTTP Server configuration parameters.
- Starts/ Stops automatically  when WebLogic Administration Server is started/ shut down.
      - On Windows start the tool manually
Administration Tools
 WebLogic Administration Console
    Enterprise Manager 11g  FMW Control
    WLST (WebLogic Scripting Tool)
WebLogic Adminisntartion Console 
 Web browser-based, graphical user interface to manage an Oracle WebLogic Server domain
 Accessed using URL:  http://[hostname]:[Admin Server port]/console
     [ Admin Server Port changes based on whether fs1 or fs2 is Run edition ]
 Use the Administration Console to:
     Configure, start, and stop WebLogic Server instances
    Configure WebLogic Server clusters
    Configure WebLogic Server services, such as database connectivity (JDBC) and  messaging (JMS)
    Configure security parameters, including managing users, groups, and roles
    Configure and deploy your applications
    Monitor server and application performance
    View server and domain log files
    View application deployment descriptors
    Edit selected run-time application deployment descriptor elements
Enterprise Manager 11g  FMW Control 
 Web application used to monitor and administer a farm
 A farm is a collection of components managed by Fusion Middleware Control.
 Contains Oracle WebLogic Server domains, one Administration Server, one or more Managed Servers, clusters, and
one or more Oracle instances
 You access Fusin Middleware Control using the below URL:
    http://hostname.domain:port/em
    The Port number can be obtained from s_wls_adminport  context variable and also listed in the file:
DOMAIN_HOME/config/config.xml.
Differences b/w FMW control and WLS console 
• WLS console is mainly used for managing java components
– Create /Clone/ Cluster/Start/Stop  Managed servers.
– Manage Datasources and connection pools
– Deploy java EE applications
– Change passwords.
– View and Manage Log files related to WLS
– Change Ports for Weblogic server and Java components.
• FMW control is mainly used for managing System components.
– Manage Oracle HTTP Server, Oracle Web Cache, Start and stop components, Start and stop applications
–  Deploy/Monitor SOA Composite applications like SOA, BPEL etc
– Start and Stop managed Servers
– Deploy Java EE applications.
– View and Manage most of the Log files.
– Change Ports  for System components
   
WLST (WebLogic Scripting Tool) 
 Command-line scripting environment based on the Java scripting interpreter, Jython.
 Use to create, manage, and monitor WebLogic Server domains.
 Invoke WLST interactively from the WLS home as below:
  java weblogic.WLST
  connect('username', 'password', 'localhost:7001')
  Online mode – equivalent to the WLS Admin Console, Connected to a running server
  Offline mode – when Domain not running
       Invoked in
 Interactive Mode - interactively, on the command line
     Script Mode - in batches, supplied in a file
      Embedded Mode - embedded in Java code (eg AD control scripts)
 The WLS command-line tools are useful:
      – For automating common administration activities
      – As an alternative to the Administration Console
      – When graphical tools are not supported
 Do not use the WLST script in the WebLogic Server home as the environment and classpath settings will not be set
properly for specific components.
 For managing Oracle HTTP Server  invoke WLST from
    <ORACLE_HOME for HTTP>/common/bin/wlst.sh
 For managing System Components invoke WLST from
    <ORACLE_HOME_for_component>/common/bin/wlst.sh

Managing, Troubleshooting, & Best Practices 
Agenda 
Managing HTTP Server Configurations
Managing Managed Server Configurations
Fusion Middleware Diagnostics Framework (FMWDFW)
Fusion Middleware Logging
Best Practices for maximizing performance

Managing HTTP Server Configurations 
Role of Autoconfig
• AutoConfig manages only initial setup of HTTP Server configuration
• Later, it can optionally be used to manage and customize a limited set of
configuration files like those related to SSL & DMZ setup
• Recommendation is to use native FMW tools to manage HTTP Server
configuration post-installation
• When updating seeded HTTP configuration:
–  Must be updated in both Context File as well as Configuration files

Updating seeded configuration
• On the Run Edition File System:
– Edit the relevant parameter from the Oracle Enterprise Manager console
– Run the following command on all application tier nodes:
 $ perl <AD_TOP>/bin/adSyncContext.pl contextfile=<CONTEXT_FILE>
 Important: Node Manager and WebLogic Admin Server must be running during execution of the
adSyncContext.pl script
– Run AutoConfig on all application tier nodes

Managing Managed Server Configurations 
Updating Managed Server Configuration
• The basic configurations are maintained in their respective deployment
plans located at <EBS_ORACLE_HOME>/deployment_plans/<managed
server>/plan.xml
• Delivered as an AutoConfig template, containing limited set of context
variables located at $FND_TOP/admin/template
    Ex: oacore_plan_FMW_xml.tmp
• AutoConfig only updates the plan.xml file if any context variables used in
the deployment plan have been customized.
• Most of the other parameters are updated via WLS Admin Console
Additional Steps Needed on Multi-Node Systems
• For multi-node system, manually update the deployment plans on the
other nodes
• Configuration changes made via WLS Console need to be synchronized with
other nodes
• Follow the steps below to synchronize the deployment plans on the other
nodes.
–  Edit the relevant deployment plan to enter the new configuration value
–  Save the deployment plan
–  Restart the managed server

Managing Classpath and JVM arguments – Command Line
• In addition, these properties can also be set from the backend using
adProvisionEBS.pl script as follows:
  $ perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl /
      ebs-set-managedsrvproperty -contextfile=<CONTEXT_FILE> /
      -managedsrvname=<MANAGED SERVER NAME>  /
      -managedsrvclasspath="<COMPLETE MANAGED SERVER
 CLASSPATH/JVM ARGUMENTS>”
Customizing the Number of Instances
• By default, every application tier node contains only a single instance of the
managed servers
• Managed server creation/deletion should be done only through the
adProvisionEBS.pl and txkSetAppsConf.pl scripts
• WebLogic Administration Console should not be used
• Addition/deletion of managed servers needs to be done on the
–  Run Edition File System when there is no active ADOP cycle
Adding a Managed Server Instance 
• Execute the following command to add a new managed server
• This will create a managed server and add a new entry to the context file:
    $ perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl \
   ebs-create-managedserver -contextfile=<CONTEXT_FILE> \
   -managedsrvname=<MANAGED_SERVER_NAME> \
   -servicetype=<SERVICE_TYPE> \
   -managedsrvport=<MANAGED_SERVER_PORT> -logfile=<LOGFILE>
–  Name must be of the form <SERVICE_TYPE>_server<n>, where n is an integer
–  Managed Server port must be unique even across the Run and Patch edition
 Add the newly added managed servers into the OHS configuration files
mod_wl_ohs.conf and apps.conf using the below steps:
 Follow it on all application tier nodes participating in the same cluster:
–  Source the Run Edition File System
–  Execute the following command:
 $ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl \
 -contextfile=<COMPLETE PATH TO CONTEXT_FILE> \
   -configoption=addMS -oacore=<host>.<domain>:<port> -oafm=<host>.<domain>:<port> \
-forms=<host>.<domain>:<port> -formsc4ws=<host>.<domain>:<port>
–  Restart HTTP server using adapcctl.sh script
Removing Managed Server Instance 
 Process is similar to adding the managed server
 Done using the adProvisionEBS.pl and txkSetAppsConf.pl scripts
 Execute adProvisionEBS.pl with option ebs-delete-managedserver
 This will delete the managed server, and also update the respective context
variables
 Run txkSetAppsConf.pl, Using the option configoption=removeMS
 This removes details of the deleted managed servers from the OHS
configuration files mod_wl_ohs.conf and apps.conf
 Restart HTTP server using adapcctl.sh script
Changing the Managed Server Ports 
 Source the Run Edition File System.
 Execute the txkSetAppsConf.pl script with configoption=removeMS to
delete references of the old port in mod_wl_ohs.conf and apps.conf files.
 Execute the txkSetAppsConf.pl script with configoption=addMS to add back
the managed server entry in the OHS configuration files with the new port.
 For more details, Please refer to the below MOS document:
 Managing Configuration of Oracle HTTP Server and Web Application
Services in Oracle E-Business Suite Release 12.2 (Doc ID 1905593.1)
Fusion Middleware Diagnostics Framework (FMWDFW) 
Incident Detection
 Provides features designed to aid in detecting, diagnosing, and resolving
problems.
 On detection of a critical error, FMWDFW will automatically capture a set
of diagnostics and include them in an incident.
 Incidents are automatically detected in two ways:
– Incident detection log filter - to detect critical errors.
– WLDF Watch and Notification component (FMWDFW ) - listens for a predefined
notification type and creates Incidents
 FMWDFW provides option to enable additional logging or alter default
Automatic Diagnostic Repository (ADR)
 Each incident is assigned a number to help with tracking and reporting
 Incident data is stored in the file-based Automatic Diagnostic
Repository (ADR)
 Includes the diagnostic dumps and associated metadata
 Incidents can be packaged as a zip file, using the ADR Command
Interpreter (ADRCI), and sent to Oracle for analysis
 For more details, Please refer to the below MOS document:
 Using Oracle Fusion Middleware Diagnostic Framework With Oracle E-
Business Suite Release 12.2 (Doc ID 1428056.1)
Best Practices for maximizing performance 
Oacore JVM Configuration
 Default oacore JVM heap size is 512 MB (roughly supports 50 users)
 We recommend to configure 2 GB JVM Heap Space for oacore.
–  This can roughly support 150 to 200 users depending on usage.
 For higher user load  additional oacore managed servers need to be
created.
 We recommend to have 1 managed server per 2 CPU.
 For best response time results, use multiple managed instances.
 Note: You should always size your systems based on tests using
representative data and workloads for your own environment.
Modifying the JVM properties for Adminserver
 When adding more managed servers, it is essential to increase the Admin
Server memory, Else Adminserver will fail with OutofMemory er ror.
 Steps for modifying Adminserver memory:
– Stop the Admin Server using adadminsrvctl.sh script
– Modify s_nm_jvm_startup_properties context variable.
– Default value is 512 MB.
– Run AutoConfig on application nodes.
– Start the Admin Server using adadminsrvctl.sh script.
Tuning Tips for Heap Sizes
 Make sure adequate Physical memory available on the server when you
tune heap space
 If this value is exceeded, the OS starts paging and performance degrades
significantly.
 The Virtual Memory (VM) always uses more memory than the heap size.
 In production environments, if you find constant growth and shrink in heap
space set the minimum heap size and the maximum heap size to the same
value.
Recommended Performance Patches 
 Below are the 3 critical connection leak bug fixes which improves
performance and stability considerably:
– 19782999:R12.FWK.C  - R: CONNECTION LEAK IN OAERRORPAGE.JSP  AND  
OAERRORDETAILPAGE.JSP
– 19807163:R12.OAM.C  - 1OFF:12.2.3:TRANSACTIONSCOPE -  CONNECTION LEAK IN
12.2.3
– 9494816:R12.FND.C  - JDBC CONNECTION LEAK IN
 ORACLE.APPS.FND.COMMON.ERRORSTACK
 It is also recommended to apply the latest TXK and AD patches following
the note : Applying the Latest AD and TXK Release Update Packs to Oracle
E-Business Suite Release 12.2 (Doc ID 1617461.1)
Useful Documents 
 WLS 10.3.6Documentation Library
 http://docs.oracle.com/cd/E23943_01/wls.htm
 EBS 12.2 Documentation Library
 http://docs.oracle.com/cd/E26401_01/index.htm
 Some important 12.2 MOS Documents
– Useful 12.2 Documents For Customers And ATG Support Engineers (Doc ID 585889.1).
– Managing Configuration of Oracle HTTP Server and Web Application  Services in
Oracle E-Business Suite Release 12.2 (Doc ID 1905593.1)
– Using Oracle Fusion Middleware Diagnostic Framework With Oracle E-Business Suite
Release 12.2 (Doc ID 1428056.1)
Some important 12.2 MOS Documents (cont..)
– How To Detect a Connection Leak Using Diagnostic JDBC Dumps (Doc ID 1502054.1)
– 12.2 Ebusiness Suite - Collecting Fusion Middleware Log Files (Doc ID 1362900.1)
– Oracle Applications E-Business Suite 12.2 Fusion Middleware Log Files: Locate,View,
and Control (Doc ID 1366187.1)
– Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite
Release 12.2 (Doc ID 1617461.1)
– Oracle E-Business Suite Release 12.2: Upgrade Sizing and Best Practices
 (Doc ID 1597531.1)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.  |
Oracle Confidential – Internal/Restricted/Highly Restricted 53