Oracle Application DBA

R12.2 Apps DBA. Powered by Blogger.

Featured Posts

Enable or Disable Archive log mode in RAC

No comments :
Enable Archivelog
1)  Login to one of the nodes , verify the archive log mode
SQL> archive log list
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 200
Current log sequence 201
SQL> select name, log_mode from v$database;
NAME LOG_MODE
--------- ------------
PUB NOARCHIVELOG
2) Disable the cluster instance parameter by setting cluster_database to FALSE from the current instance:
sqlplus "/ as sysdba"
SQL> alter system set cluster_database=false scope=spfile sid='orcl1;
3) Shutdown all instances accessing the clustered database:
  srvctl stop database -d orcl
4)Using the local instance, MOUNT the database:
sqlplus "/ as sysdba"
SQL> startup mount
5) Enable archiving:
SQL> alter database archivelog;
6) Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE from the current instance:
SQL> alter system set cluster_database=true scope=spfile sid='orcl1;
7) Shutdown the local instance:
SQL> shutdown immediate
8) Bring all instance back up using srvctl:
srvctl start database -d orcl
9)Login to the local instance and verify Archive Log Mode is enabled:
sqlplus "/ as sysdba"
SQL> archive log list
Database log mode            Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     200
Next log sequence to archive  201
After enabling Archive Log Mode, each instance in the RAC configuration can automatically archive redologs.
Disable Archivelog
1) export oracle_sid=orcl1
SQL> alter system set cluster_database=false scope=spfile sid='orcl1';
System altered.
2) srvctl stop database -d orcl -o immediate
3) SQL> startup mount
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 18
Next log sequence to archive 19
Current log sequence 19
SQL> alter database noarchivelog;
Database altered.
SQL> alter system set cluster_database=true scope=spfile sid='orcl1';
System altered.
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
4) srvctl start database -d orcl
5) export oracle_sid=orcl1
sqlplus / as sysdba
SQL> archive log list
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 18
Current log sequence 19
SQL>

Data Guard Physical Standby Setup in Oracle Database

No comments :

Primary Server Setup

Logging

Check that the primary database is in archivelog mode.
SELECT log_mode FROM v$database;

LOG_MODE
------------
NOARCHIVELOG

SQL>
If it is noarchivelog mode, switch is to archivelog mode.
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE ARCHIVELOG;
ALTER DATABASE OPEN;
Enabled forced logging by issuing the following command.
ALTER DATABASE FORCE LOGGING;

Initialization Parameters

Check the setting for the DB_NAME and DB_UNIQUE_NAME parameters. In this case they are both set to "DB11G" on the primary database.
SQL> show parameter db_name

NAME         TYPE  VALUE
------------------------------------ ----------- ------------------------------
db_name         string  DB11G

SQL> show parameter db_unique_name

NAME         TYPE  VALUE
------------------------------------ ----------- ------------------------------
db_unique_name        string  DB11G

SQL>
The DB_NAME of the standby database will be the same as that of the primary, but it must have a different DB_UNIQUE_NAME value. The DB_UNIQUE_NAME values of the primary and standby database should be used in the DG_CONFIG setting of the LOG_ARCHIVE_CONFIG parameter. For this example, the standby database will have the value "DB11G_STBY".
ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(DB11G,DB11G_STBY)';
Set suitable remote archive log destinations. In this case I'm using the fast recovery area for the local location, but you could specify an location explicitly if you prefer. Notice the SERVICE and the DB_UNIQUE_NAME for the remote location reference the standby location.
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY';
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;
The LOG_ARCHIVE_FORMAT and LOG_ARCHIVE_MAX_PROCESSES parameters must be set to appropriate values and the REMOTE_LOGIN_PASSWORDFILE must be set to exclusive.
ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE;
ALTER SYSTEM SET LOG_ARCHIVE_MAX_PROCESSES=30;
ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE SCOPE=SPFILE;
In addition to the previous setting, it is recommended to make sure the primary is ready to switch roles to become a standby. For that to work properly we need to set the following parameters. Adjust the *_CONVERTparameters to account for your filename and path differences between the servers.
ALTER SYSTEM SET FAL_SERVER=DB11G_STBY;
--ALTER SYSTEM SET DB_FILE_NAME_CONVERT='DB11G_STBY','DB11G' SCOPE=SPFILE;
--ALTER SYSTEM SET LOG_FILE_NAME_CONVERT='DB11G_STBY','DB11G'  SCOPE=SPFILE;
ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;
Remember, some of the parameters are not modifiable, so the database will need to be restarted before they take effect.

Service Setup

Entries for the primary and standby databases are needed in the "$ORACLE_HOME/network/admin/tnsnames.ora" files on both servers. You can create these using the Network Configuration Utility (netca) or manually. The following entries were used during this setup.
DB11G =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ol5-112-dga1)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = DB11G.WORLD)
    )
  )

DB11G_STBY =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ol5-112-dga2)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = DB11G.WORLD)
    )
  )

Backup Primary Database

If you are planning to use an active duplicate to create the standby database, then this step is unnecessary. For a backup-based duplicate, or a manual restore, take a backup of the primary database.
$ rman target=/

RMAN> BACKUP DATABASE PLUS ARCHIVELOG;

Create Standby Controlfile and PFILE

Create a controlfile for the standby database by issuing the following command on the primary database.
ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/db11g_stby.ctl';
Create a parameter file for the standby database.
CREATE PFILE='/tmp/initDB11G_stby.ora' FROM SPFILE;
Amend the PFILE making the entries relevant for the standby database. I'm making a replica of the original server, so in my case I only had to amend the following parameters.
*.db_unique_name='DB11G_STBY'
*.fal_server='DB11G'
*.log_archive_dest_2='SERVICE=db11g ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G'

Standby Server Setup (Manual)

Copy Files

Create the necessary directories on the standby server.
$ mkdir -p /u01/app/oracle/oradata/DB11G
$ mkdir -p /u01/app/oracle/fast_recovery_area/DB11G
$ mkdir -p /u01/app/oracle/admin/DB11G/adump
Copy the files from the primary to the standby server.
$ # Standby controlfile to all locations.
$ scp oracle@ol5-112-dga1:/tmp/db11g_stby.ctl /u01/app/oracle/oradata/DB11G/control01.ctl
$ cp /u01/app/oracle/oradata/DB11G/control01.ctl /u01/app/oracle/fast_recovery_area/DB11G/control02.ctl

$ # Archivelogs and backups
$ scp -r oracle@ol5-112-dga1:/u01/app/oracle/fast_recovery_area/DB11G/archivelog /u01/app/oracle/fast_recovery_area/DB11G
$ scp -r oracle@ol5-112-dga1:/u01/app/oracle/fast_recovery_area/DB11G/backupset /u01/app/oracle/fast_recovery_area/DB11G

$ # Parameter file.
$ scp oracle@ol5-112-dga1:/tmp/initDB11G_stby.ora /tmp/initDB11G_stby.ora

$ # Remote login password file.
$ scp oracle@ol5-112-dga1:$ORACLE_HOME/dbs/orapwDB11G $ORACLE_HOME/dbs
Notice, the backups were copied across to the standby server as part of the FRA copy. If your backups are not held within the FRA, you must make sure you copy them to the standby server and make them available from the same path as used on the primary server.

Start Listener

Make sure the listener is started on the standby server.
$ lsnrctl start

Restore Backup

Create the SPFILE form the amended PFILE.
$ export ORACLE_SID=DB11G
$ sqlplus / as sysdba

SQL> CREATE SPFILE FROM PFILE='/tmp/initDB11G_stby.ora';
Restore the backup files.
$ export ORACLE_SID=DB11G
$ rman target=/

RMAN> STARTUP MOUNT;
RMAN> RESTORE DATABASE;

Create Redo Logs

Create online redo logs for the standby. It's a good idea to match the configuration of the primary server.
ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL;
ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/DB11G/online_redo01.log') SIZE 50M;
ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/DB11G/online_redo02.log') SIZE 50M;
ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/DB11G/online_redo03.log') SIZE 50M;
ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;
In addition to the online redo logs, you should create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs should be at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs. In my case, the following standby redo logs must be created on both servers.
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo01.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo02.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo03.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo04.log') SIZE 50M;
Once this is complete, we can start the apply process.

Standby Server Setup (DUPLICATE)

Copy Files

Create the necessary directories on the standby server.
$ mkdir -p /u01/app/oracle/oradata/DB11G
$ mkdir -p /u01/app/oracle/fast_recovery_area/DB11G
$ mkdir -p /u01/app/oracle/admin/DB11G/adump
Copy the files from the primary to the standby server.
$ # Standby controlfile to all locations.
$ scp oracle@ol5-112-dga1:/tmp/db11g_stby.ctl /u01/app/oracle/oradata/DB11G/control01.ctl
$ cp /u01/app/oracle/oradata/DB11G/control01.ctl /u01/app/oracle/fast_recovery_area/DB11G/control02.ctl

$ # Parameter file.
$ scp oracle@ol5-112-dga1:/tmp/initDB11G_stby.ora /tmp/initDB11G_stby.ora

$ # Remote login password file.
$ scp oracle@ol5-112-dga1:$ORACLE_HOME/dbs/orapwDB11G $ORACLE_HOME/dbs

Start Listener

When using active duplicate, the standby server requires static listener configuration in a "listener.ora" file. In this case I used the following configuration.
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = DB11G.WORLD)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)
      (SID_NAME = DB11G)
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ol5-112-dga2.localdomain)(PORT = 1521))
    )
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

ADR_BASE_LISTENER = /u01/app/oracle
Make sure the listener is started on the standby server.
$ lsnrctl start

Create Standby Redo Logs on Primary Server

The DUPLICATE command automatically creates the standby redo logs on the standby. To make sure the primary database is configured for switchover, we must create the standby redo logs on the primary server.
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo01.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo02.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo03.log') SIZE 50M;
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo04.log') SIZE 50M;

Create Standby Using DUPLICATE

Start the auxillary instance on the standby server by starting it using the temporary "init.ora" file.
$ export ORACLE_SID=DB11G
$ sqlplus / as sysdba

SQL> STARTUP NOMOUNT PFILE='/tmp/initDB11G_stby.ora';
Connect to RMAN, specifying a full connect string for both the TARGET and AUXILLARY instances. DO not attempt to use OS authentication.
$ rman TARGET sys/password@DB11G AUXILIARY sys/password@DB11G_STBY
Now issue the following DUPLICATE command.
DUPLICATE TARGET DATABASE
  FOR STANDBY
  FROM ACTIVE DATABASE
  DORECOVER
  SPFILE
    SET db_unique_name='DB11G_STBY' COMMENT 'Is standby'
    SET LOG_ARCHIVE_DEST_2='SERVICE=db11g ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G'
    SET FAL_SERVER='DB11G' COMMENT 'Is primary'
  NOFILENAMECHECK;
A brief explanation of the individual clauses is shown below.
  • FOR STANDBY: This tells the DUPLICATE command is to be used for a standby, so it will not force a DBID change.
  • FROM ACTIVE DATABASE: The DUPLICATE will be created directly from the source datafile, without an additional backup step.
  • DORECOVER: The DUPLICATE will include the recovery step, bringing the standby up to the current point in time.
  • SPFILE: Allows us to reset values in the spfile when it is copied from the source server.
  • NOFILENAMECHECK: Destination file locations are not checked.
Once the command is complete, we can start the apply process.

Start Apply Process

Start the apply process on standby server.
# Foreground redo apply. Session never returns until cancel. 
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE;

# Background redo apply. Control is returned to the session once the apply process is started.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
If you need to cancel the apply process, issue the following command.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
If you prefer, you can set a delay between the arrival of the archived redo log and it being applied on the standby server using the following commands.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DELAY 30 DISCONNECT FROM SESSION;

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT FROM SESSION;
Provided you have configured standby redo logs, you can start real-time apply using the following command.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE;

Test Log Transport

On the primary server, check the latest archived redo log and force a log switch.
ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';

SELECT sequence#, first_time, next_time
FROM   v$archived_log
ORDER BY sequence#;

ALTER SYSTEM SWITCH LOGFILE;
Check the new archived redo log has arrived at the standby server and been applied.
ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';

SELECT sequence#, first_time, next_time, applied
FROM   v$archived_log
ORDER BY sequence#;

Protection Mode

There are three protection modes for the primary database:
  • Maximum Availability: Transactions on the primary do not commit until redo information has been written to the online redo log and the standby redo logs of at least one standby location. If no standby location is available, it acts in the same manner as maximum performance mode until a standby becomes available again.
  • Maximum Performance: Transactions on the primary commit as soon as redo information has been written to the online redo log. Transfer of redo information to the standby server is asynchronous, so it does not impact on performance of the primary.
  • Maximum Protection: Transactions on the primary do not commit until redo information has been written to the online redo log and the standby redo logs of at least one standby location. If not suitable standby location is available, the primary database shuts down.
By default, for a newly created standby database, the primary database is in maximum performance mode.
SELECT protection_mode FROM v$database;

PROTECTION_MODE
--------------------
MAXIMUM PERFORMANCE

SQL>
The mode can be switched using the following commands. Note the alterations in the redo transport attributes.
-- Maximum Availability.
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby AFFIRM SYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY';
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY;

-- Maximum Performance.
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY';
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE;

-- Maximum Protection.
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby AFFIRM SYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY';
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PROTECTION;
ALTER DATABASE OPEN;

Database Switchover

A database can be in one of two mutually exclusive modes (primary or standby). These roles can be altered at runtime without loss of data or resetting of redo logs. This process is known as a Switchover and can be performed using the following statements.
-- Convert primary database to standby
CONNECT / AS SYSDBA
ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY;

-- Shutdown primary database
SHUTDOWN IMMEDIATE;

-- Mount old primary database as standby database
STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
On the original standby database issue the following commands.
-- Convert standby database to primary
CONNECT / AS SYSDBA
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

-- Shutdown standby database
SHUTDOWN IMMEDIATE;

-- Open old standby database as primary
STARTUP;
Once this is complete, test the log transport as before. If everything is working fine, switch the primary database back to the original server by doing another switchover. This is known as a switchback.

Failover

If the primary database is not available the standby database can be activated as a primary database using the following statements.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;
ALTER DATABASE ACTIVATE STANDBY DATABASE;
Since the standby database is now the primary database it should be backed up immediately.
The original primary database can now be configured as a standby. If Flashback Database was enabled on the primary database, then this can be done relatively easily (shown here). If not, the whole setup process must be followed, but this time using the original primary server as the standby.

Flashback Database

It was already mentioned in the previous section, but it is worth drawing your attention to Flashback Database once more. Although a switchover/switchback is safe for both the primary and standby database, a failover renders the original primary database useless for converting to a standby database. If flashback database is not enabled, the original primary must be scrapped and recreated as a standby database.
An alternative is to enable flashback database on the primary (and the standby if desired) so in the event of a failover, the primary can be flashed back to the time before the failover and quickly converted to a standby database. That process is shown here.

Read-Only Standby and Active Data Guard

Once a standby database is configured, it can be opened in read-only mode to allow query access. This is often used to offload reporting to the standby server, thereby freeing up resources on the primary server. When open in read-only mode, archive log shipping continues, but managed recovery is stopped, so the standby database becomes increasingly out of date until managed recovery is resumed.
To switch the standby database into read-only mode, do the following.
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE OPEN READ ONLY;
To resume managed recovery, do the following.
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
In 11g, Oracle introduced the Active Data Guard feature. This allows the standby database to be open in read-only mode, but still apply redo information. This means a standby can be available for querying, yet still be up to date. There are licensing implications for this feature, but the following commands show how active data guard can be enabled.
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE OPEN READ ONLY;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
Since managed recovery continues with active data guard, there is no need to switch back to managed recovery from read-only mode in this case.

Snapshot Standby

Introduced in 11g, snapshot standby allows the standby database to be opened in read-write mode. When switched back into standby mode, all changes made whilst in read-write mode are lost. This is achieved using flashback database, but the standby database does not need to have flashback database explicitly enabled to take advantage of this feature, thought it works just the same if it is.
If you are using RAC, turn off all but one of the RAC instances. Make sure the instance is in MOUNT mode.
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
Make sure managed recovery is disabled.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
Convert the standby to a snapshot standby. The following example queries the V$DATABASE view to show that flashback database is not enabled prior to the conversion operation.
SELECT flashback_on FROM v$database;

FLASHBACK_ON
------------------
NO

ALTER DATABASE CONVERT TO SNAPSHOT STANDBY;
ALTER DATABASE OPEN;
SELECT flashback_on FROM v$database;

FLASHBACK_ON
------------------
RESTORE POINT ONLY

SQL>
You can now do treat the standby like any read-write database.
To convert it back to the physical standby, losing all the changes made since the conversion to snapshot standby, issue the following commands.
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
SHUTDOWN IMMEDIATE;
STARTUP NOMOUNT;
ALTER DATABASE MOUNT STANDBY DATABASE;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
SELECT flashback_on FROM v$database;

FLASHBACK_ON
------------------
NO

SQL>

The standby is once again in managed recovery and archivelog shipping is resumed. Notice that flashback database is still not enabled.
==============================================
Physical Standby and RMAN
Using RMAN to configure a standby is also known as the "Power user" method, you have complete control over the configuration, also using this method you get a complete understanding on what does on in the background if your were to use the Enterprise Manager. There are many ways to setup a standby database manually copying files across the network, use mirroring and snapshotting, but the best method is to use RMAN, most dba's have a good understanding of RMAN. In this section i am only going to cover 11g, if yo are using 10g then I will point you to the internet on how to create a standby database using RMAN using 10g, I may come back to this topic when I have enough time to revisit the 10g side of things.
RMAN 11g has a number of improvements than the last version, which makes life simpler.
There are a number of parameters that you should be aware of, there are three types
  • Those that are independent of the database
  • Those that are specific for the primary
  • Those that are specific for the standby
As there are numerous parameters I am only going to touch on what you require to get a standby database configured
Independent Parameters
DB_UNIQUE_NAMEthis parameter defines the unique name of the database, it has to be different for each primary or standby database, Data Guard uses this parameter to identify each server within the configuration. Change this parameter does not require a bounce of the database.

db_unique_name='Prod1'
LOG_ARCHIVE_CONFIGthis defines the list of valid DB_UNIQUQ_NAME parameters for you Data Guard configuration, it provides a security check for Data Guard that the connection between the databases is allowed

log_archive_config='db_config=(Prod1,Prod1DR)'
LOG_ARCHIVE_MAX_PROCESSESthis parameter has a default of 2 which is not enough, archive processes are responsible for archiving the ORL files as they become full and for resolving gaps in the redo stream to a standby database and on a standby database they are responsible for archiving the SRL files and forwarding the archive logs to a cascaded standby database.
log_archivemax_processes='4'
DB_CREATE_FILE_DESTthis parameter needs to be defined a the standby database if using ASM

db_create_file_dest='+DATA'
Primary Parameters
LOG_ARCHIVE_DEST_nthis is the main redo transport parameter, normally you should be using the flash recovery area, this parameter has a number of attributes, the following are required
  • SERVICE - specifies the TNSNMES descriptor
  • SYNC - send redo using synchronous method
  • ASYNC - send redo using asynchronous method
  • NET_TIMEOUT - number of seconds before the LGWR process will wait for the LNS process to respond
  • REOPEN - the wait time that allows the primary database to attempt a reconnection to a failed standby database
  • DB_UNIQUE_NAME - the unique name that you specify for a database
  • VALID_FOR - when to use the log_archive_dest_n parameter and on what type of log
The example below is a standby database using maximum protection, this standby would be closer to the primary and network latency would be low, hence why we use the synchronous method

log_archive_dest_2='service=prod1dr
                            sync reopen=15 net_timeout=15
                            valid_for=(online_logfiles, primary_role)
                            db_unique_name=prod1dr0'

The example below is a standby database this database is further away and has not such good network latency hence why we use the asynchronous method and remove the timeout attribute.

log_archive_dest_3='service=prod1dr
                            async reopen=15
                            valid_for=(online_logfiles, primary_role)
                            db_unique_name=prod1dr1'
Remember you can have up to nine standby databases
The following attributes are optional
  • AFFIRM - the LNS process waits for the RFS to perform a direct I/O on the SRL file before returning a success message
  • NOAFFIRM - the LNS process will not wait and sends a success message immediately
  • COMPRESSION - uses the advanced compression option, thus any archiving sent to the standby databases will be compressed
  • MAX_CONNECTIONS - maximum number of archive processes used when sending a gap, not used in 11g
  • DELAY - this attribute delays the apply processes by the number of seconds specified, however you should be using flashback
  • ALTERNATE - used to specify a alternate location for the archive redo, not really used due to flash recovery area
  • LOCATION - used to specify the location ORL and SRL files, again not really used due to flash recovery area
  • MANDATORY - states that the ORL must be sent to this location, it will not be reused until it has been sent (dangerous attribute)
  • MAX_FAILURE - defines how many times at log switch the LGWR will attempt to reconnect to a failed standby database
  • NOREGISTER - by default any redo sent gets registered at that standby database when archived to disk
LOGARCHIVE_DEST_STATE_nthis parameter enables or disables its companion parameter log_archive_dest_n
Standby Parameter
DB_FILE_NAME_CONVERTallows you to logically move the data files from their primary database location to your standby database location. For example

db_file_name_convert='/Prod1/','/Prod1DR/'

this would translate database files from this

/u01/oradata/Prod1/sysaux.dbf

to this

/u01/oradata/Prod1DR/sysaux.dbf

you can allow do this with ASM disks

db_file_name_convert='+DATA','+RECOVERY'
LOG_FILE_NAME_CONVERTthis performs he same function as above for ORL and any SRL files
FAL_SERVERFAL is Fetch Archive Log, it is only used on a physical standby database and is used to fetch missing archive log files from one of the databases (primary or standby), it is sometimes referred to as reactive gap resolution, make use you use the TNS names.

fal_server='Prod1, Prod1DR'
FAL_CLIENTthis parameter list the name of the standby database that wishes to receive the gap request
fal_client='Prod1DR'
STANDBY_FILE_MANAGEMENTthis parameter is only used for physical standby databases, when ever data files are added or dropped from the primary the changes are reflected automatically to the standby database if this parameter has been set to AUTO, basically the DDL is executed on the standby database to create or remove any of the data files.
Now that we have a good understanding of the parameters required it is time to create a physical standby database
Create a physical standby database using RMAN
Quick tests before we start## Just do a quick tnsping on both instances from each database, fix any errors

## Run from the primary
ping primaydg01
ping physicaldg01
tnsping Prod1
tnsping Prod1DR

## Run from the standby
ping primaydg01
ping physicaldg01
tnsping Prod1
tnsping Prod1DR
Prepare the standby database## First create a static listener entry for the standby database, remember to reload the
## listener
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = Prod1DR)
      (ORACLE_HOME = /u01/app/oracle/products/11.2.0/db_home1)
      (SID_NAME = Prod1DR)
    )
  )
## Now reload the listener
lsnrctl reload
## Create a init.ora file with only the db_name in it, this file will be relaced by RMAN

echo 'DB_NAME=WHATEVER' > $ORACLE_HOME/dbs/initProd1DR.ora

## Create a password file with the primary database sys password

orapwd file=$ORACLE_HOME/dbs/orapwProd1 password=oracle

## Start up the standby database in NOMOUNT mode so that RMAN ca attach to it

export ORACLE_SID=Prod1DR
sqlplus '/ as sysdba'
SQL> startup nomount pfile=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/initProd1DR.ora
Prepare the primary database## I will be using ASM and creating 4 SRL logfiles, I will not be multiplexing he SRL log files
## hence i will be using the FLASH area only, run the following on the primary database

db_create_file_dest='+DATA';
db_create_online_log_dest_1='+FLASH';
db_create_online_log_dest_2='+DATA';

alter database add standby logfile '+FLASH' size 50m;
alter database add standby logfile '+FLASH' size 50m;
alter database add standby logfile '+FLASH' size 50m;
alter database add standby logfile '+FLASH' size 50m;
Create the standby database
(run on the primary)
## RMAN will create the standby database, the below can be run for either the primary or the
## standby database

RMAN> connect target sys/oracle@Prod1;
      connect auxiliary sys/oracle@Prod1DR;

run {
  allocate channel prmy1 type disk;
  allocate channel prmy2 type disk;
  allocate channel prmy3 type disk;
  allocate channel prmy4 type disk;
  allocate channel prmy5 type disk;

  allocate auxiliary channel stby1 type disk;

  duplicate target database for standby from active database
    spfile
      parameter_value_convert 'Prod1','Prod1DR'
      set 'db_unique_name'='Prod1DR'
      set control_files='+DATA/Prod1DR/control.ctl'
      set db_create_file_dest='+DATA'
      set db_create_online_log_dest_1='+FLASH'
      set db_create_online_log_dest_2='+DATA'
      set db_recovery_file_dest='+FLASH'
      set db_recovery_file_dest_size='4G'
    nofilenamecheck
}
Note: if you ever get an error message stating the the audit trail file could not be created look back in the output and find where the audit trail file is trying to be created, I got this error message and it was trying to create it in /u01/app/oracle/admin/PROD1LR/adump, so make sure that directory exists on the physical standby, create this directory if needed.
Finish off the standby## To finish off the standby and get start the apply process follow below, on the standby run the
## following commands

alter system set fal_server='Prod1';
alter system set fal_client='Prod1DR';
alter system set log_archive_config='dg_config=(Prod1,Prod1DR)';
alter system set standby_file_management='auto';
alter system set log_archive_dest_2='service=Prod1 ASYNC db_unique_name=Prod1
                                     valid_for=(primary_role,online_logfile);

## Now start the apply process
alter database recover managed standby database using current logfile disconnect;
Finish off the primary## Lastly we finish off the primary, run the following commands on the primary database

alter system set log_archive_config='dg_config=(Prod1,Prod1DR)';
alter system set log_archive_dest_2='service=Prod1DR ASYNC db_unique_name=Prod1DR
                                     valid_for=(primary_role,online_logfile);
alter system switch logfile;
alter system set fal_server='Prod1DR';
alter system set fal_client='Prod1';
alter system set standby_file_management='auto';

Weblogic Server Startup Fails with Error JPS-01050, "Caused by: java.io.IOException: Disc quota exceeded"

No comments :
Starting WLS with line:
/erp/apps/java/jdk1.8.0_65/bin/java -server -d64 -Xms4096m -Xmx4096m -Dweblogic.Name=AdminServer -Djava.security.policy=/erp/apps/obi12/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dopss.version=12.2.1 -
............
141107> <Version: WebLogic Server 12.2.1.0.0 Tue Oct 6 10:05:47 PDT 2015 1721936>
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.credstore.ssp.CsfWalletManager openWallet
WARNING: Opening of wallet based credential store failed. Reason java.io.IOException
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.config.OpssCommonStartup preStart
INFO: Jps startup failed.
<Nov 17, 2016 1:09:04 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>
<Nov 17, 2016 1:09:04 PM CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
oracle.security.jps.JpsException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.config.OpssCommonStartup.preStart(OpssCommonStartup.java:334)
at oracle.security.jps.JpsStartup.preStart(JpsStartup.java:286)
Caused by: oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:187)
Caused by: java.io.IOException
at oracle.security.pki.OracleWallet.open(Unknown Source)
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:175)
... 30 more
Caused by: java.lang.ExceptionInInitializerError
at oracle.security.pki.OracleFileSSOWalletImpl.a(Unknown Source)
... 32 more
Caused by: java.lang.RuntimeException: java.io.IOException: Disc quota exceeded
at oracle.security.pki.FileLocker.<clinit>(Unknown Source)
... 33 more
Caused by: java.io.IOException: Disc quota exceeded
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at java.io.File.createTempFile(File.java:2070)
... 34 more
SOLUTION
Review the OS level quota setting for the user and either increase it or move unwanted files to make more space
For example "quota" command will display the users' disk usage and limits on Linux OS

Weblogic AdminServer fails with "unable to get file lock, will retry" error message

No comments :
Jan 7, 2014 9:09:20 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:30 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:40 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:50 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:10:00 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
SOLUTION
When a server is starting, a .lok file is created. This is found under $FMW_HOME/servers/SERVERNAME/tmp/SERVERNAME.lok. If the file exists , then it cannot be created and the server cannot start.
To correct this delete the existing file or kill the running process that is holding such file.

Weblogic Fails To Start With Error Unable To Obtain Lock

No comments :
After killing the Weblogic server, the following errors were raised on restart:
<23-Jul-009 12:57:00 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. R
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
        at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:75)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:374)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:125)
And then on a re-attempt:
<23-Jul-2009 12:58:06 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:44)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:107)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Caused by: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.Heap.open(Heap.java:166)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:85)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:350)
        at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:202)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:61)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:42)
        ... 4 more
Caused by: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
        at weblogic.store.io.file.StoreDir.throwIOException(StoreDir.java:213)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:103)
        at weblogic.store.io.file.Heap.open(Heap.java:164)
        ... 9 more
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.open(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:51)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:138)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:142)
        at weblogic.store.io.file.StoreDir.open(StoreDir.java:95)
        ... 10 more

This problem sometimes occurs if you killed the bin/startWebLogic process via:
SOLUTION
There are two possible solutions:
remove the lock files and restart
kill any remaining server processes (if they exist) and restart
Solution1: Removing the lock files and restarting
1. Find -name "*.DAT" files in user_projects/domains/<your domain name> directory (as appropriate) and move or remove them, for example:
$ find . -name "*DAT"
./servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
./servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT
NOTE: Gentle reminder that Unix is case sensitive. you do need to make sure you do NOT delete the file <CM HOME>/security/SerializedSystemIni.dat, it is the files ending with "*.DAT" not "*.dat"
Deleting .DAT files: with reference to Note 1332274.1. Please be aware of the implications of removing the .DAT files. "...There may be error messages for transactions that were already committed and finished during the WebLogic Server run. However, the transaction entries will still be available in the transaction persistent store file (.DAT file)..."
2. Remove lock files: "EmbeddedLDAP.lok" and "AdminServer.lok" (maybe some more). These files are usually under paths such as below
/opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok
opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok
3. You can use the following shell script to find out which process is listening to your port, and kill this via kill -TERM:
#!/bin/ksh
line='---------------------------------------------'
pids=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')
if [ $# -eq 0 ]; then
   read ans?"Enter port you would like to know pid for: "
else
   ans=$1
fi
for f in $pids
do
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $ans"
   if [ $? -eq 0 ]; then
      echo $line
      echo "Port: $ans is being used by PID:\c"
      /usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
   fi
done
exit 0
4. Execute:
nohup ./startWebLogic.sh &
Solution2: kill any remaining server processes (if they exist) and restart
Using the example of a managed server
1. Find the PID of the managed server
ps -ef | grep ManagedServer
2. Kill the managed server process with a -KILL
kill -9 <pid of managed server>
3. Restart the Admin server and Managed servers
nohup ./startWebLogic.sh &
Note: the startWebLogic.sh script is usually run with nohup so the user can exit the shell leaving the process running.
For Windows based servers please see the following
1. The AdminServer.lok file default location is usually: C:\Oracle\Middleware\user_projects\domains\<domain_name>\servers\AdminServer\tmp\
2. Start startWebLogic.cmd
Note: If AdminServer.lok cannot be deleted:
i) end all java.exe processes in Task Manager on WebLogic server,
ii) delete AdminServer.lok file

WebLogic Admin Server Startup Fails.

No comments :
JPS-01538: The default policy provider was not found
JPS-01514: The default context is missing in jps-config.xml.
JPS-04301: Cannot read from policy store
<Jun 22, 2016 12:19:11 AM PDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.>
<Jun 22, 2016 12:19:11 AM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1394)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1018)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:273)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:705)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01514: The default context is missing in jps-config.xml.
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:665)
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:650)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:650)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-04301: Cannot read from policy store.
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.buildFromFile(XmlPolicyStore.java:298)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.<init>(XmlPolicyStore.java:177)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:106)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:76)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
SOLUTION
Delete the pki*.lck files present under /tmp folder and restart Admin Server
Note: Verify the TEMP folder used by Weblogic and look for the lock files under that folder, as it may not be /tmp always