Featured Posts
Enable or Disable Archive log mode in RAC
Enable Archivelog
1) Login to one of the nodes , verify the archive log mode
SQL> archive log list
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 200
Current log sequence 201
SQL> select name, log_mode from v$database;
NAME LOG_MODE
--------- ------------
PUB NOARCHIVELOG
2) Disable the cluster instance parameter by setting cluster_database to FALSE from the current instance:
sqlplus "/ as sysdba"
SQL> alter system set cluster_database=false scope=spfile sid='orcl1;
3) Shutdown all instances accessing the clustered database:
srvctl stop database -d orcl
4)Using the local instance, MOUNT the database:
sqlplus "/ as sysdba"
SQL> startup mount
5) Enable archiving:
SQL> alter database archivelog;
6) Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE from the current instance:
SQL> alter system set cluster_database=true scope=spfile sid='orcl1;
7) Shutdown the local instance:
SQL> shutdown immediate
8) Bring all instance back up using srvctl:
srvctl start database -d orcl
9)Login to the local instance and verify Archive Log Mode is enabled:
sqlplus "/ as sysdba"
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 200
Next log sequence to archive 201
After enabling Archive Log Mode, each instance in the RAC configuration can automatically archive redologs.
Disable Archivelog
1) export oracle_sid=orcl1
SQL> alter system set cluster_database=false scope=spfile sid='orcl1';
System altered.
2) srvctl stop database -d orcl -o immediate
3) SQL> startup mount
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 18
Next log sequence to archive 19
Current log sequence 19
SQL> alter database noarchivelog;
Database altered.
SQL> alter system set cluster_database=true scope=spfile sid='orcl1';
System altered.
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
4) srvctl start database -d orcl
5) export oracle_sid=orcl1
sqlplus / as sysdba
SQL> archive log list
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 18
Current log sequence 19
SQL>
1) Login to one of the nodes , verify the archive log mode
SQL> archive log list
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 200
Current log sequence 201
SQL> select name, log_mode from v$database;
NAME LOG_MODE
--------- ------------
PUB NOARCHIVELOG
2) Disable the cluster instance parameter by setting cluster_database to FALSE from the current instance:
sqlplus "/ as sysdba"
SQL> alter system set cluster_database=false scope=spfile sid='orcl1;
3) Shutdown all instances accessing the clustered database:
srvctl stop database -d orcl
4)Using the local instance, MOUNT the database:
sqlplus "/ as sysdba"
SQL> startup mount
5) Enable archiving:
SQL> alter database archivelog;
6) Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE from the current instance:
SQL> alter system set cluster_database=true scope=spfile sid='orcl1;
7) Shutdown the local instance:
SQL> shutdown immediate
8) Bring all instance back up using srvctl:
srvctl start database -d orcl
9)Login to the local instance and verify Archive Log Mode is enabled:
sqlplus "/ as sysdba"
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 200
Next log sequence to archive 201
After enabling Archive Log Mode, each instance in the RAC configuration can automatically archive redologs.
Disable Archivelog
1) export oracle_sid=orcl1
SQL> alter system set cluster_database=false scope=spfile sid='orcl1';
System altered.
2) srvctl stop database -d orcl -o immediate
3) SQL> startup mount
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 18
Next log sequence to archive 19
Current log sequence 19
SQL> alter database noarchivelog;
Database altered.
SQL> alter system set cluster_database=true scope=spfile sid='orcl1';
System altered.
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
4) srvctl start database -d orcl
5) export oracle_sid=orcl1
sqlplus / as sysdba
SQL> archive log list
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 18
Current log sequence 19
SQL>
Data Guard Physical Standby Setup in Oracle Database
Primary Server Setup
Logging
Check that the primary database is in archivelog mode.
SELECT log_mode FROM v$database; LOG_MODE ------------ NOARCHIVELOG SQL>
If it is noarchivelog mode, switch is to archivelog mode.
SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE ARCHIVELOG; ALTER DATABASE OPEN;
Enabled forced logging by issuing the following command.
ALTER DATABASE FORCE LOGGING;
Initialization Parameters
Check the setting for the
DB_NAME
and DB_UNIQUE_NAME
parameters. In this case they are both set to "DB11G" on the primary database.SQL> show parameter db_name NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ db_name string DB11G SQL> show parameter db_unique_name NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ db_unique_name string DB11G SQL>
The
DB_NAME
of the standby database will be the same as that of the primary, but it must have a different DB_UNIQUE_NAME
value. The DB_UNIQUE_NAME
values of the primary and standby database should be used in the DG_CONFIG
setting of the LOG_ARCHIVE_CONFIG
parameter. For this example, the standby database will have the value "DB11G_STBY".ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(DB11G,DB11G_STBY)';
Set suitable remote archive log destinations. In this case I'm using the fast recovery area for the local location, but you could specify an location explicitly if you prefer. Notice the
SERVICE
and the DB_UNIQUE_NAME
for the remote location reference the standby location.ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY'; ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;
The
LOG_ARCHIVE_FORMAT
and LOG_ARCHIVE_MAX_PROCESSES
parameters must be set to appropriate values and the REMOTE_LOGIN_PASSWORDFILE
must be set to exclusive.ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE; ALTER SYSTEM SET LOG_ARCHIVE_MAX_PROCESSES=30; ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE SCOPE=SPFILE;
In addition to the previous setting, it is recommended to make sure the primary is ready to switch roles to become a standby. For that to work properly we need to set the following parameters. Adjust the
*_CONVERT
parameters to account for your filename and path differences between the servers.ALTER SYSTEM SET FAL_SERVER=DB11G_STBY; --ALTER SYSTEM SET DB_FILE_NAME_CONVERT='DB11G_STBY','DB11G' SCOPE=SPFILE; --ALTER SYSTEM SET LOG_FILE_NAME_CONVERT='DB11G_STBY','DB11G' SCOPE=SPFILE; ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;
Remember, some of the parameters are not modifiable, so the database will need to be restarted before they take effect.
Service Setup
Entries for the primary and standby databases are needed in the "$ORACLE_HOME/network/admin/tnsnames.ora" files on both servers. You can create these using the Network Configuration Utility (netca) or manually. The following entries were used during this setup.
DB11G = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ol5-112-dga1)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = DB11G.WORLD) ) ) DB11G_STBY = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ol5-112-dga2)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = DB11G.WORLD) ) )
Backup Primary Database
If you are planning to use an active duplicate to create the standby database, then this step is unnecessary. For a backup-based duplicate, or a manual restore, take a backup of the primary database.
$ rman target=/ RMAN> BACKUP DATABASE PLUS ARCHIVELOG;
Create Standby Controlfile and PFILE
Create a controlfile for the standby database by issuing the following command on the primary database.
ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/db11g_stby.ctl';
Create a parameter file for the standby database.
CREATE PFILE='/tmp/initDB11G_stby.ora' FROM SPFILE;
Amend the PFILE making the entries relevant for the standby database. I'm making a replica of the original server, so in my case I only had to amend the following parameters.
*.db_unique_name='DB11G_STBY' *.fal_server='DB11G' *.log_archive_dest_2='SERVICE=db11g ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G'
Standby Server Setup (Manual)
Copy Files
Create the necessary directories on the standby server.
$ mkdir -p /u01/app/oracle/oradata/DB11G $ mkdir -p /u01/app/oracle/fast_recovery_area/DB11G $ mkdir -p /u01/app/oracle/admin/DB11G/adump
Copy the files from the primary to the standby server.
$ # Standby controlfile to all locations. $ scp oracle@ol5-112-dga1:/tmp/db11g_stby.ctl /u01/app/oracle/oradata/DB11G/control01.ctl $ cp /u01/app/oracle/oradata/DB11G/control01.ctl /u01/app/oracle/fast_recovery_area/DB11G/control02.ctl $ # Archivelogs and backups $ scp -r oracle@ol5-112-dga1:/u01/app/oracle/fast_recovery_area/DB11G/archivelog /u01/app/oracle/fast_recovery_area/DB11G $ scp -r oracle@ol5-112-dga1:/u01/app/oracle/fast_recovery_area/DB11G/backupset /u01/app/oracle/fast_recovery_area/DB11G $ # Parameter file. $ scp oracle@ol5-112-dga1:/tmp/initDB11G_stby.ora /tmp/initDB11G_stby.ora $ # Remote login password file. $ scp oracle@ol5-112-dga1:$ORACLE_HOME/dbs/orapwDB11G $ORACLE_HOME/dbs
Notice, the backups were copied across to the standby server as part of the FRA copy. If your backups are not held within the FRA, you must make sure you copy them to the standby server and make them available from the same path as used on the primary server.
Start Listener
Make sure the listener is started on the standby server.
$ lsnrctl start
Restore Backup
Create the SPFILE form the amended PFILE.
$ export ORACLE_SID=DB11G $ sqlplus / as sysdba SQL> CREATE SPFILE FROM PFILE='/tmp/initDB11G_stby.ora';
Restore the backup files.
$ export ORACLE_SID=DB11G $ rman target=/ RMAN> STARTUP MOUNT; RMAN> RESTORE DATABASE;
Create Redo Logs
Create online redo logs for the standby. It's a good idea to match the configuration of the primary server.
ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL; ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/DB11G/online_redo01.log') SIZE 50M; ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/DB11G/online_redo02.log') SIZE 50M; ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/DB11G/online_redo03.log') SIZE 50M; ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;
In addition to the online redo logs, you should create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs should be at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs. In my case, the following standby redo logs must be created on both servers.
ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo01.log') SIZE 50M; ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo02.log') SIZE 50M; ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo03.log') SIZE 50M; ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo04.log') SIZE 50M;
Once this is complete, we can start the apply process.
Standby Server Setup (DUPLICATE)
Copy Files
Create the necessary directories on the standby server.
$ mkdir -p /u01/app/oracle/oradata/DB11G $ mkdir -p /u01/app/oracle/fast_recovery_area/DB11G $ mkdir -p /u01/app/oracle/admin/DB11G/adump
Copy the files from the primary to the standby server.
$ # Standby controlfile to all locations. $ scp oracle@ol5-112-dga1:/tmp/db11g_stby.ctl /u01/app/oracle/oradata/DB11G/control01.ctl $ cp /u01/app/oracle/oradata/DB11G/control01.ctl /u01/app/oracle/fast_recovery_area/DB11G/control02.ctl $ # Parameter file. $ scp oracle@ol5-112-dga1:/tmp/initDB11G_stby.ora /tmp/initDB11G_stby.ora $ # Remote login password file. $ scp oracle@ol5-112-dga1:$ORACLE_HOME/dbs/orapwDB11G $ORACLE_HOME/dbs
Start Listener
When using active duplicate, the standby server requires static listener configuration in a "listener.ora" file. In this case I used the following configuration.
SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = DB11G.WORLD) (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1) (SID_NAME = DB11G) ) ) LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ol5-112-dga2.localdomain)(PORT = 1521)) ) (DESCRIPTION = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521)) ) ) ADR_BASE_LISTENER = /u01/app/oracle
Make sure the listener is started on the standby server.
$ lsnrctl start
Create Standby Redo Logs on Primary Server
The
DUPLICATE
command automatically creates the standby redo logs on the standby. To make sure the primary database is configured for switchover, we must create the standby redo logs on the primary server.ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo01.log') SIZE 50M; ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo02.log') SIZE 50M; ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo03.log') SIZE 50M; ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/DB11G/standby_redo04.log') SIZE 50M;
Create Standby Using DUPLICATE
Start the auxillary instance on the standby server by starting it using the temporary "init.ora" file.
$ export ORACLE_SID=DB11G $ sqlplus / as sysdba SQL> STARTUP NOMOUNT PFILE='/tmp/initDB11G_stby.ora';
Connect to RMAN, specifying a full connect string for both the TARGET and AUXILLARY instances. DO not attempt to use OS authentication.
$ rman TARGET sys/password@DB11G AUXILIARY sys/password@DB11G_STBY
Now issue the following DUPLICATE command.
DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE DORECOVER SPFILE SET db_unique_name='DB11G_STBY' COMMENT 'Is standby' SET LOG_ARCHIVE_DEST_2='SERVICE=db11g ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G' SET FAL_SERVER='DB11G' COMMENT 'Is primary' NOFILENAMECHECK;A brief explanation of the individual clauses is shown below.
FOR STANDBY
: This tells theDUPLICATE
command is to be used for a standby, so it will not force a DBID change.FROM ACTIVE DATABASE
: TheDUPLICATE
will be created directly from the source datafile, without an additional backup step.DORECOVER
: TheDUPLICATE
will include the recovery step, bringing the standby up to the current point in time.SPFILE
: Allows us to reset values in the spfile when it is copied from the source server.NOFILENAMECHECK
: Destination file locations are not checked.
Once the command is complete, we can start the apply process.
Start Apply Process
Start the apply process on standby server.
# Foreground redo apply. Session never returns until cancel. ALTER DATABASE RECOVER MANAGED STANDBY DATABASE; # Background redo apply. Control is returned to the session once the apply process is started. ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
If you need to cancel the apply process, issue the following command.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
If you prefer, you can set a delay between the arrival of the archived redo log and it being applied on the standby server using the following commands.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DELAY 30 DISCONNECT FROM SESSION; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT FROM SESSION;
Provided you have configured standby redo logs, you can start real-time apply using the following command.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE;
Test Log Transport
On the primary server, check the latest archived redo log and force a log switch.
ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS'; SELECT sequence#, first_time, next_time FROM v$archived_log ORDER BY sequence#; ALTER SYSTEM SWITCH LOGFILE;
Check the new archived redo log has arrived at the standby server and been applied.
ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS'; SELECT sequence#, first_time, next_time, applied FROM v$archived_log ORDER BY sequence#;
Protection Mode
There are three protection modes for the primary database:
- Maximum Availability: Transactions on the primary do not commit until redo information has been written to the online redo log and the standby redo logs of at least one standby location. If no standby location is available, it acts in the same manner as maximum performance mode until a standby becomes available again.
- Maximum Performance: Transactions on the primary commit as soon as redo information has been written to the online redo log. Transfer of redo information to the standby server is asynchronous, so it does not impact on performance of the primary.
- Maximum Protection: Transactions on the primary do not commit until redo information has been written to the online redo log and the standby redo logs of at least one standby location. If not suitable standby location is available, the primary database shuts down.
By default, for a newly created standby database, the primary database is in maximum performance mode.
SELECT protection_mode FROM v$database; PROTECTION_MODE -------------------- MAXIMUM PERFORMANCE SQL>
The mode can be switched using the following commands. Note the alterations in the redo transport attributes.
-- Maximum Availability. ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby AFFIRM SYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY'; ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY; -- Maximum Performance. ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY'; ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE; -- Maximum Protection. ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=db11g_stby AFFIRM SYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=DB11G_STBY'; SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PROTECTION; ALTER DATABASE OPEN;
Database Switchover
A database can be in one of two mutually exclusive modes (primary or standby). These roles can be altered at runtime without loss of data or resetting of redo logs. This process is known as a Switchover and can be performed using the following statements.
-- Convert primary database to standby CONNECT / AS SYSDBA ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY; -- Shutdown primary database SHUTDOWN IMMEDIATE; -- Mount old primary database as standby database STARTUP NOMOUNT; ALTER DATABASE MOUNT STANDBY DATABASE; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
On the original standby database issue the following commands.
-- Convert standby database to primary CONNECT / AS SYSDBA ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY; -- Shutdown standby database SHUTDOWN IMMEDIATE; -- Open old standby database as primary STARTUP;
Once this is complete, test the log transport as before. If everything is working fine, switch the primary database back to the original server by doing another switchover. This is known as a switchback.
Failover
If the primary database is not available the standby database can be activated as a primary database using the following statements.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH; ALTER DATABASE ACTIVATE STANDBY DATABASE;
Since the standby database is now the primary database it should be backed up immediately.
The original primary database can now be configured as a standby. If Flashback Database was enabled on the primary database, then this can be done relatively easily (shown here). If not, the whole setup process must be followed, but this time using the original primary server as the standby.
Flashback Database
It was already mentioned in the previous section, but it is worth drawing your attention to Flashback Database once more. Although a switchover/switchback is safe for both the primary and standby database, a failover renders the original primary database useless for converting to a standby database. If flashback database is not enabled, the original primary must be scrapped and recreated as a standby database.
An alternative is to enable flashback database on the primary (and the standby if desired) so in the event of a failover, the primary can be flashed back to the time before the failover and quickly converted to a standby database. That process is shown here.
Read-Only Standby and Active Data Guard
Once a standby database is configured, it can be opened in read-only mode to allow query access. This is often used to offload reporting to the standby server, thereby freeing up resources on the primary server. When open in read-only mode, archive log shipping continues, but managed recovery is stopped, so the standby database becomes increasingly out of date until managed recovery is resumed.
To switch the standby database into read-only mode, do the following.
SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE OPEN READ ONLY;
To resume managed recovery, do the following.
SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
In 11g, Oracle introduced the Active Data Guard feature. This allows the standby database to be open in read-only mode, but still apply redo information. This means a standby can be available for querying, yet still be up to date. There are licensing implications for this feature, but the following commands show how active data guard can be enabled.
SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE OPEN READ ONLY; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
Since managed recovery continues with active data guard, there is no need to switch back to managed recovery from read-only mode in this case.
Snapshot Standby
Introduced in 11g, snapshot standby allows the standby database to be opened in read-write mode. When switched back into standby mode, all changes made whilst in read-write mode are lost. This is achieved using flashback database, but the standby database does not need to have flashback database explicitly enabled to take advantage of this feature, thought it works just the same if it is.
If you are using RAC, turn off all but one of the RAC instances. Make sure the instance is in MOUNT mode.
SHUTDOWN IMMEDIATE; STARTUP MOUNT;
Make sure managed recovery is disabled.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
Convert the standby to a snapshot standby. The following example queries the
V$DATABASE
view to show that flashback database is not enabled prior to the conversion operation.SELECT flashback_on FROM v$database; FLASHBACK_ON ------------------ NO ALTER DATABASE CONVERT TO SNAPSHOT STANDBY; ALTER DATABASE OPEN; SELECT flashback_on FROM v$database; FLASHBACK_ON ------------------ RESTORE POINT ONLY SQL>
You can now do treat the standby like any read-write database.
To convert it back to the physical standby, losing all the changes made since the conversion to snapshot standby, issue the following commands.
SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE CONVERT TO PHYSICAL STANDBY; SHUTDOWN IMMEDIATE; STARTUP NOMOUNT; ALTER DATABASE MOUNT STANDBY DATABASE; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT; SELECT flashback_on FROM v$database; FLASHBACK_ON ------------------ NO SQL>
The standby is once again in managed recovery and archivelog shipping is resumed. Notice that flashback database is still not enabled.
==============================================
==============================================
Using RMAN to configure a standby is also known as the "Power user" method, you have complete control over the configuration, also using this method you get a complete understanding on what does on in the background if your were to use the Enterprise Manager. There are many ways to setup a standby database manually copying files across the network, use mirroring and snapshotting, but the best method is to use RMAN, most dba's have a good understanding of RMAN. In this section i am only going to cover 11g, if yo are using 10g then I will point you to the internet on how to create a standby database using RMAN using 10g, I may come back to this topic when I have enough time to revisit the 10g side of things.
RMAN 11g has a number of improvements than the last version, which makes life simpler.
There are a number of parameters that you should be aware of, there are three types
- Those that are independent of the database
- Those that are specific for the primary
- Those that are specific for the standby
As there are numerous parameters I am only going to touch on what you require to get a standby database configured
Independent Parameters
| |
DB_UNIQUE_NAME | this parameter defines the unique name of the database, it has to be different for each primary or standby database, Data Guard uses this parameter to identify each server within the configuration. Change this parameter does not require a bounce of the database. db_unique_name='Prod1' |
LOG_ARCHIVE_CONFIG | this defines the list of valid DB_UNIQUQ_NAME parameters for you Data Guard configuration, it provides a security check for Data Guard that the connection between the databases is allowed log_archive_config='db_config=(Prod1,Prod1DR)' |
LOG_ARCHIVE_MAX_PROCESSES | this parameter has a default of 2 which is not enough, archive processes are responsible for archiving the ORL files as they become full and for resolving gaps in the redo stream to a standby database and on a standby database they are responsible for archiving the SRL files and forwarding the archive logs to a cascaded standby database. log_archivemax_processes='4' |
DB_CREATE_FILE_DEST | this parameter needs to be defined a the standby database if using ASM db_create_file_dest='+DATA' |
Primary Parameters
| |
LOG_ARCHIVE_DEST_n | this is the main redo transport parameter, normally you should be using the flash recovery area, this parameter has a number of attributes, the following are required
log_archive_dest_2='service=prod1dr sync reopen=15 net_timeout=15 valid_for=(online_logfiles, primary_role) db_unique_name=prod1dr0' The example below is a standby database this database is further away and has not such good network latency hence why we use the asynchronous method and remove the timeout attribute. log_archive_dest_3='service=prod1dr async reopen=15 valid_for=(online_logfiles, primary_role) db_unique_name=prod1dr1' Remember you can have up to nine standby databases The following attributes are optional
|
LOGARCHIVE_DEST_STATE_n | this parameter enables or disables its companion parameter log_archive_dest_n |
Standby Parameter
| |
DB_FILE_NAME_CONVERT | allows you to logically move the data files from their primary database location to your standby database location. For example db_file_name_convert='/Prod1/','/Prod1DR/' this would translate database files from this /u01/oradata/Prod1/sysaux.dbf to this /u01/oradata/Prod1DR/sysaux.dbf you can allow do this with ASM disks db_file_name_convert='+DATA','+RECOVERY' |
LOG_FILE_NAME_CONVERT | this performs he same function as above for ORL and any SRL files |
FAL_SERVER | FAL is Fetch Archive Log, it is only used on a physical standby database and is used to fetch missing archive log files from one of the databases (primary or standby), it is sometimes referred to as reactive gap resolution, make use you use the TNS names. fal_server='Prod1, Prod1DR' |
FAL_CLIENT | this parameter list the name of the standby database that wishes to receive the gap request fal_client='Prod1DR' |
STANDBY_FILE_MANAGEMENT | this parameter is only used for physical standby databases, when ever data files are added or dropped from the primary the changes are reflected automatically to the standby database if this parameter has been set to AUTO, basically the DDL is executed on the standby database to create or remove any of the data files. |
Now that we have a good understanding of the parameters required it is time to create a physical standby database
Create a physical standby database using RMAN
| |
Quick tests before we start | ## Just do a quick tnsping on both instances from each database, fix any errors ## Run from the primary ping primaydg01 ping physicaldg01 tnsping Prod1 tnsping Prod1DR ## Run from the standby ping primaydg01 ping physicaldg01 tnsping Prod1 tnsping Prod1DR |
Prepare the standby database | ## First create a static listener entry for the standby database, remember to reload the ## listener SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = Prod1DR) (ORACLE_HOME = /u01/app/oracle/products/11.2.0/db_home1) (SID_NAME = Prod1DR) ) ) ## Now reload the listener lsnrctl reload ## Create a init.ora file with only the db_name in it, this file will be relaced by RMAN echo 'DB_NAME=WHATEVER' > $ORACLE_HOME/dbs/initProd1DR.ora ## Create a password file with the primary database sys password orapwd file=$ORACLE_HOME/dbs/orapwProd1 password=oracle ## Start up the standby database in NOMOUNT mode so that RMAN ca attach to it export ORACLE_SID=Prod1DR sqlplus '/ as sysdba' SQL> startup nomount pfile=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/initProd1DR.ora |
Prepare the primary database | ## I will be using ASM and creating 4 SRL logfiles, I will not be multiplexing he SRL log files ## hence i will be using the FLASH area only, run the following on the primary database db_create_file_dest='+DATA'; db_create_online_log_dest_1='+FLASH'; db_create_online_log_dest_2='+DATA'; alter database add standby logfile '+FLASH' size 50m; alter database add standby logfile '+FLASH' size 50m; alter database add standby logfile '+FLASH' size 50m; alter database add standby logfile '+FLASH' size 50m; |
Create the standby database (run on the primary) | ## RMAN will create the standby database, the below can be run for either the primary or the ## standby database RMAN> connect target sys/oracle@Prod1; connect auxiliary sys/oracle@Prod1DR; run { allocate channel prmy1 type disk; allocate channel prmy2 type disk; allocate channel prmy3 type disk; allocate channel prmy4 type disk; allocate channel prmy5 type disk; allocate auxiliary channel stby1 type disk; duplicate target database for standby from active database spfile parameter_value_convert 'Prod1','Prod1DR' set 'db_unique_name'='Prod1DR' set control_files='+DATA/Prod1DR/control.ctl' set db_create_file_dest='+DATA' set db_create_online_log_dest_1='+FLASH' set db_create_online_log_dest_2='+DATA' set db_recovery_file_dest='+FLASH' set db_recovery_file_dest_size='4G' nofilenamecheck } Note: if you ever get an error message stating the the audit trail file could not be created look back in the output and find where the audit trail file is trying to be created, I got this error message and it was trying to create it in /u01/app/oracle/admin/PROD1LR/adump, so make sure that directory exists on the physical standby, create this directory if needed. |
Finish off the standby | ## To finish off the standby and get start the apply process follow below, on the standby run the ## following commands alter system set fal_server='Prod1'; alter system set fal_client='Prod1DR'; alter system set log_archive_config='dg_config=(Prod1,Prod1DR)'; alter system set standby_file_management='auto'; alter system set log_archive_dest_2='service=Prod1 ASYNC db_unique_name=Prod1 valid_for=(primary_role,online_logfile); ## Now start the apply process alter database recover managed standby database using current logfile disconnect; |
Finish off the primary | ## Lastly we finish off the primary, run the following commands on the primary database alter system set log_archive_config='dg_config=(Prod1,Prod1DR)'; alter system set log_archive_dest_2='service=Prod1DR ASYNC db_unique_name=Prod1DR valid_for=(primary_role,online_logfile); alter system switch logfile; alter system set fal_server='Prod1DR'; alter system set fal_client='Prod1'; alter system set standby_file_management='auto'; |
Weblogic Server Startup Fails with Error JPS-01050, "Caused by: java.io.IOException: Disc quota exceeded"
Starting WLS with line:
/erp/apps/java/jdk1.8.0_65/bin/java -server -d64 -Xms4096m -Xmx4096m -Dweblogic.Name=AdminServer -Djava.security.policy=/erp/apps/obi12/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dopss.version=12.2.1 -
............
141107> <Version: WebLogic Server 12.2.1.0.0 Tue Oct 6 10:05:47 PDT 2015 1721936>
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.credstore.ssp.CsfWalletManager openWallet
WARNING: Opening of wallet based credential store failed. Reason java.io.IOException
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.config.OpssCommonStartup preStart
INFO: Jps startup failed.
<Nov 17, 2016 1:09:04 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>
<Nov 17, 2016 1:09:04 PM CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
oracle.security.jps.JpsException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.config.OpssCommonStartup.preStart(OpssCommonStartup.java:334)
at oracle.security.jps.JpsStartup.preStart(JpsStartup.java:286)
Caused by: oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:187)
Caused by: java.io.IOException
at oracle.security.pki.OracleWallet.open(Unknown Source)
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:175)
... 30 more
Caused by: java.lang.ExceptionInInitializerError
at oracle.security.pki.OracleFileSSOWalletImpl.a(Unknown Source)
... 32 more
Caused by: java.lang.RuntimeException: java.io.IOException: Disc quota exceeded
at oracle.security.pki.FileLocker.<clinit>(Unknown Source)
... 33 more
Caused by: java.io.IOException: Disc quota exceeded
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at java.io.File.createTempFile(File.java:2070)
... 34 more
SOLUTION
Review the OS level quota setting for the user and either increase it or move unwanted files to make more space
For example "quota" command will display the users' disk usage and limits on Linux OS
/erp/apps/java/jdk1.8.0_65/bin/java -server -d64 -Xms4096m -Xmx4096m -Dweblogic.Name=AdminServer -Djava.security.policy=/erp/apps/obi12/Oracle/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dopss.version=12.2.1 -
............
141107> <Version: WebLogic Server 12.2.1.0.0 Tue Oct 6 10:05:47 PDT 2015 1721936>
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.credstore.ssp.CsfWalletManager openWallet
WARNING: Opening of wallet based credential store failed. Reason java.io.IOException
Nov 17, 2016 1:09:00 PM oracle.security.jps.internal.config.OpssCommonStartup preStart
INFO: Jps startup failed.
<Nov 17, 2016 1:09:04 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
<Nov 17, 2016 1:09:04 PM CST> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>
<Nov 17, 2016 1:09:04 PM CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
oracle.security.jps.JpsException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.config.OpssCommonStartup.preStart(OpssCommonStartup.java:334)
at oracle.security.jps.JpsStartup.preStart(JpsStartup.java:286)
Caused by: oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:187)
Caused by: java.io.IOException
at oracle.security.pki.OracleWallet.open(Unknown Source)
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:175)
... 30 more
Caused by: java.lang.ExceptionInInitializerError
at oracle.security.pki.OracleFileSSOWalletImpl.a(Unknown Source)
... 32 more
Caused by: java.lang.RuntimeException: java.io.IOException: Disc quota exceeded
at oracle.security.pki.FileLocker.<clinit>(Unknown Source)
... 33 more
Caused by: java.io.IOException: Disc quota exceeded
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at java.io.File.createTempFile(File.java:2070)
... 34 more
SOLUTION
Review the OS level quota setting for the user and either increase it or move unwanted files to make more space
For example "quota" command will display the users' disk usage and limits on Linux OS
Weblogic AdminServer fails with "unable to get file lock, will retry" error message
Jan 7, 2014 9:09:20 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:30 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:40 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:50 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:10:00 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
SOLUTION
When a server is starting, a .lok file is created. This is found under $FMW_HOME/servers/SERVERNAME/tmp/SERVERNAME.lok. If the file exists , then it cannot be created and the server cannot start.
To correct this delete the existing file or kill the running process that is holding such file.
<Jan 7, 2014 9:09:30 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:40 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:09:50 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
<Jan 7, 2014 9:10:00 AM PST> <Info> <Management> <BEA-141281> <unable to get file lock, will retry ...>
SOLUTION
When a server is starting, a .lok file is created. This is found under $FMW_HOME/servers/SERVERNAME/tmp/SERVERNAME.lok. If the file exists , then it cannot be created and the server cannot start.
To correct this delete the existing file or kill the running process that is holding such file.
Weblogic Fails To Start With Error Unable To Obtain Lock
After killing the Weblogic server, the following errors were raised on restart:
<23-Jul-009 12:57:00 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. R
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:75)
at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:374)
at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:125)
And then on a re-attempt:
<23-Jul-2009 12:58:06 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:44)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:107)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Caused by: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
at weblogic.store.io.file.Heap.open(Heap.java:166)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:85)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:350)
at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:202)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:61)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:42)
... 4 more
Caused by: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
at weblogic.store.io.file.StoreDir.throwIOException(StoreDir.java:213)
at weblogic.store.io.file.StoreDir.open(StoreDir.java:103)
at weblogic.store.io.file.Heap.open(Heap.java:164)
... 9 more
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.open(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:51)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:138)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:142)
at weblogic.store.io.file.StoreDir.open(StoreDir.java:95)
... 10 more
This problem sometimes occurs if you killed the bin/startWebLogic process via:
SOLUTION
There are two possible solutions:
remove the lock files and restart
kill any remaining server processes (if they exist) and restart
Solution1: Removing the lock files and restarting
1. Find -name "*.DAT" files in user_projects/domains/<your domain name> directory (as appropriate) and move or remove them, for example:
$ find . -name "*DAT"
./servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
./servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT
NOTE: Gentle reminder that Unix is case sensitive. you do need to make sure you do NOT delete the file <CM HOME>/security/SerializedSystemIni.dat, it is the files ending with "*.DAT" not "*.dat"
Deleting .DAT files: with reference to Note 1332274.1. Please be aware of the implications of removing the .DAT files. "...There may be error messages for transactions that were already committed and finished during the WebLogic Server run. However, the transaction entries will still be available in the transaction persistent store file (.DAT file)..."
2. Remove lock files: "EmbeddedLDAP.lok" and "AdminServer.lok" (maybe some more). These files are usually under paths such as below
/opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok
opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok
3. You can use the following shell script to find out which process is listening to your port, and kill this via kill -TERM:
#!/bin/ksh
line='---------------------------------------------'
pids=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')
if [ $# -eq 0 ]; then
read ans?"Enter port you would like to know pid for: "
else
ans=$1
fi
for f in $pids
do
/usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $ans"
if [ $? -eq 0 ]; then
echo $line
echo "Port: $ans is being used by PID:\c"
/usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
fi
done
exit 0
4. Execute:
nohup ./startWebLogic.sh &
Solution2: kill any remaining server processes (if they exist) and restart
Using the example of a managed server
1. Find the PID of the managed server
ps -ef | grep ManagedServer
2. Kill the managed server process with a -KILL
kill -9 <pid of managed server>
3. Restart the Admin server and Managed servers
nohup ./startWebLogic.sh &
Note: the startWebLogic.sh script is usually run with nohup so the user can exit the shell leaving the process running.
For Windows based servers please see the following
1. The AdminServer.lok file default location is usually: C:\Oracle\Middleware\user_projects\domains\<domain_name>\servers\AdminServer\tmp\
2. Start startWebLogic.cmd
Note: If AdminServer.lok cannot be deleted:
i) end all java.exe processes in Task Manager on WebLogic server,
ii) delete AdminServer.lok file
<23-Jul-009 12:57:00 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. R
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:75)
at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:374)
at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:125)
And then on a re-attempt:
<23-Jul-2009 12:58:06 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:44)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:107)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Caused by: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
at weblogic.store.io.file.Heap.open(Heap.java:166)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:85)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:350)
at weblogic.store.PersistentStoreManager.createFileStore(PersistentStoreManager.java:202)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:61)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:42)
... 4 more
Caused by: java.io.IOException: [Store:280021]There was an error while opening the file store file "WLS_DIAGNOSTICS000000.DAT"
at weblogic.store.io.file.StoreDir.throwIOException(StoreDir.java:213)
at weblogic.store.io.file.StoreDir.open(StoreDir.java:103)
at weblogic.store.io.file.Heap.open(Heap.java:164)
... 9 more
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.open(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:51)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:138)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:142)
at weblogic.store.io.file.StoreDir.open(StoreDir.java:95)
... 10 more
This problem sometimes occurs if you killed the bin/startWebLogic process via:
SOLUTION
There are two possible solutions:
remove the lock files and restart
kill any remaining server processes (if they exist) and restart
Solution1: Removing the lock files and restarting
1. Find -name "*.DAT" files in user_projects/domains/<your domain name> directory (as appropriate) and move or remove them, for example:
$ find . -name "*DAT"
./servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
./servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT
NOTE: Gentle reminder that Unix is case sensitive. you do need to make sure you do NOT delete the file <CM HOME>/security/SerializedSystemIni.dat, it is the files ending with "*.DAT" not "*.dat"
Deleting .DAT files: with reference to Note 1332274.1. Please be aware of the implications of removing the .DAT files. "...There may be error messages for transactions that were already committed and finished during the WebLogic Server run. However, the transaction entries will still be available in the transaction persistent store file (.DAT file)..."
2. Remove lock files: "EmbeddedLDAP.lok" and "AdminServer.lok" (maybe some more). These files are usually under paths such as below
/opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/tmp/AdminServer.lok
opt/bea/user_projects/domains/ORACLE_COMMUNICATIONS/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok
3. You can use the following shell script to find out which process is listening to your port, and kill this via kill -TERM:
#!/bin/ksh
line='---------------------------------------------'
pids=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')
if [ $# -eq 0 ]; then
read ans?"Enter port you would like to know pid for: "
else
ans=$1
fi
for f in $pids
do
/usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $ans"
if [ $? -eq 0 ]; then
echo $line
echo "Port: $ans is being used by PID:\c"
/usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
fi
done
exit 0
4. Execute:
nohup ./startWebLogic.sh &
Solution2: kill any remaining server processes (if they exist) and restart
Using the example of a managed server
1. Find the PID of the managed server
ps -ef | grep ManagedServer
2. Kill the managed server process with a -KILL
kill -9 <pid of managed server>
3. Restart the Admin server and Managed servers
nohup ./startWebLogic.sh &
Note: the startWebLogic.sh script is usually run with nohup so the user can exit the shell leaving the process running.
For Windows based servers please see the following
1. The AdminServer.lok file default location is usually: C:\Oracle\Middleware\user_projects\domains\<domain_name>\servers\AdminServer\tmp\
2. Start startWebLogic.cmd
Note: If AdminServer.lok cannot be deleted:
i) end all java.exe processes in Task Manager on WebLogic server,
ii) delete AdminServer.lok file
WebLogic Admin Server Startup Fails.
JPS-01538: The default policy provider was not found
JPS-01514: The default context is missing in jps-config.xml.
JPS-04301: Cannot read from policy store
<Jun 22, 2016 12:19:11 AM PDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.>
<Jun 22, 2016 12:19:11 AM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1394)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1018)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:273)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:705)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01514: The default context is missing in jps-config.xml.
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:665)
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:650)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:650)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-04301: Cannot read from policy store.
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.buildFromFile(XmlPolicyStore.java:298)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.<init>(XmlPolicyStore.java:177)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:106)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:76)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
SOLUTION
Delete the pki*.lck files present under /tmp folder and restart Admin Server
Note: Verify the TEMP folder used by Weblogic and look for the lock files under that folder, as it may not be /tmp always
JPS-01514: The default context is missing in jps-config.xml.
JPS-04301: Cannot read from policy store
<Jun 22, 2016 12:19:11 AM PDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.>
<Jun 22, 2016 12:19:11 AM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1394)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1018)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:875)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:273)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01538: The default policy provider was not found.
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:705)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:265)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01514: The default context is missing in jps-config.xml.
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:665)
at oracle.security.jps.internal.policystore.PolicyUtil$1.run(PolicyUtil.java:650)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPolicyStore(PolicyUtil.java:650)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:271)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-04301: Cannot read from policy store.
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.buildFromFile(XmlPolicyStore.java:298)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.<init>(XmlPolicyStore.java:177)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:106)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStoreProvider.getInstance(XmlPolicyStoreProvider.java:76)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
SOLUTION
Delete the pki*.lck files present under /tmp folder and restart Admin Server
Note: Verify the TEMP folder used by Weblogic and look for the lock files under that folder, as it may not be /tmp always
Subscribe to:
Posts
(
Atom
)