Oracle Application DBA

adop prepare phase failed
ssh is not enabled for the following nodes
    - - - - - - - - - - - - - - - - - - - - - -
    pdhx01appd06
    [UNEXPECTED]ssh is not enabled.
    [UNEXPECTED]Hint: These are some available options:
    [UNEXPECTED]- If you would like to enable ssh, execute the following command:
     perl /u11/home/pdhuat/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh \
          -contextfile=/u11/home/pdhuat/fs2/inst/apps/pdhuat_pdhx01appd05/appl/admin/pdhuat_pdhx01appd05.xml \
          -hosts=pdhx01appd05,pdhx01appd06
Solution:
1. Password is expired.
[applpdhu@pdhx01appd05 ~]$ ssh pdhx01appd06
[applpdhu@pdhx01appd06 ~]$ ssh pdhx01appd05
Your account has expired; please contact your system administrator
Connection closed by 10.2.217.143
2. set the password of applmgr user on both nodes.
[applpdhu@pdhx01appd05 clone]$ ssh pdhx01appd06
Last login: Fri Jan  8 13:56:17 2016 from pdhx01appd05.hostname.com
3. run again adop phase=prepare on primary node

/etc/init.d/sendmail stop
/etc/init.d/sendmail start

Two months before I was performed Linux OS upgradation on live production and test environment for one of my big client. Both environments are 2 node RAC configured.
Below mentioned steps for OS upgradation,
OS Upgrade Pre-requisites Steps:-
Kindly cross check with Oracle and RedHat for license certification matrix
Take fullbackup of OS and Database.
OS Upgrade Steps:-
(1) Shutdown DB, DBconsole (if any), Cluster, ASMlib (if any in Linux)
As RDBMS owner id “oracle”:
$ srvctl stop database -d -o immediate #– run once in any node
$ emctl stop dbconsole #– run on all nodes
As UNIX/Linux “root”:
# /bin/crsctl stop crs #– run on all nodes
# /bin/crsctl disable crs #– run on all nodes
# /etc/init.d/oracleasm stop #– run on all nodes
# /etc/init.d/oracleasm disable #– run on all nodes
(2) Backup filesystem for GRID ORACLE_HOME & RDBMS ORACLE_HOME
(3) Upgrade RedHat OS
configure RHN (Give Rhn_register command thru root user).
You have to give Redhat login user/password and your proxy address & port.
RHN will collect your system information then will send that to redhat network.
Once RHN configured then give command like #yum update --no plugin (or yum update)
It will download latest packages from redhat repository and install automatically, final stage the OS will ask you for reboot the machine.
After reboot you can get the latest kernal (using this command check the kernal and redhat os version #uname-a, cat /etc/redhat-release)
(4) Upgrade ASMlib (Download ASM packages based on latest kernal version)
# rpm -Uvh oracleasm-support-xxx
# rpm -Uvh oracleasm-2.6.xxx
# rpm -Uvh oracleasmlib-xxx
# /etc/init.d/oracleasm enable
# /etc/init.d/oracleasm lisdisks
(5) Relink RDBMS ORACLE_HOME Binary
Note: complete full steps in one node, then proceed to next
As RDBMS owner id “oracle”:
$ /bin/relink all
(6) Relink GRID ORACLE_HOME Binary
Note: complete full steps in one node, then proceed to next
As UNIX/Linux “root”:
# cd /crs/install
# ./rootcrs.pl -unlock
As RDBMS owner id “oracle”:
$ cd /bin
$ ./relink all
As UNIX/Linux “root”:
# cd /rdbms/install
# ./rootadd_rdbms.sh
# cd /crs/install
# ./rootcrs.pl –patch
(7) Enable CRS auto-startup and start CRS on all nodes
Note: complete full steps in one node, then proceed to next
As UNIX/Linux “root”:
# /bin/crsctl start crs
# /bin/crsctl enable crs
(8) Double check if DB, instances and all resrouces are ONLINE.
As RDBMS owner id “oracle”:
$ crsctl stat res -t
Reference :: How to Check Whether Oracle Binary/Instance is RAC Enabled and Relink Oracle Binary in RAC [ID 284785.1]

MIN HOUR DOM MON DOW CMD
Table: Crontab Fields and Allowed Ranges (Linux Crontab Syntax)
Field    Description    Allowed Value
MIN    Minute field    0 to 59
HOUR    Hour field    0 to 23
DOM    Day of Month    1-31
MON    Month field    1-12
DOW    Day Of Week    0-6
CMD    Command    Any command to be executed.
1. Scheduling a Job For a Specific Time
This will execute the Full backup shell script (full-backup) on 10th June 08:30 AM.
Please note that the time field uses 24 hours format. So, for 8 AM use 8, and for 8 PM use 20.
30 08 10 06 * /home/apdba/full-backup
30 – 30th Minute
08 – 08 AM
10 – 10th Day
06 – 6th Month (June)
* – Every day of the week
2. Schedule a Job For More Than One Instance (e.g. Twice a Day)
The following script take a incremental backup twice a day every day.
This example executes the specified incremental backup shell script (incremental-backup) at 11:00 and 16:00 on every day.
The comma separated value in a field specifies that the command needs to be executed in all the mentioned time.
00 11,16 * * * /home/apdba/bin/incremental-backup
00 – 0th Minute (Top of the hour)
11,16 – 11 AM and 4 PM
* – Every day
* – Every month
* – Every day of the week
3. Schedule a Job for Specific Range of Time (e.g. Only on Weekdays)
If you wanted a job to be scheduled for every hour with in a specific range of time then use the following.
Cron Job everyday during working hours
This example checks the status of the database everyday (including weekends) during the working hours 9 a.m – 6 p.m
00 09-18 * * * /home/apdba/bin/check-db-status
00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
* – Every day of the week
Cron Job every weekday during working hours
This example checks the status of the database every weekday (i.e excluding Sat and Sun) during the working hours 9 a.m – 6 p.m.
00 09-18 * * 1-5 /home/apdba/bin/check-db-status
00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
1-5 -Mon, Tue, Wed, Thu and Fri (Every Weekday)
4. How to View Crontab Entries?
View Current Logged-In User’s Crontab entries
To view your crontab entries type crontab -l from your unix account as shown below.
apdba@dev-db$ crontab -l
@yearly /home/apdba/annual-maintenance
*/10 * * * * /home/apdba/check-disk-space
[Note: This displays crontab of the current logged in user]
View Root Crontab entries
Login as root user (su – root) and do crontab -l as shown below.
root@dev-db# crontab -l
no crontab for root
Crontab HowTo: View Other Linux User’s Crontabs entries
To view crontab entries of other Linux users, login to root and use -u {username} -l as shown below.
root@dev-db# crontab -u sathiya -l
@monthly /home/sathiya/monthly-backup
00 09-18 * * * /home/sathiya/check-db-status
5. How to Edit Crontab Entries?
Edit Current Logged-In User’s Crontab entries
To edit a crontab entries, use crontab -e as shown below. By default this will edit the current logged-in users crontab.
apdba@dev-db$ crontab -e
@yearly /home/apdba/centos/bin/annual-maintenance
*/10 * * * * /home/apdba/debian/bin/check-disk-space
~
"/tmp/crontab.XXXXyjWkHw" 2L, 83C
[Note: This will open the crontab file in Vim editor for editing.
Please note cron created a temporary /tmp/crontab.XX... ]
When you save the above temporary file with :wq, it will save the crontab and display the following message indicating the crontab is successfully modified.
"crontab.XXXXyjWkHw" 2L, 83C written
crontab: installing new crontab
Edit Root Crontab entries
Login as root user (su – root) and do crontab -e as shown below.
root@dev-db# crontab -e
Edit Other Linux User’s Crontab File entries
To edit crontab entries of other Linux users, login to root and use -u {username} -e as shown below.
root@dev-db# crontab -u sathiya -e
@monthly /home/sathiya/fedora/bin/monthly-backup
00 09-18 * * * /home/sathiya/ubuntu/bin/check-db-status

"/tmp/crontab.XXXXyjWkHw" 2L, 83C
6. Schedule a Job for Every Minute Using Cron.
Ideally you may not have a requirement to schedule a job every minute. But understanding this example will will help
you understand the other examples mentioned below in this article.
* * * * * CMD
The * means all the possible unit — i.e every minute of every hour through out the year. More than using this * directly,
 you will find it very useful in the following cases.
When you specify */5 in minute field means every 5 minutes.
When you specify 0-10/2 in minute field mean every 2 minutes in the first 10 minute.
Thus the above convention can be used for all the other 4 fields.
7. Schedule a Background Cron Job For Every 10 Minutes.
Use the following, if you want to check the disk space every 10 minutes.
*/10 * * * * /home/apdba/check-disk-space
It executes the specified command check-disk-space every 10 minutes through out the year. But you may have a requirement
of executing the command only during office hours or vice versa. The above examples shows how to do those things.
Instead of specifying values in the 5 fields, we can specify it using a single keyword as mentioned below.
There are special cases in which instead of the above 5 fields you can use @ followed by a keyword — such as reboot, midnight, yearly, hourly.
Table: Cron special keywords and its meaning
Keyword    Equivalent
@yearly    0 0 1 1 *
@daily    0 0 * * *
@hourly    0 * * * *
@reboot    Run at startup.
8. Schedule a Job For First Minute of Every Year using @yearly
If you want a job to be executed on the first minute of every year, then you can use the @yearly cron keyword as shown below.
This will execute the system annual maintenance using annual-maintenance shell script at 00:00 on Jan 1st for every year.
@yearly /home/apdba/red-hat/bin/annual-maintenance
9. Schedule a Cron Job Beginning of Every Month using @monthly
It is as similar as the @yearly as above. But executes the command monthly once using @monthly cron keyword.
This will execute the shell script tape-backup at 00:00 on 1st of every month.
@monthly /home/apdba/suse/bin/tape-backup
10. Schedule a Background Job Every Day using @daily
Using the @daily cron keyword, this will do a daily log file cleanup using cleanup-logs shell scriptat 00:00 on every day.
@daily /home/apdba/arch-linux/bin/cleanup-logs "day started"
11. How to Execute a Linux Command After Every Reboot using @reboot?
Using the @reboot cron keyword, this will execute the specified command once after the machine got booted every time.
@reboot CMD
12. How to Disable/Redirect the Crontab Mail Output using MAIL keyword?
By default crontab sends the job output to the user who scheduled the job. If you want to redirect the output to a specific user,
add or update the MAIL variable in the crontab as shown below.
apdba@dev-db$ crontab -l
MAIL="apdba"
@yearly /home/apdba/annual-maintenance
*/10 * * * * /home/apdba/check-disk-space
[Note: Crontab of the current logged in user with MAIL variable]
If you wanted the mail not to be sent to anywhere, i.e to stop the crontab output to be emailed, add or update the MAIL variable in the crontab as shown below.
MAIL=""
13. How to Execute a Linux Cron Jobs Every Second Using Crontab.
You cannot schedule a every-second cronjob. Because in cron the minimum unit you can specify is minute. In a typical scenario,
there is no reason for most of us to run any job every second in the system.
14. Specify PATH Variable in the Crontab
All the above examples we specified absolute path of the Linux command or the shell-script that needs to be executed.
For example, instead of specifying /home/apdba/tape-backup, if you want to just specify tape-backup, then add the path /home/apdba to the PATH variable in the crontab as shown below.
apdba@dev-db$ crontab -l
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/home/apdba
@yearly annual-maintenance
*/10 * * * * check-disk-space
[Note: Crontab of the current logged in user with PATH variable]
15. Installing Crontab From a Cron File
Instead of directly editing the crontab file, you can also add all the entries to a cron-file first. Once you have all thoese entries in the file,
you can upload or install them to the cron as shown below.
apdba@dev-db$ crontab -l
no crontab for apdba
$ cat cron-file.txt
@yearly /home/apdba/annual-maintenance
*/10 * * * * /home/apdba/check-disk-space
apdba@dev-db$ crontab cron-file.txt
apdba@dev-db$ crontab -l
@yearly /home/apdba/annual-maintenance
*/10 * * * * /home/apdba/check-disk-space
Note: This will install the cron-file.txt to your crontab, which will also remove your old cron entries. So, please be careful while uploading cron entries from a cron-file.txt.