Visit Counter

Wednesday, April 15, 2015

Oracle Basic RAC Administration Commands



Oracle RAC Basic Administration Commands.

1. Startup / Download database (all nodes)


/ora/12/grid/bin

[oracle@rac1 bin]$ ./srvctl status database -d racora1
Instance racora11 is running on node rac1
Instance racora12 is running on node rac2
Instance racora13 is running on node rac3
[oracle@rac1 bin]$

[oracle@rac1 bin]$ ./srvctl stop database -d racora1

[oracle@rac1 bin]$ ./srvctl status database -d racora1
Instance racora11 is not running on node rac1
Instance racora12 is not running on node rac2
Instance racora13 is not running on node rac3

[oracle@rac1 bin]$ ./srvctl start  database -d racora1

[oracle@rac1 bin]$ ./srvctl status database -d racora1
Instance racora11 is running on node rac1
Instance racora12 is running on node rac2
Instance racora13 is running on node rac3
[oracle@rac1 bin]$



[oracle@rac1 bin]$ ./sqlplus /nolog

SQL*Plus: Release 12.1.0.1.0 Production on Tue Apr 14 21:43:23 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

SQL> conn sys/afi123@racora1 as sysdba
Connected.
SQL> select inst_name from v$active_instances;

INST_NAME
------------------------------------------------------------
rac1.localdomain:racora11
rac2.localdomain:racora12
rac3.localdomain:racora13



SQL> !cat /etc/oratab
#Backup file is  /ora/oracle/app/oracle/product/12.1.0/dbhome_1/srvm/admin/oratab.bak.rac1 line added by Agent
#



# This file is used by ORACLE utilities.  It is created by root.sh
# and updated by either Database Configuration Assistant while creating
# a database or ASM Configuration Assistant while creating ASM instance.

# A colon, ':', is used as the field terminator.  A new line terminates
# the entry.  Lines beginning with a pound sign, '#', are comments.
#
# Entries are of the form:
#   $ORACLE_SID:$ORACLE_HOME:<N|Y>:
#
# The first and second fields are the system identifier and home
# directory of the database respectively.  The third field indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
+ASM1:/ora/12/grid:N: # line added by Agent
racora1:/ora/oracle/app/oracle/product/12.1.0/dbhome_1:N: # line added by Agent

SQL>

SQL> startup
ORACLE instance started.

Total System Global Area  705662976 bytes
Fixed Size    2292384 bytes
Variable Size  360711520 bytes
Database Buffers  339738624 bytes
Redo Buffers    2920448 bytes
Database mounted.
Database opened.
SQL>


SQL> select instance_name,status from gv$instance;

INSTANCE_NAME STATUS
---------------- ------------
racora12 OPEN
racora11 OPEN
racora13 OPEN


2. Start / Stop & check CRS (Cluster Ready Service)


[oracle@rac1 bin]$ ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


[oracle@rac1 bin]$ ./crs_stat -t
Name           Type           Target    State     Host      
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1      
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1      
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1      
ora....N2.lsnr ora....er.type ONLINE    ONLINE    rac3      
ora....N3.lsnr ora....er.type ONLINE    ONLINE    rac2      
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1      
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac2      
ora....network ora....rk.type ONLINE    ONLINE    rac1      
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE              
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1      
ora....SM1.asm application    ONLINE    ONLINE    rac1      
ora....C1.lsnr application    ONLINE    ONLINE    rac1      
ora.rac1.ons   application    ONLINE    ONLINE    rac1      
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1      
ora....SM2.asm application    ONLINE    ONLINE    rac2      
ora....C2.lsnr application    ONLINE    ONLINE    rac2      
ora.rac2.ons   application    ONLINE    ONLINE    rac2      
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2      
ora....SM3.asm application    ONLINE    ONLINE    rac3      
ora....C3.lsnr application    ONLINE    ONLINE    rac3      
ora.rac3.ons   application    ONLINE    ONLINE    rac3      
ora.rac3.vip   ora....t1.type ONLINE    ONLINE    rac3      
ora.racdb.db   ora....se.type ONLINE    OFFLINE              
ora.racora1.db ora....se.type ONLINE    ONLINE    rac1      
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1      
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    rac3      
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    rac2      


[oracle@rac1 bin]$ ./crsctl check cluster -all
**************************************************************
rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac3:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************


3. Start /Stop an individual instance

[oracle@rac1 bin]$ ./srvctl stop instance -d racora1 -i racora13
[oracle@rac1 bin]$ ./srvctl start instance -d racora1 -i racora13

4. Locate the voting Disk


[oracle@rac1 bin]$ ./crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   825b153bbbc84fa7bf3f735175abf312 (/dev/oracleasm/disks/DISK1) [DATA]
Located 1 voting disk(s).

[oracle@rac1 bin]$ ocrcheck
bash: ocrcheck: command not found
[oracle@rac1 bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          4
Total space (kbytes)     :     409568
Used space (kbytes)      :       1628
Available space (kbytes) :     407940
ID                       : 1066761921
Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user


5. To prevent a database starting at boot time.

[oracle@rac1 bin]$ ./srvctl disable database -d racora1



Thursday, April 9, 2015

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED



I've got this message below when trying to connect via SSH.


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is 51:82:00:1c:7e:6f:ac:ac:de:f1:53:08:1c:7d:55:68. Please contact your system administrator. Add correct host key in /Users/isaacalves/.ssh/known_hosts to get rid of this message. Offending RSA key in /Users/isaacalves/.ssh/known_hosts:12 RSA host key for 104.131.16.158 has changed and you have requested strict checking. Host key verification failed.


#cd .ssh
#vi known_hosts

192.0.0.10 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAz6ja9nTPW/DnrUpr81ch Mwb7FXI/pfUVmIxryY/M+6AVIZgTydSVf4wcsyRzCW4R2OzNBx7my59pYpdUd2pVIvzLUr1CEMEVer2x Ej1EVJuziACVv7c+vsfamoD5I9ePTOPE99QzkL70LS85hraGHJclY2bQmrulnbW/75Fhl6k=afi-ssaerp2,192.0.0.11 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAzlXY5C/s36VI0Lkq9wDG
YJnoBufIAD2hDyfC75An1hlVar4gahWmOu191M3ZjpVOT6kZalpNmsL8+XCC3xVcKAWdqiSmDujwUjN1 nryDGT2cY8ml84tIMZat6PaFywdQ20f/nlOjtkP3nMcha1+EUoVlPPzuSI1WEWgIrYjSLE0=
192.0.0.25 ssh-rsa AAAABBAAAAB3NzaC1yc2EAAAABIwAAAIEA8Xcy1QiFMu4xQPOxoKrTf/InqJUi1VERJvoZb7jmv1rcUVTiDFlJJ7PXCgeGjyvUDpUuCd6Zy+1hgcQaSx9/XLB59x9PMEcZDPmwj6dsv4wajZA/v
AyOUbGyHFTpX250bHoq8SvdR9R3iSQ3JYSZ6byZlmX7TwTHt2UihRBMIq8=



I deleted above lines in .ssh/known_hosts to resolve this issue.
Since you updated your ssh key, getting the above message is normal.
Just edit ~/.ssh/known_hosts and delete line 7, as the message pointed you.

Wednesday, April 8, 2015

[INS-30160] Installer has detected the nodes specified for add node operation have uncleaned Inventory



I encountered this error while extending   2 node 12c cluster to 3 nodes.
















Cause:
In /ora/oracle/app/oraInventory directory there was an oraInventory directory exists. Though the directory is emptly oui still assuming that there is some content in this directory.
Action:
Delete/Remove oraInventory directory from nodes3 on which you are facing Issues. OUI will create this directory after successful addition of nodes.

Tuesday, April 7, 2015

Adding new node to 12c RAC Cluster



This is a step by step article for adding a new node to an existing 2  node 12c Grid Infrastructure cluster. I've demonstrated this post on virtual box. This post will be helpful for those who wants to practice adding of nodes for TESTING/DEMONSTRATION purpose and it will be also helpful for those who wants to add a node in a real production environment.



1) Install OS same as other surviving cluster nodes
2) Configure shared Storage
3) Configure network (public, private and virtual)
4) Configure all OS pre-requisites (users, groups, directories, kernel parameters etc)
5) Run cluvfy to verify addition of node
6) Add node to an existing 12c cluster
7) Verify Services on all cluster nodes
8) Add Instance to a third node























































 Configure SSH Setup for racnode1 racnode2 racnode3


$ mkdir .ssh  (Under /ora/oracle/.ssh) if directory not exists.
$ chmod 700 .ssh
$ /usr/bin/ssh-keygen -t  dsa

At the prompts, accept the default location for the key file (press Enter).


Same steps you have to do in rac2,rac3
Then, add the DSA key to the authorized_keys file using the following commands:
$ ls
$ cat id_dsa.pub >> authorized_keys
$ scp authorized_keys rac2:/ora/oracle/.ssh/


















RAC3 


$cat id_dsa.pub >> authorized_keys



All racnodes having authorized_keys file including each racnode id_dsa.pub content as well.

Here u can see the authorized_keys file rac1.localdomain,rac2.localdomain,rac3.localdomain entries.




Verify ssh setup from all cluster nodes:


RAC1
-----------------------------------------------

$ssh rac2 date

return date

$ ssh rac3 date

retrun date

-------------------------------------------------


RAC2
-------------------------------------------------

$ssh rac1 date
return date
$ssh rac3 date
return date

RAC3
---------------------------------------------------

$ssh rac1 date
$ssh rac2 date



- Run cluvfy to verify all pre-requisites are in place and node is ready to be added in an existing cluster.

[oracle@rac1] cd /ora/12/grid/bin/
[oracle@rac1] cluvfy stage -pre nodeadd -n rac3 -verbose


6) Add node to an existing 12c cluster:

[oracle@rac1] cd /ora/12/grid/bin/
[oracle@rac1] ./addnode.sh

























running root.sh script:


[root@racnode3 ~]# /u01/oraInventory/orainstRoot.sh
Changing permissions of /u01/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/oraInventory to dbarac.
The execution of the script is complete.
[root@racnode3 ~]# 



------------------


[root@racnode3 ~]# sh /u01/grid_12c/root.sh
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /ora/12/grid/

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/grid_12c/crs/install/crsconfig_params
2014/05/13 10:58:57 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2014/05/13 10:59:39 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racnode3'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racnode3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac3'
CRS-2672: Attempting to start 'ora.evmd' on 'rac3'
CRS-2676: Start of 'ora.mdnsd' on 'racnode3' succeeded
CRS-2676: Start of 'ora.evmd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac3'
CRS-2676: Start of 'ora.gpnpd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode3'
CRS-2676: Start of 'ora.gipcd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac3'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac3'
CRS-2676: Start of 'ora.diskmon' on 'racnode3' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'racnode3'
CRS-2676: Start of 'ora.cssd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'racnode3'
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode3'
CRS-2676: Start of 'ora.ctssd' on 'racnode3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac3'
CRS-2676: Start of 'ora.asm' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac3'
CRS-2676: Start of 'ora.storage' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac3'
CRS-2676: Start of 'ora.crsd' on 'racnode3' succeeded
CRS-6017: Processing resource auto-start for servers: racnode3
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'racnode2'
CRS-2672: Attempting to start 'ora.ons' on 'rac3'
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac3'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan2.vip' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac3'
CRS-2676: Start of 'ora.scan2.vip' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'racnode3'
CRS-2676: Start of 'ora.ons' on 'racnode3' succeeded
CRS-2676: Start of 'ora.DATA.dg' on 'racnode3' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'racnode3' succeeded
CRS-6016: Resource auto-start has completed for server racnode3
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/05/13 11:06:04 CLSRSC-343: Successfully started Oracle clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2014/05/13 11:06:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... suceeded

























Sunday, April 5, 2015

Enable Archive Log mode in a RAC Environment

--Archive file location

SQL> alter system set log_archive_dest_1='location=/ora/oracle/archive1/' scope=spfile;
system altered.


--Archive file format


SQL> alter system set log_archive_format='arch_%t_%s_%r.arc' scope=spfile;
system altered.


--stop the database


$ srvctl stop database -d racdb1


--start the database in mount


$ srvctl start database -d racdb1  -o mount


--enable the archivelog


sql> alter database archivelog;


--stop the database.


$ srvctl stop database -d racdb1


--start all database


$ srvctl start database -d racdb1


--check the database 


sql> archive log list;


database log mode                   Archive Mode
Automatic archiveal                 Enabled
Archive Destination                 /ora/oracle/archive1
oldert online log sequence       9
Next log sequence to archive  10
current log sequence                  10  

Wednesday, April 1, 2015

Cluster verify utility fails with the user eqivelence test



[oracle@rac1 bin]$ ./cluvfy stage -pre dbcfg -n rac1,rac2 -d /ora/oracle/app/oracle/product/12.1.0/dbhome_1 -verbose 

Performing pre-checks for database configuration

ERROR:
PRVF-0006 : Unable to check the existence of Oracle Home "/ora/oracle/app/oracle/product/12.1.0/dbhome_1"

ERROR:
Oracle home "/ora/oracle/app/oracle/product/12.1.0/dbhome_1" does not exist

Checking node reachability...

Check: Node reachability from node "rac1"
Destination Node Reachable?
------------------------------------ ------------------------
rac1 yes
rac2 yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Status
------------------------------------ ------------------------
rac2 failed
rac1 failed
PRVG-2019 : Check for equivalence of user "oracle" from node "rac1" to node "rac2" failed

PRKC-1044 : Failed to check remote command execution setup for node rac2 using shells /usr/bin/ssh and /usr/bin/rsh
File "/usr/bin/rsh" does not exist on node "rac2"
Agent admitted failure to sign using the key.Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
PRVG-2019 : Check for equivalence of user "oracle" from node "rac1" to node "rac1" failed

PRKC-1044 : Failed to check remote command execution setup for node rac1 using shells /usr/bin/ssh and /usr/bin/rsh
File "/usr/bin/rsh" does not exist on node "rac1"
Agent admitted failure to sign using the key.Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

ERROR:
User equivalence unavailable on all the specified nodes
Verification cannot proceed


Pre-check for database configuration was unsuccessful on all the nodes. 

Fixed: 
Say you have two nodes, rac1 and rac2 and user oracle. You have created the ssh keys in oracle on both nodes.

On node: rac1

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add

On node: rac2


$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add




[oracle@rac1 bin]$ ./cluvfy stage -pre dbcfg -n rac1,rac2 -d /ora/oracle/app/oracle/product/12.1.0/dbhome_1 -verbose




Performing pre-checks for database configuration 

Checking node reachability...

Check: Node reachability from node "rac1"
  Destination Node                      Reachable?              
  ------------------------------------  ------------------------
  rac1                                  yes                     
  rac2                                  yes                     
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status                  
  ------------------------------------  ------------------------
  rac2                                  passed                  
  rac1                                  passed                  
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status                  
  ------------------------------------  ------------------------
  rac1                                  passed                  
  rac2                                  passed                  

Verification of the hosts config file successful


Interface information for node "rac1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.10.1    192.168.10.0    0.0.0.0         UNKNOWN         08:00:27:23:AD:3F 1500  
 eth0   169.254.134.21  169.254.0.0     0.0.0.0         UNKNOWN         08:00:27:23:AD:3F 1500  
 eth2   192.0.0.22      192.0.0.0       0.0.0.0         UNKNOWN         08:00:27:F9:4D:79 1500  
 eth2   192.0.0.26      192.0.0.0       0.0.0.0         UNKNOWN         08:00:27:F9:4D:79 1500  
 eth2   192.0.0.34      192.0.0.0       0.0.0.0         UNKNOWN         08:00:27:F9:4D:79 1500  
 eth2   192.0.0.36      192.0.0.0       0.0.0.0         UNKNOWN         08:00:27:F9:4D:79 1500  


Interface information for node "rac2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU   
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.10.2    192.168.10.0    0.0.0.0         UNKNOWN         08:00:27:23:AD:3F 1500  
 eth0   169.254.190.151 169.254.0.0     0.0.0.0         UNKNOWN         08:00:27:23:AD:3F 1500  
 eth2   192.0.0.23      192.0.0.0       0.0.0.0         UNKNOWN         08:00:27:F9:4D:79 1500  
 eth2   192.0.0.35      192.0.0.0       0.0.0.0         UNKNOWN         08:00:27:F9:4D:79 1500  
 eth2   192.0.0.29      192.0.0.0       0.0.0.0         UNKNOWN         08:00:27:F9:4D:79 1500  


Check: Node connectivity using interfaces on subnet "192.0.0.0"

Check: Node connectivity of subnet "192.0.0.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  rac1[192.0.0.36]                rac1[192.0.0.22]                yes             
  rac1[192.0.0.36]                rac2[192.0.0.23]                yes             
  rac1[192.0.0.36]                rac1[192.0.0.26]                yes             
  rac1[192.0.0.36]                rac1[192.0.0.34]                yes             
  rac1[192.0.0.36]                rac2[192.0.0.35]                yes             
  rac1[192.0.0.36]                rac2[192.0.0.29]                yes             
  rac1[192.0.0.22]                rac2[192.0.0.23]                yes             
  rac1[192.0.0.22]                rac1[192.0.0.26]                yes             
  rac1[192.0.0.22]                rac1[192.0.0.34]                yes             
  rac1[192.0.0.22]                rac2[192.0.0.35]                yes             
  rac1[192.0.0.22]                rac2[192.0.0.29]                yes             
  rac2[192.0.0.23]                rac1[192.0.0.26]                yes             
  rac2[192.0.0.23]                rac1[192.0.0.34]                yes             
  rac2[192.0.0.23]                rac2[192.0.0.35]                yes             
  rac2[192.0.0.23]                rac2[192.0.0.29]                yes             
  rac1[192.0.0.26]                rac1[192.0.0.34]                yes             
  rac1[192.0.0.26]                rac2[192.0.0.35]                yes             
  rac1[192.0.0.26]                rac2[192.0.0.29]                yes             
  rac1[192.0.0.34]                rac2[192.0.0.35]                yes             
  rac1[192.0.0.34]                rac2[192.0.0.29]                yes             
  rac2[192.0.0.35]                rac2[192.0.0.29]                yes             
Result: Node connectivity passed for subnet "192.0.0.0" with node(s) rac1,rac2


Check: TCP connectivity of subnet "192.0.0.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  rac1:192.0.0.36                 rac1:192.0.0.22                 passed          
  rac1:192.0.0.36                 rac2:192.0.0.23                 passed          
  rac1:192.0.0.36                 rac1:192.0.0.26                 passed          
  rac1:192.0.0.36                 rac1:192.0.0.34                 passed          
  rac1:192.0.0.36                 rac2:192.0.0.35                 passed          
  rac1:192.0.0.36                 rac2:192.0.0.29                 passed          
Result: TCP connectivity check passed for subnet "192.0.0.0"


Check: Node connectivity using interfaces on subnet "192.168.10.0"

Check: Node connectivity of subnet "192.168.10.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  rac2[192.168.10.2]              rac1[192.168.10.1]              yes             
Result: Node connectivity passed for subnet "192.168.10.0" with node(s) rac2,rac1


Check: TCP connectivity of subnet "192.168.10.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  rac2:192.168.10.2               rac1:192.168.10.1               passed          
Result: TCP connectivity check passed for subnet "192.168.10.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed for subnet "192.0.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.10.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Check: Group existence for "dba" 
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  rac2          passed                    exists                  
  rac1          passed                    exists                  
Result: Group existence check passed for "dba"

Check: Total memory 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          2.1961GB (2302768.0KB)    1GB (1048576.0KB)         passed    
  rac1          2.1961GB (2302768.0KB)    1GB (1048576.0KB)         passed    
Result: Total memory check passed

Check: Available memory 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          1.614GB (1692392.0KB)     50MB (51200.0KB)          passed    
  rac1          1.4389GB (1508812.0KB)    50MB (51200.0KB)          passed    
Result: Available memory check passed

Check: Swap space 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          9.7656GB (1.0239992E7KB)  2.1961GB (2302768.0KB)    passed    
  rac1          9.7656GB (1.0239992E7KB)  2.1961GB (2302768.0KB)    passed    
Result: Swap space check passed

Check: Free disk space for "rac2:/tmp" 
  Path              Node Name     Mount point   Available     Required      Status      
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac2          /             8.1064GB      1GB           passed      
Result: Free disk space check passed for "rac2:/tmp"

Check: Free disk space for "rac1:/tmp" 
  Path              Node Name     Mount point   Available     Required      Status      
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rac1          /             8.1104GB      1GB           passed      
Result: Free disk space check passed for "rac1:/tmp"

Check: User existence for "oracle" 
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  rac2          passed                    exists(501)             
  rac1          passed                    exists(501)             

Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed 
Result: User existence check passed for "oracle"

Check: Group existence for "oinstall" 
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  rac2          passed                    exists                  
  rac1          passed                    exists                  
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba" 
  Node Name     Status                    Comment                 
  ------------  ------------------------  ------------------------
  rac2          passed                    exists                  
  rac1          passed                    exists                  
Result: Group existence check passed for "dba"

Check: Membership of user "oracle" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status      
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac2              yes           yes           yes           yes           passed      
  rac1              yes           yes           yes           yes           passed      
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Check: Membership of user "oracle" in group "dba" 
  Node Name         User Exists   Group Exists  User in Group  Status          
  ----------------  ------------  ------------  ------------  ----------------
  rac2              yes           yes           yes           passed          
  rac1              yes           yes           yes           passed          
Result: Membership check for user "oracle" in group "dba" passed

Check: Run level 
  Node Name     run level                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          5                         3,5                       passed    
  rac1          5                         3,5                       passed    
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors" 
  Node Name         Type          Available     Required      Status          
  ----------------  ------------  ------------  ------------  ----------------
  rac2              hard          65536         65536         passed          
  rac1              hard          65536         65536         passed          
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors" 
  Node Name         Type          Available     Required      Status          
  ----------------  ------------  ------------  ------------  ----------------
  rac2              soft          4096          1024          passed          
  rac1              soft          4096          1024          passed          
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes" 
  Node Name         Type          Available     Required      Status          
  ----------------  ------------  ------------  ------------  ----------------
  rac2              hard          16384         16384         passed          
  rac1              hard          16384         16384         passed          
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes" 
  Node Name         Type          Available     Required      Status          
  ----------------  ------------  ------------  ------------  ----------------
  rac2              soft          2047          2047          passed          
  rac1              soft          2047          2047          passed          
Result: Soft limits check passed for "maximum user processes"

Check: System architecture 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          x86_64                    x86_64                    passed    
  rac1          x86_64                    x86_64                    passed    
Result: System architecture check passed

Check: Kernel version 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          2.6.32-100.28.5.el6.x86_64  2.6.32                    passed    
  rac1          2.6.32-100.28.5.el6.x86_64  2.6.32                    passed    
Result: Kernel version check passed

Check: Kernel parameter for "semmsl" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              250           250           250           passed          
  rac2              250           250           250           passed          
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              32000         32000         32000         passed          
  rac2              32000         32000         32000         passed          
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              100           100           100           passed          
  rac2              100           100           100           passed          
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              128           128           128           passed          
  rac2              128           128           128           passed          
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              33554432      unknown       1179017216    failed        Current value incorrect. Configured value unknown. 
  rac2              33554432      unknown       1179017216    failed        Current value incorrect. Configured value unknown. 
Result: Kernel parameter check failed for "shmmax"

Check: Kernel parameter for "shmmni" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4096          4096          4096          passed          
  rac2              4096          4096          4096          passed          
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              2097152       unknown       230276        failed (ignorable)  Configured value unknown. 
  rac2              2097152       unknown       230276        failed (ignorable)  Configured value unknown. 
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              6815744       6815744       6815744       passed          
  rac2              6815744       6815744       6815744       passed          
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed          
  rac2              between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed          
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              262144        262144        262144        passed          
  rac2              262144        262144        262144        passed          
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              4194304       4194304       4194304       passed          
  rac2              4194304       4194304       4194304       passed          
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              126976        unknown       262144        failed        Current value incorrect. Configured value unknown. 
  rac2              126976        unknown       262144        failed        Current value incorrect. Configured value unknown. 
Result: Kernel parameter check failed for "wmem_default"

Check: Kernel parameter for "wmem_max" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048586       1048586       1048576       passed          
  rac2              1048586       1048586       1048576       passed          
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac1              1048576       1048576       1048576       passed          
  rac2              1048576       1048576       1048576       passed          
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          binutils-2.20.51.0.2-5.11.el6  binutils-2.20.51.0.2      passed    
  rac1          binutils-2.20.51.0.2-5.11.el6  binutils-2.20.51.0.2      passed    
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libcap1-1.10-1     compat-libcap1-1.10       passed    
  rac1          compat-libcap1-1.10-1     compat-libcap1-1.10       passed    
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed    
  rac1          compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed    
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          libgcc(x86_64)-4.4.4-13.el6  libgcc(x86_64)-4.4.4      passed    
  rac1          libgcc(x86_64)-4.4.4-13.el6  libgcc(x86_64)-4.4.4      passed    
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++(x86_64)-4.4.4-13.el6  libstdc++(x86_64)-4.4.4   passed    
  rac1          libstdc++(x86_64)-4.4.4-13.el6  libstdc++(x86_64)-4.4.4   passed    
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          libstdc++-devel(x86_64)-4.4.4-13.el6  libstdc++-devel(x86_64)-4.4.4  passed    
  rac1          libstdc++-devel(x86_64)-4.4.4-13.el6  libstdc++-devel(x86_64)-4.4.4  passed    
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          sysstat-9.0.4-11.el6      sysstat-9.0.4             passed    
  rac1          sysstat-9.0.4-11.el6      sysstat-9.0.4             passed    
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-4.4.4-13.el6          gcc-4.4.4                 passed    
  rac1          gcc-4.4.4-13.el6          gcc-4.4.4                 passed    
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          gcc-c++-4.4.4-13.el6      gcc-c++-4.4.4             passed    
  rac1          gcc-c++-4.4.4-13.el6      gcc-c++-4.4.4             passed    
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          missing                   ksh-...                   failed    
  rac1          missing                   ksh-...                   failed    
Result: Package existence check failed for "ksh"

Check: Package existence for "make" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          make-3.81-19.el6          make-3.81                 passed    
  rac1          make-3.81-19.el6          make-3.81                 passed    
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc(x86_64)-2.12-1.7.el6  glibc(x86_64)-2.12        passed    
  rac1          glibc(x86_64)-2.12-1.7.el6  glibc(x86_64)-2.12        passed    
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          glibc-devel(x86_64)-2.12-1.7.el6  glibc-devel(x86_64)-2.12  passed    
  rac1          glibc-devel(x86_64)-2.12-1.7.el6  glibc-devel(x86_64)-2.12  passed    
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed    
  rac1          libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed    
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)" 
  Node Name     Available                 Required                  Status    
  ------------  ------------------------  ------------------------  ----------
  rac2          libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed    
  rac1          libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed    
Result: Package existence check passed for "libaio-devel(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed 

Check: Current group ID 
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status                  
  ------------------------------------  ------------------------
  rac2                                  passed                  
  rac1                                  passed                  

Check for consistency of root user's primary group passed

Checking CRS integrity...

Clusterware version consistency passed.
The Oracle Clusterware is healthy on node "rac1"
The Oracle Clusterware is healthy on node "rac2"

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application (required)
  Node Name     Required                  Running?                  Comment   
  ------------  ------------------------  ------------------------  ----------
  rac1          yes                       yes                       passed    
  rac2          yes                       yes                       passed    
VIP node application check passed

Checking existence of NETWORK node application (required)
  Node Name     Required                  Running?                  Comment   
  ------------  ------------------------  ------------------------  ----------
  rac1          yes                       yes                       passed    
  rac2          yes                       yes                       passed    
NETWORK node application check passed

Checking existence of ONS node application (optional)
  Node Name     Required                  Running?                  Comment   
  ------------  ------------------------  ------------------------  ----------
  rac1          no                        yes                       passed    
  rac2          no                        yes                       passed    
ONS node application check passed

Check: Time zone consistency 
Result: Time zone consistency check passed

Pre-check for database configuration was unsuccessful on all the nodes. 
[oracle@rac1 bin]$