Adding new node to 12c RAC Cluster

This is a step by step article for adding a new node to an existing 2  node 12c Grid Infrastructure cluster. I've demonstrated this post on virtual box. This post will be helpful for those who wants to practice adding of nodes for TESTING/DEMONSTRATION purpose and it will be also helpful for those who wants to add a node in a real production environment.



1) Install OS same as other surviving cluster nodes
2) Configure shared Storage
3) Configure network (public, private and virtual)
4) Configure all OS pre-requisites (users, groups, directories, kernel parameters etc)
5) Run cluvfy to verify addition of node
6) Add node to an existing 12c cluster
7) Verify Services on all cluster nodes
8) Add Instance to a third node

 Configure SSH Setup for racnode1 racnode2 racnode3


$ mkdir .ssh  (Under /ora/oracle/.ssh) if directory not exists.
$ chmod 700 .ssh
$ /usr/bin/ssh-keygen -t  dsa

At the prompts, accept the default location for the key file (press Enter).


Same steps you have to do in rac2,rac3
Then, add the DSA key to the authorized_keys file using the following commands:
$ ls
$ cat id_dsa.pub >> authorized_keys
$ scp authorized_keys rac2:/ora/oracle/.ssh/

RAC3 


$cat id_dsa.pub >> authorized_keys



All racnodes having authorized_keys file including each racnode id_dsa.pub content as well.

Here u can see the authorized_keys file rac1.localdomain,rac2.localdomain,rac3.localdomain entries.




Verify ssh setup from all cluster nodes:


RAC1
-----------------------------------------------

$ssh rac2 date

return date

$ ssh rac3 date

retrun date

-------------------------------------------------


RAC2
-------------------------------------------------

$ssh rac1 date
return date
$ssh rac3 date
return date

RAC3
---------------------------------------------------

$ssh rac1 date
$ssh rac2 date



- Run cluvfy to verify all pre-requisites are in place and node is ready to be added in an existing cluster.

[oracle@rac1] cd /ora/12/grid/bin/
[oracle@rac1] cluvfy stage -pre nodeadd -n rac3 -verbose


6) Add node to an existing 12c cluster:

[oracle@rac1] cd /ora/12/grid/bin/
[oracle@rac1] ./addnode.sh

running root.sh script:


[root@racnode3 ~]# /u01/oraInventory/orainstRoot.sh
Changing permissions of /u01/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/oraInventory to dbarac.
The execution of the script is complete.
[root@racnode3 ~]# 



------------------


[root@racnode3 ~]# sh /u01/grid_12c/root.sh
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /ora/12/grid/

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/grid_12c/crs/install/crsconfig_params
2014/05/13 10:58:57 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2014/05/13 10:59:39 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racnode3'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racnode3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac3'
CRS-2672: Attempting to start 'ora.evmd' on 'rac3'
CRS-2676: Start of 'ora.mdnsd' on 'racnode3' succeeded
CRS-2676: Start of 'ora.evmd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac3'
CRS-2676: Start of 'ora.gpnpd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode3'
CRS-2676: Start of 'ora.gipcd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac3'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac3'
CRS-2676: Start of 'ora.diskmon' on 'racnode3' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'racnode3'
CRS-2676: Start of 'ora.cssd' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'racnode3'
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode3'
CRS-2676: Start of 'ora.ctssd' on 'racnode3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac3'
CRS-2676: Start of 'ora.asm' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac3'
CRS-2676: Start of 'ora.storage' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac3'
CRS-2676: Start of 'ora.crsd' on 'racnode3' succeeded
CRS-6017: Processing resource auto-start for servers: racnode3
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'racnode2'
CRS-2672: Attempting to start 'ora.ons' on 'rac3'
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac3'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan2.vip' on 'racnode2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac3'
CRS-2676: Start of 'ora.scan2.vip' on 'racnode3' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'racnode3'
CRS-2676: Start of 'ora.ons' on 'racnode3' succeeded
CRS-2676: Start of 'ora.DATA.dg' on 'racnode3' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'racnode3' succeeded
CRS-6016: Resource auto-start has completed for server racnode3
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/05/13 11:06:04 CLSRSC-343: Successfully started Oracle clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2014/05/13 11:06:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... suceeded