Visit Counter

Monday, June 6, 2016

CLSRSC-117: Failed to start Oracle Clusterware stack Oracle 12c clusterware


Install Grid software on RAC01 was sucessfully but Second RAC02 having a problem while running the root.sh script.


RAC02:


#/ora/grid/root.sh


[INS-35423]The installer has detected that Oracle Clusterware is not running on the local Node.

I am running the runinstaller as oracle user and it can see the cluster fine.

$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Can not Cluster Ready Services is offline
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is offline







/ora/grid/log/rac02/agent/ohasd/oraagent_oracle/oraagent_oracle.log


status msg [CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00443: background process "MMNL" did not start
. For details refer to "(:CLSN00107:)" in "/ora/grid/log/rac02/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
] for start for resource: ora.asm 1 1
2016-06-06 23:03:49.377: [    AGFW][3043145488] {0:9:5} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:584
2016-06-06 23:03:49.449: [ora.asm][3045246736] {0:9:5} [start] (:CLSN00107:) clsn_agent::start }






Solution:


Rac installation  on Virtual Machine

I increased the memory of rac02 server.

[root@rac02 grid]# ./root.sh
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /ora/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /ora/grid/crs/install/crsconfig_params
2016/06/07 00:00:02 CLSRSC-363: User ignored prerequisites during installation

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac02'
CRS-2673: Attempting to stop 'ora.asm' on 'rac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac02'
CRS-2677: Stop of 'ora.asm' on 'rac02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac01'
CRS-2676: Start of 'ora.scan1.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac01'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac02'
CRS-2677: Stop of 'ora.ons' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac02'
CRS-2677: Stop of 'ora.net1.network' on 'rac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac02'
CRS-2677: Stop of 'ora.gpnpd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rac02'
CRS-2673: Attempting to stop 'ora.storage' on 'rac02'
CRS-2677: Stop of 'ora.storage' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac02'
CRS-2677: Stop of 'ora.evmd' on 'rac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac02'
CRS-2677: Stop of 'ora.cssd' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac02'
CRS-2677: Stop of 'ora.gipcd' on 'rac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac02'
CRS-2672: Attempting to start 'ora.evmd' on 'rac02'
CRS-2676: Start of 'ora.mdnsd' on 'rac02' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac02'
CRS-2676: Start of 'ora.gpnpd' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac02'
CRS-2676: Start of 'ora.gipcd' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac02'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac02'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac02'
CRS-2676: Start of 'ora.diskmon' on 'rac02' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rac02'
CRS-2676: Start of 'ora.cssd' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac02'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac02'
CRS-2676: Start of 'ora.ctssd' on 'rac02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac02'
CRS-2676: Start of 'ora.asm' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac02'
CRS-2676: Start of 'ora.storage' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac02'
CRS-2676: Start of 'ora.crsd' on 'rac02' succeeded
CRS-6017: Processing resource auto-start for servers: rac02
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac01'
CRS-2672: Attempting to start 'ora.ons' on 'rac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac02'
CRS-2676: Start of 'ora.scan1.vip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac02'
CRS-2676: Start of 'ora.ons' on 'rac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac02' succeeded
CRS-6016: Resource auto-start has completed for server rac02
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/06/07 00:04:38 CLSRSC-343: Successfully started Oracle clusterware stack

2016/06/07 00:05:22 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@rac02 grid]#