Friday, July 25, 2014

CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server

In my test environment with 2 VM I had a situation when the Oracle cluster was unable to start:

-bash-4.1# ./crsctl stop cluster
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4000: Command Stop failed, or completed with errors.
-bash-4.1# ./crsctl start cluster
CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'rac1'
CRS-4000: Command Start failed, or completed with errors.
-bash-4.1# ./crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.


Looking in ASM at the state of diskgroup (only the one with OCR and voting disks), it was dismounted and the cluster needed it in order to start.

[grid@rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Fri Jul 25 22:42:22 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select NAME , STATE FROM V$ASM_DISKGROUP;

NAME                   STATE
------------------------------ -----------
VOTE                   DISMOUNTED

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@rac1 ~]$ asmcmd mount VOTE
[grid@rac1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Fri Jul 25 22:43:46 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL>                
SQL> select NAME , STATE FROM V$ASM_DISKGROUP;

NAME                   STATE
------------------------------ -----------
VOTE                   MOUNTED

SQL> exit


Then as root, try to start the cluster and check its status.

-bash-4.1# ./crsctl start cluster
CRS-4691: Oracle Clusterware is already running
CRS-4000: Command Start failed, or completed with errors.
-bash-4.1# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                                        
               ONLINE  ONLINE       rac2                                        
ora.VOTE.dg
               ONLINE  ONLINE       rac1                                        
               ONLINE  ONLINE       rac2                                        
ora.asm
               ONLINE  ONLINE       rac1                     Started            
               ONLINE  ONLINE       rac2                     Started            
ora.gsd
               OFFLINE OFFLINE      rac1                                        
               OFFLINE OFFLINE      rac2                                        
ora.net1.network
               ONLINE  ONLINE       rac1                                        
               ONLINE  ONLINE       rac2                                        
ora.ons
               ONLINE  ONLINE       rac1                                        
               ONLINE  ONLINE       rac2                                        
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac1                                        
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac2                                        
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac2                                        
ora.cvu
      1        ONLINE  ONLINE       rac2                                        
ora.oc4j
      1        ONLINE  ONLINE       rac2                                        
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                        
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                        
ora.scan1.vip
      1        ONLINE  ONLINE       rac1                                        
ora.scan2.vip
      1        ONLINE  ONLINE       rac2                                        
ora.scan3.vip
      1        ONLINE  ONLINE       rac2                                        


No comments:

Post a Comment