Tuesday, March 10, 2015

Cannot start ASM - ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”

Today, I faced and issue with an ASM instance. After bouncing the server, CRS went up along with the ASM instance, but the diskgroups were offline.

$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  OFFLINE      hosta
ora.FRA.dg
               ONLINE  OFFLINE      hosta
ora.LISTENER.lsnr
               ONLINE  ONLINE       hosta
ora.LISTENER_1.lsnr
               ONLINE  ONLINE       hosta
ora.asm
               ONLINE  ONLINE       hosta                 Started
ora.ons
               OFFLINE OFFLINE      hosta
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       hosta
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       hosta
ora.database.db
      1        ONLINE  OFFLINE                               Instance Shutdown
ora.database1.db
      1        OFFLINE OFFLINE                               Instance Shutdown

I tried to start ora.DATA.dg resource, but it failed.

$ crsctl start resource ora.DATA.dg
CRS-2672: Attempting to start 'ora.DATA.dg' on 'hosta'
CRS-5017: The resource action "ora.DATA.dg start" encountered the following error:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
. For details refer to "(:CLSN00107:)" in "/u01/oracle/grid/log/hosta/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.DATA.dg' on 'hosta' failed
CRS-2679: Attempting to clean 'ora.DATA.dg' on 'hosta'
CRS-2681: Clean of 'ora.DATA.dg' on 'hosta' succeeded
CRS-4000: Command Start failed, or completed with errors.

After checking the RAW devices on the host, everything appeared to be properly configured. Then I checked the configuration of the ASM instance, finding the ASM_DISKSTRING empty.

$ crsctl stat resource ora.asm -f
NAME=ora.asm
TYPE=ora.asm.type
STATE=OFFLINE
TARGET=OFFLINE
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=
ASM_DISKSTRING=
AUTO_START=restore
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CREATION_SEED=11
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=+ASM
ID=ora.asm
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
SPFILE=+DATA/asm/asmparameterfile/registry.123.785123625
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(ora.cssd)
STOP_TIMEOUT=600
TYPE_VERSION=1.2
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.3.0

I updated ASM_DISKSTRING with the discovery path of the disks and then bounced ASM instance.

$ srvctl modify asm -d ‘/dev/sd*’
$ srvctl stop asm
$ srvctl start asm

After this the ASM instance came up cleanly and the diskgroups were mounted.

$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE      hosta
ora.FRA.dg
               ONLINE  ONLINE      hosta
ora.LISTENER.lsnr
               ONLINE  ONLINE       hosta
ora.LISTENER_1.lsnr
               ONLINE  ONLINE       hosta
ora.asm
               ONLINE  ONLINE       hosta                 Started
ora.ons
               OFFLINE OFFLINE      hosta
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       hosta
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       hosta
ora.database.db
      1        ONLINE  ONLINE                               Open
ora.database1.db
      1        OFFLINE ONLINE                               Open

Hope this help you to troubleshoot and fix the issue on your ASM, when is not able to find the disks.

Thanks,

Alfredo

1 comment:

  1. Thank you. It helped me to fix my issue with asm.

    ReplyDelete