Recovery steps for OCR (Oracle Cluster Registry), voting disks, or ASM passwords in an Oracle RAC 19c
Losing OCR (Oracle Cluster Registry), voting disks, or ASM passwords in an Oracle RAC 19c environment is critical but recoverable. This guide provides a structured recovery approach.
PREREQUISITES
-
Recent backups (OCR, voting disk, ASM metadata)
-
ASM diskgroup redundancy (NORMAL or HIGH)
-
Root/sudo access on all nodes
-
Grid infrastructure installation files
1. RECOVERY FROM OCR LOSS
Symptoms
-
crsctl check crsfails -
Clusterware services won’t start
-
ORA-15077, ORA-29701 errors
Recovery Steps
A. Verify and Stop Clusterware
# As root on all nodes
crsctl stop crs -fB. Restore OCR from Backup
# Identify OCR location and backup
ocrconfig -showbackup
# Restore OCR (from last known good backup)
ocrconfig -restore /u01/app/19.0.0/grid/cdata/<cluster_name>/backup_<timestamp>.ocr
# Alternative: Manual restore if no backup
ocrconfig -import /u01/app/19.0.0/grid/cdata/<cluster_name>/<backup_file>C. Verify OCR Integrity
ocrcheck
cluvfy comp ocr -n all -verboseD. Restart Clusterware
crsctl start crs2. RECOVERY FROM VOTING DISK LOSS
Symptoms
-
Cluster instability
-
Node evictions
-
ORA-15040, ORA-15041 errors
Recovery Steps
A. Stop Clusterware on All Nodes
crsctl stop crs -fB. Identify Voting Disk Locations
crsctl query css votediskC. Restore Voting Disks
# If using ASM redundancy (preferred method)
crsctl replace votedisk +DATA
# If using external redundancy (manual copy)
# 1. Mount ASM diskgroup on one node
# 2. Copy from backup
dd if=/backup/votingdisk.dbf of=/dev/asm_disk bs=4kD. Verify and Start
crsctl query css votedisk
crsctl start crs3. RECOVERY FROM ASM PASSWORD LOSS
Symptoms
-
ASMCA/ASMCMD authentication failures
-
Inability to connect to ASM instances
Recovery Steps
A. Reset ASM Password (Local)
# Connect to ASM instance (as grid user)
export ORACLE_SID=+ASM1
sqlplus / as sysasm
# Reset password
ALTER USER SYS IDENTIFIED BY <new_password>;
ALTER USER ASMSNMP IDENTIFIED BY <new_password>;B. Update Password File on All Nodes
# On each node (as grid user)
orapwd file=$ORACLE_HOME/dbs/orapw+ASM1 password=<new_password> entries=10C. Update Wallet (if used)
# Update Oracle Wallet
mkstore -wrl $ORACLE_HOME/wallet -modifyEntry ORACLE.WALLET.PASSWORD <new_password>4. COMPLETE LOSS SCENARIO (OCR + Voting Disks)
Complete Recovery Procedure
Phase 1: Emergency Shutdown
# On all nodes as root
crsctl stop crs -fPhase 2: Restore ASM Diskgroups
# Start ASM instance in restricted mode
sqlplus / as sysasm
STARTUP RESTRICT;
# Restore from ASM metadata backup (if available)
ALTER DISKGROUP DATA MOUNT RESTRICTED;
ALTER DISKGROUP DATA REBALANCE POWER 1;Phase 3: Recreate OCR and Voting Disks
# Recreate voting disks
crsctl replace votedisk +DATA
# Recreate OCR
ocrconfig -add +DATA
ocrconfig -delete /previous_ocr_locationPhase 4: Reinitialize Clusterware
# Execute on one node only
crsctl start crs -excl
ocrconfig -import /backup/ocr_backup.ocr
crsctl stop crs -fPhase 5: Restart Cluster
# On all nodes
crsctl start crs5. VALIDATION CHECKS
Post-Recovery Verification
# Clusterware status
crsctl check crs crsctl status resource -t
# OCR and voting disk health
ocrcheck
crsctl query css votedisk
# ASM connectivity
asmcmd lsdg
asmcmd lsct6. PREVENTION AND BEST PRACTICES
Backup Strategy
# Regular OCR backups (auto every 4 hours)
ocrconfig -showbackup
# Manual backup
ocrconfig -manualbackup
# ASM metadata backup
ALTER DISKGROUP DATA CHECK ALL REPAIR;Configuration
-
Use ASM redundancy for OCR/voting disks (NORMAL or HIGH)
-
Maintain multiple voting disks (minimum 3, odd number)
-
Regular backups to separate storage
-
Document passwords in secure vault
7. EMERGENCY CONTACTS AND FILES
Critical File Locations
OCR Backups: $GRID_HOME/cdata/<cluster_name>/ Voting Disks: +DATA/<cluster_name>/votingdisk/ Password Files: $ORACLE_HOME/dbs/orapw* ASM SPFILE: +DATA/<cluster_name>/ASMPARAMETERFILE/
This recovery recipe assumes standard Oracle RAC 19c deployments. Adjust for custom configurations, and always refer to Oracle’s official documentation for version-specific details.
