Disaster Recovery in Veritas Volume Manager 4.x
I just tried to recover data from a destroyed diskgroup in Veritas volume manager using its simple restoration facility. Here, I have written two scenarios where the possible data loss can occur:
Case 1: System Admin [SA] has mistakenly destroyed volumes and diskgroup
Case 2: SA has uninitialized [unsetup] the disks from Veritas
Case 1: Recreating volumes from a destroyed diskgroup
I have created few test volumes:
bash-3.00$ df -kh
Filesystem size used avail capacity Mounted on
-------------- o/p truncated ------------------------------
/dev/vx/dsk/testdg/TestVol01
5.0G 18M 4.7G 1% /TestVol01
/dev/vx/dsk/testdg/TestVol02
5.0G 18M 4.7G 1% /TestVol02
/dev/vx/dsk/testdg/TestVol03
5.0G 18M 4.7G 1% /TestVol03
/dev/vx/dsk/testdg/TestVol04
4.0G 18M 3.7G 1% /TestVol04
Configbackup daemon usually saves the config info at "/etc/dgbackups"; and we can initiate the backup at our convenience too.
bash-3.00$ sudo ls -l /etc/dgbackups
total 62
-rw------- 1 root other 31681 Apr 22 19:41 testdg.04-22-2009-4142
bash-3.00$ sudo /usr/lib/vxvm/bin/vxconfigbackup -l /etc/dgbackups
Start backing up diskgroup testdg to /etc/dgbackups/testdg.1240614864.103.diego ...
VxVM NOTICE V-5-2-3100 Backup complete for diskgroup: testdg
bash-3.00$ sudo ls -l /etc/dgbackups
total 64
-rw------- 1 root other 31681 Apr 22 19:41 testdg.04-22-2009-4142
drwxr-xr-x 2 root other 512 Apr 24 23:29 testdg.1240614864.103.diego
bash-3.00$ sudo ls -l /etc/dgbackups/testdg.1240614864.103.diego
total 1548
-rw-r-r- 1 root other 758272 Apr 24 23:29 1240614864.103.diego.binconfig
-rw-r-r- 1 root other 16957 Apr 24 23:28 1240614864.103.diego.cfgrec
-rw-r-r- 1 root other 1292 Apr 24 23:28 1240614864.103.diego.dginfo
-rw-r-r- 1 root other 2197 Apr 24 23:28 1240614864.103.diego.diskinfo
I`m now destroying the DG and trying to recover the volumes back using the config info that I have saved, as stated above.
This would fail, as shown below, since there are two backups existing under the given path for the same DG.
bash-3.00# /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups testdg
VxVM vxconfigrestore ERROR V-5-2-3717 There are two backups that have the same diskgroup name testdg with different diskgroup id:
cat: cannot open /etc/dgbackups/testdg.04-22-2009-4142_bkp/04-22-2009-4142_bkp.dginfo
04-22-2009-4142_bkp -- backup at
1240614864.103.diego -- backup at Fri Apr 24 23:28:59 GMT 2009
VxVM vxconfigrestore INFO V-5-2-3721 Use diskgroup_id to do the restoration.
So, I just move the old one and start using the new backup:
bash-3.00# mv testdg.04-22-2009-4142 /export/home/<MyHomeDir>/
bash-3.00# ls -l
total 2
drwxr-xr-x 2 root other 512 Apr 24 23:29 testdg.1240614864.103.diego
bash-3.00# /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups testdg
Diskgroup testdg configuration restoration started ......
Installing volume manager disk header for Disk_0 ...
Installing volume manager disk header for Disk_2 ...
|
testdg's diskgroup configuration is restored (in precommit state).
Diskgroup can be accessed as read-only and can be examined using
vxprint in this state.
Run:
vxconfigrestore -l /etc/dgbackups -c testdg ==> to commit the restoration.
vxconfigrestore -l /etc/dgbackups -d testdg ==> to abort the restoration.
Restoration is not committed yet; I`ll just verify the volume status and if it looks good, it can be committed.
-----------------------------------------------------------------------------------------------------------------
bash-3.00# vxprint -Aht
Disk group: testdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg testdg default default 40000 1240614864.103.diego
dm testdg01 Disk_0 auto 2303 20967040 -
dm testdg02 Disk_2 auto 2047 20952992 -
v TestVol01 - ENABLED ACTIVE 10485760 SELECT TestVol01-01 fsgen
pl TestVol01-01 TestVol01 ENABLED ACTIVE 10485760 STRIPE 2/512 RW
sd testdg01-01 TestVol01-01 testdg01 0 5242880 0/0 Disk_0 ENA
sd testdg02-01 TestVol01-01 testdg02 0 5242880 1/0 Disk_2 ENA
v TestVol02 - ENABLED ACTIVE 10485760 SELECT TestVol02-01 fsgen
pl TestVol02-01 TestVol02 ENABLED ACTIVE 10485760 STRIPE 2/512 RW
sd testdg01-02 TestVol02-01 testdg01 5242880 5242880 0/0 Disk_0 ENA
sd testdg02-02 TestVol02-01 testdg02 5242880 5242880 1/0 Disk_2 ENA
v TestVol03 - ENABLED ACTIVE 10485760 SELECT TestVol03-01 fsgen
pl TestVol03-01 TestVol03 ENABLED ACTIVE 10485760 STRIPE 2/512 RW
sd testdg01-03 TestVol03-01 testdg01 10485760 5242880 0/0 Disk_0 ENA
sd testdg02-03 TestVol03-01 testdg02 10485760 5242880 1/0 Disk_2 ENA
v TestVol04 - ENABLED ACTIVE 8388608 SELECT TestVol04-01 fsgen
pl TestVol04-01 TestVol04 ENABLED ACTIVE 8388608 STRIPE 2/512 RW
sd testdg01-04 TestVol04-01 testdg01 15728640 4194304 0/0 Disk_0 ENA
sd testdg02-04 TestVol04-01 testdg02 15728640 4194304 1/0 Disk_2 ENA
------------------------------------------------------------------------------------------------------------------
bash-3.00# /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups -c testdg
Committing configuration restoration for diskgroup testdg ....
testdg's diskgroup configuration restoration is committed.
Volumes are now recovered and everything looks good.
Case 2: Recovering volumes from uninitialized disks
If I use the above procedure after uninitializing disks, I`ll land upon this error:
bash-3.00# /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups testdg
Diskgroup testdg configuration restoration started ......
Installing volume manager disk header for Disk_0 ...
Disk_0 disk format has been changed from sliced to none.
VxVM vxdisk ERROR V-5-1-5433 Device Disk_0: init failed:
Disk VTOC does not list private partition
Installing volume manager disk header for Disk_2 ...
Disk_2 disk format has been changed from sliced to none.
VxVM vxdisk ERROR V-5-1-5433 Device Disk_2: init failed:
Disk VTOC does not list private partition
VxVM vxconfigrestore ERROR V-5-2-3710 No disk(s) available for restoration. Diskgroup configuration restoration for testdg failed.
If you think that you can encapsulate those disks again and would include it into the diskgroup, you`re wrong. It won`t allow you to encapsulate the disks where the VTOC would still look like a valid one. But you can reinitialize it..
Let`s see how can it be done:
Volumes in good state:
bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
Disk_0 auto:sliced testdg04 testdg online
Disk_1 auto:sliced testdg03 testdg online
c0t0d0s2 auto:none - - online invalid
c0t1d0s2 auto:none - - online invalid
bash-3.00# vxprint -Aht
Disk group: testdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg testdg default default 6000 1240731089.169.f50318-11
dm testdg03 Disk_1 auto 2303 20967040 -
dm testdg04 Disk_0 auto 2047 20952992 -
v TestVol01 - ENABLED ACTIVE 10485760 SELECT TestVol01-01 fsgen
pl TestVol01-01 TestVol01 ENABLED ACTIVE 10485760 STRIPE 2/512 RW
sd testdg04-01 TestVol01-01 testdg04 0 5242880 0/0 Disk_0 ENA
sd testdg03-01 TestVol01-01 testdg03 0 5242880 1/0 Disk_1 ENA
v TestVol02 - ENABLED ACTIVE 10485760 SELECT TestVol02-01 fsgen
pl TestVol02-01 TestVol02 ENABLED ACTIVE 10485760 STRIPE 2/512 RW
sd testdg04-02 TestVol02-01 testdg04 5242880 5242880 0/0 Disk_0 ENA
sd testdg03-02 TestVol02-01 testdg03 5242880 5242880 1/0 Disk_1 ENA
v TestVol03 - ENABLED ACTIVE 10485760 SELECT TestVol03-01 fsgen
pl TestVol03-01 TestVol03 ENABLED ACTIVE 10485760 STRIPE 2/512 RW
sd testdg04-03 TestVol03-01 testdg04 10485760 5242880 0/0 Disk_0 ENA
sd testdg03-03 TestVol03-01 testdg03 10485760 5242880 1/0 Disk_1 ENA
v TestVol04 - ENABLED ACTIVE 8388608 SELECT TestVol04-01 fsgen
pl TestVol04-01 TestVol04 ENABLED ACTIVE 8388608 STRIPE 2/512 RW
sd testdg04-04 TestVol04-01 testdg04 15728640 4194304 0/0 Disk_0 ENA
sd testdg03-04 TestVol04-01 testdg03 15728640 4194304 1/0 Disk_1 ENA
Writing some data into the volumes:
bash-3.00# df -kh | grep -i test
/dev/vx/dsk/testdg/TestVol01 5.0G 621M 4.1G 13% /TestVol01
/dev/vx/dsk/testdg/TestVol02 5.0G 75M 4.6G 2% /TestVol02
/dev/vx/dsk/testdg/TestVol03 5.0G 39M 4.7G 1% /TestVol03
/dev/vx/dsk/testdg/TestVol04 4.0G 39M 3.7G 2% /TestVol04
Destroying volumes and uninitializing disks:
bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
Disk_0 auto:none - - online invalid
Disk_1 auto:none - - online invalid
c0t0d0s2 auto:none - - online invalid
c0t1d0s2 auto:none - - online invalid
-------------------------------------------------------------------------------------------------
I now want to restore all the volumes to the last known state; Reinitializing and adding into DG using "vxdiskadm":
bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
Disk_0 auto:sliced testdg04 testdg online
Disk_1 auto:sliced testdg03 testdg online
c0t0d0s2 auto:none - - online invalid
c0t1d0s2 auto:none - - online invalid
I`m now manually creating a file with the volume specifications as follows; It`s for the volume TestVol01:
bash-3.00# cat /tmp/TestVol01
sd testdg04-01 disk=testdg04 len=5242880
sd testdg03-01 disk=testdg03 len=5242880
plex TestVol01-01 layout=STRIPE ncolumn=2 stwidth=512 sd=testdg04-01:0/0,testdg03-01:1/0
vol TestVol01 readpol=SELECT usetype=fsgen len=10485760 plex=TestVol01-01
Using that file, I'll create the deleted volume.
Before recovery:
bash-3.00# vxprint -Aht
Disk group: testdg DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO dg testdg default default 18000 1240732350.175.f50318-11 dm testdg03 Disk_1 auto 2303 20967040 -
dm testdg04 Disk_0 auto 2047 20952992 -
Recreating the volume 'TestVol01` from the specifications mentioned in the file:
bash-3.00# vxmake -g testdg -d /tmp/TestVol01
After recovery:
bash-3.00# vxprint -Aht
Disk group: testdg DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO dg testdg default default 18000 1240732350.175.f50318-11 dm testdg03 Disk_1 auto 2303 20967040 -
dm testdg04 Disk_0 auto 2047 20952992 - v TestVol01 - DISABLED EMPTY 10485760 SELECT - fsgen
pl TestVol01-01 TestVol01 DISABLED EMPTY 10485760 STRIPE 2/512 RW
sd testdg04-01 TestVol01-01 testdg04 0 5242880 0/0 Disk_0 ENA
sd testdg03-01 TestVol01-01 testdg03 0 5242880 1/0 Disk_1 ENA
I now start up the volume and mount it on the respective mountpoint [without creating new fs]:
bash-3.00# df -kh | grep -i test
/dev/vx/dsk/testdg/TestVol01 5.0G 621M 4.1G 13% /TestVol01
(Not displaying the contents of the filesystems)
==> We`ve got our data back 
Note: Please let me know if you need any additional info on this or if you have any better idea about how to achieve this.
Thanks,
Sudhakar R
|
Sign up or Log in to add a comment or watch this page.