Verifying the DR Readiness of a VVR Setup

When setting up a Disaster Recovery (DR) solution, it is very important to verify the effectiveness of the DR solution. Although VVR guarantees that integrity of data is maintained between the Primary and Secondary data volumes, validating data is necessary to ensure that there has been no data loss due to administrative error, user error, or some other technical reasons. Validation also helps you to be certain that the data that has been replicated to the Secondary (Disaster Recovery site) can be used to bring up the applications in case of a disaster.

The way to validate the DR readiness of the DR site is to bring up the application on the DR site. This can be done in two ways. One is to migrate the Primary role to the Secondary and then run the applications on the new Secondary using the replicated data. Another way of performing a firedrill is using the snapshot feature. Using this feature VVR creates snapshots of the data volumes that can be used to bring up the application on the Secondary.

Data validation can be used to verify the integrity of the data that has been replicated to the Secondary from the Primary. This is done by comparing the data with the data on the Primary. When the Secondary data volumes are validated after the replication has been stopped, the volumes are dissociated from an RVG. This could be very useful in case you want to validate the data volume before adding it back to the RDS. However, data can also be validated online, that is, when the replication is in progress. This is achieved by using the instant space-optimized snapshot feature to create point in time snapshots of the primary and secondary data volumes. In this case, instead of the actual volumes the snapshot volumes are compared and validated. For information on the methods to create the snapshots, refer to Creating RVG Snapshots.

VVR enables you to validate the DR Readiness of the Secondary by using one of the following methods.

• Performing a Failover
• Performing a Firedrill
• Verifying the Data on the Secondary

Performing a Failover

A disaster like scenario can be tested by using the migrate operation to perform a complete failover testing. This can be done by migrating the role of the Secondary to a Primary and making sure that the application is running on the new Primary. For more information on how we can perform the role transfer refer to the chapter Transferring the Primary Role.

Performing a Firedrill

Firedrill is a process of bringing up the application on the Secondary using the replicated data. This data is then used to perform some processing in order to validate the consistency and correctness of the data.

To test the failover you can use a point-in-time image of the data on the Secondary data volumes. VVR provides you with the option of creating instant full and space-optimized snapshots. For any information on the methods to create snapshots see section Creating RVG Snapshots. You can use the appropriate type of snapshot method to create the snapshots. The instant space-optimized snapshot has much lesser space requirement compared to the instant full or plex-breakoff snapshots. These space-optimized snapshots are used to test the Secondary failover.

Points to Note

• A firedrill cannot be performed using the Secondary volumes therefore the snapshots must be used.
• The secondary must be in a consistent state when the snapshots are taken.
• When performing a firedrill no IBC messages are required to be sent so that a failover scenario similar to a real one is simulated.

Automating the Firedrill Procedure

The firedrill procedure is most effective only when it is performed on a regular basis. The above method requires you to test the Secondary failover, manually, at frequent intervals. However, in case VVR is used in a VCS setup where the appropriate agents are installed, then the firedrill procedure can be automated using the RVGSnapshot and RVGPrimary agents that VCS provides. For more information on how you can use these agents to automate the firedrill testing, refer to the VCS Documents.

Verifying the Data on the Secondary

VVR enables you to verify that the data on the Secondary is identical to the data on the Primary data volumes, either when the application is active or inactive. VVR provides the following methods to verify the data at the Secondary site: online data verification and offline data verification.

Online data verification allows you to validate the data even when replication is in progress. In this method instead of the actual volumes, the point-in-time snapshots are compared. This method is referred to as online data verification.

Offline data verification can be performed only when replication is not active. If you have already created the Primary and Secondary volumes and replication is in progress, you need to pause replication and then perform data validation between the corresponding Primary and Secondary volumes to make sure that the data volumes on the Primary and Secondary are the same. To do this, use the vradmin syncrvg command with the -verify option. To use this command for verifying the data, the Secondary must be up-to-date. This command performs a comparison of the checksums on the corresponding Primary and Secondary volumes.

You can also validate the data on new data volumes before adding them to an RDS. For information, refer to Verifying the Data on the Primary and Secondary Volumes.

Performing Online Data Verification

The space-optimized snapshots that are created using the vxrvg snapshot command can be used to verify whether the data on the Primary and Secondary RVG volumes is the same.

The major advantage of this feature over the vradmin -verify syncrvg command is that you do not need to stop the replication. The verification can be done even while the replication is in progress because the point-in-time snapshots, and not the volumes, are compared. This feature is very useful if you want to check the integrity of the data volumes on the Secondary when replication is in progress.

The vradmin verifydata command creates the space-optimized snapshots on the Primary and the Secondary before it proceeds with performing online data verification. The vradmin verifydata command also ensures that the snapshots are taken only after the replication has been paused using the vxibc freeze command. As a result there may be a momentary pause in the replication. It is necessary to freeze the writes so that the snapshots can be taken at an identical point in replication time, on each of the required hosts.

The vradmin verifydata then verifies the data between the remote and local hosts by comparing the space-optimized snapshots.

To perform online data verification, use the command:

The attribute sechost specifies the name of the Secondary host.

The cache attribute specifies a name for the precreated cache object, on which the snapshots for the volumes in the specified RVG will be created. The cachesize attribute specifies a default size for the cache object with respect to the source volume. If the RVG on either the Primary or the Secondary has VxVM ISP volumes, then you cannot use the cachesize attribute.

You must specify only one of these attributes at one time for the command to create a cache object for each snapshot.

This command performs the following tasks:

• Registering the application on the Primary and the Secondary.
• Freezing replication on the Primary and Secondary.
• Taking snapshots and verifying the data.
• Destroying the snapshots.

By default, the vradmin verifydata command destroys the snapshot volume and the cache object after the data verification has proceeded successfully. However, if you want to preserve the snapshot volumes then you must use the vradmin verifydata command with the -k snap option. If you want to preserve the cache object then use the
vradmin verifydata command with the -k cache option. The same cache object can then be reused when creating future snapshots. You cannot use the -k option if you have used the cachesize option, as it is an invalid combination and the command fails with an error message. Note that when specifying the -k option you must specify either the cache or the snap argument with it.

VVR also provides you with sample scripts that can be used to freeze the replication and then take instant space-optimized snapshots. For details, refer to Sample Scripts.

1. Prepare the volumes that need to be included in the snapshot. For more information on preparing volumes, refer to Preparing the Volumes.
2. Create the required cache object within the same disk group as the data volume. For more details on creating the cache object, refer to Preparing the RVG Volumes for Snapshot Operation.
3. Use the following command to perform online data verification:
   vradmin [-g diskgroup] [-k {cache|snap}] verifydata rvg_name \
     sechost {cache=cacheobj | cachesize=size}

Performing Offline Data Verification

VVR enables you to verify whether the data on the Secondary is identical to the data on the Primary data volumes when the application is inactive. The vradmin syncrvg command with the -verify option verifies and reports any differences between the data volumes associated with the Secondary RVG and the corresponding Primary RVG. The vradmin -verify syncrvg command only reports whether the Primary and Secondary volumes are identical or not. It does not make them identical. As the command runs, it reports the progress every 10 seconds. An MD5 checksum is used to calculate the difference between the Primary and the Secondary data volumes. Refer to Using Difference-Based Synchronization.

Prerequisite for using the vradmin -verify syncrvg command:

  All applications using the Primary data volumes must be stopped before running the vradmin -verify syncrvg command.

  To verify the differences between the Primary and Secondary data volumes

When this command is invoked, you are prompted to confirm that the Primary data volumes are not in use. You can use the -s option to skip this confirmation step.

The argument local_rvgname is the name of the RVG on the local host and represents the RDS.

The argument sec_hostname is a space-separated list of the names of the Secondary hosts as displayed in the output of the vradmin printrvg command.

This command checks the status of the Primary RLINK to each of the Secondary RVGs being verified. If any of the RLINKs are not up-to-date, the vradmin -verify syncrvg command returns with a message to indicate that the RLINKs are not up-to-date. In this scenario, verification is not be performed. Use the vxrlink status command to determine the extent to which the Secondary is behind.

Example:

To verify the data differences between the Primary RVG hr_rvg on seattle and the Secondary RVG on host london, issue the following command from any host in the RDS:

The output resembles the following if the Primary and Secondary data volumes
are identical:

If there are differences in the data volumes, the output looks similar to the one shown below: