Symptoms:
The ZVM service and/or ZVM VM are malfunctioning. You are likely to encounter some of the following:
  • You cannot login into the ZVM UI
  • A ZVM disconnection alert is displayed
  • Windows Event Viewer reports a ZVM application error
Cause:
The ZVR Solution has the ZVM at its very center. The ZVM manages replication for the entire domain, keeps track of applications and information in motion in real time.
If the ZVM does not work, replication is disrupted - failover is still possible but new checkpoints cannot be created. The procedures in this KB describe how you can safeguard your ZVM. Select the relevant procedure, depending on your environment
Solution:

Protecting and Recovering Zerto Virtual Manager

To protect and recover your ZVM use one of the following procedures, based on the environment at your site.
These procedures apply only to environments where both the protected platform and the recovery platform are vCenter. It is recommended to use the methods described below to protect the ZVM on both the protected and recovery sites.
Note: You cannot protect ZVM within the same vCenter site.
 

You want to...Use this process when...
Protect and recover a ZVM
machine with a local database running on the ZVM machine
  • The database, which is used by the ZVM, is embedded as part of the ZVM application.

- Or -

  • The database, which is used by the ZVM, is external to the ZVM application, but installed on the same VM as the ZVM.
Protect and recover a ZVM
machine with an external
ZVM database
  • The database, which is used by the ZVM runs on a separate VM to the ZVM machine.

- And -

  • The ZVM database is the only database on the SQL database.
  • The database, which is used by the ZVM runs on a separate VM to the ZVM machine.

- And -

  • The ZVM database is one of many databases on the SQL database.

Zerto does not recommend this option for protecting the ZVM, because if any changes were made in the site environment after exporting the ZVM, this may cause unexpected results when you need to recover the ZVM.

  • Other methods for protecting and recovering the ZVM are possible but they have not been tested at Zerto.
  • Attempting to backup and restore the ZVM using methods not recommended by Zerto may have hazardous effects. These may include having to delete and recreate the VPG, or in an extreme cases, to reinstall the ZVM.

 

Protecting and Recovering a ZVM Machine with a Local Database

Use these procedures to protect and recover a ZVM machine, when the ZVM database is either embedded as part of the ZVM application, or it is external to the ZVM application, but installed on the same VM as the ZVM.
In this procedure, you protect and recover both the ZVM machine and the ZVM database.


Before You Begin:
Verify one of the following conditions are met at your site:

  • The ZVM database is embedded as part of the ZVM application
    • Or -
  • The ZVM database is external to the ZVM application, but installed on the same VM as the ZVM.

 

To Protect a ZVM Machine with a Local Database:

  1. On the protected site, create a VPG of the protected ZVM VM. In this procedure we will refer to it as VPG_ZVM.
    • There must be no other VMs in this VPG.
  2. On the VPG_ZVM, which you created in Step 1 configure:
  3. FAILOVER TEST network with an IP address which is different to the production IP address.
  4. FAILOVER LIVE network with the same network and IP as the production network and IP address.
  5. Verify that initial sync, which is the result of the VPG creation, is complete.
  6. Best Practice: After configuring the VPG, perform a FAILOVER TEST on the VPG_ZVM. When FAILOVER TEST is successful, the original production ZVM VM is protected.
  7. Stop the FAILOVER TEST.

In the event of a disaster, continue with the next procedure “To Recover the ZVM VM with a Local Database:”, below.
 

To Recover the ZVM VM with a Local Database:

  1. On the protected site, power down the ZVM VM. As a result, the following occurs:
    • An alert on the recovery site appears: The Zerto Virtual Manager is not connected to site: [name of site where ZVM crashed](IP:port)
    • The VPGs which are protected to the site where the ZVM crashed will be in the state: site disconnection
    • The VPGs which are protected to the recovery site will be in the state: recovery is possible. This includes VPG_ZVM.
  2. On the recovery site, perform a FAILOVER LIVE on the VPG_ZVM. It is highly recommended to use the latest checkpoint.
    • Note that reverse protection in the Failover wizard will be disabled.
    • After failover, the VPG_ZVM that was failed over will be in the state need configuration.
  3. In the Dashboard > Events or Task area, verify that Failover is committed.
  4. Rename the original protected ZVM VM.
  5. Verify that the recovered ZVM VM is up and running, and that the ZVM VM is configured with the same IP as the original protected ZVM VM.
  6. On the recovery site, power off the failed over ZVM VM.
  7. From the recovery site, export the failed over ZVM VM to an OVF/OVA file.
  8. Copy the exported OVF/OVA file from the recovery site to the protected site.
  9. On the protected site, import the OVF/OVA file, which was copied over in Step 8.
  10. On the protected site, power on the ZVM VM which was imported in Step 9.

All errors and alerts which will have occurred up to this point are resolved, apart from VPG_ZVM, which will still be in the state need configuration.

  1. On the recovery site delete the powered off ZVM VM, which was failed over in Step 2. Deleting it triggers the deletion of its VPG, and the error goes away.
  1. Delete the ZVM VM that was renamed in Step 4.
  2. You must now protect this new ZVM.

Note: After protecting and recovering ZVM, there might be environment changes that could cause inconsistencies in the ZVM recovery process. For descriptions and recommended actions, see “Troubleshooting Protecting and Recovering Zerto Virtual Manager”, on page 8.

Note:
If the ZVM is running Zerto 5.5 and above, freshly installed (not upgraded), the DB file location is:

%ProgramData%\Zerto\Data\zvm_db.mdf

If the ZVM is running an older version than 5.5, or it was upgraded to 5.5, the DB file location is:

%ProgramFiles%\Zerto\Zerto Virtual Replication\zvm.sdf

Protecting and Recovering a ZVM Machine with an External Exclusive SQL Database

Use these procedures to protect and recover a ZVM machine, when the ZVM database is external to the ZVM machine, and is the only database on an SQL database.
In this procedure, you protect and recover both the ZVM machine and the ZVM database.
Before You Begin:
Verify both the following conditions are met at your site:

  • The ZVM database runs on a separate VM to the ZVM machine
    • And -
  • The ZVM database is the only database on the SQL database

 

To Protect a ZVM Machine with an External Exclusive SQL Database:

  1. On the protected site, create a VPG of the protected ZVM VM and SQL VM. In this procedure, we will refer to it as VPG_ZVM_SQL.
    • Only the ZVM VM and the SQL VM must be in this VPG.
  2. On VPG_ZVM_SQL, which you created in Step 1, configure FAILOVER TEST network separately, for the protected ZVM VM and separately for the protected SQL VM, with an IP address which is different to the production IP address.
  3. On VPG_ZVM_SQL which you created in Step 1, configure FAILOVER LIVE network separately, for the protected ZVM VM and separately for the protected SQL VM with the same network and IP as the production network and IP address.
  4. Verify that initial sync, which is the result of the VPG creation, is complete.
  5. Best Practice: After configuring the VPG, perform a FAILOVER TEST on the VPG_ZVM_SQL.

When FAILOVER TEST is successful, the original production ZVM VM, and the SQL VM are protected.

  1. Stop the FAILOVER TEST.

In the event of a disaster, continue with the next procedure “To Recover a ZVM Machine with an External Exclusive SQL Database:”, below.
 

To Recover a ZVM Machine with an External Exclusive SQL Database:

  1. On the protected site, power down the ZVM VM and SQL VM. As a result, the following occurs:
    • An alert on the recovery site appears: The Zerto Virtual Manager is not connected to site: [name of site where ZVM crashed](IP:port)
    • The VPGs which are protected to the site where the ZVM crashed will be in the state: site disconnection
    • The VPGs which are protected to the recovery site will be in the state: recovery is possible. This includes VPG_ZVM_SQL.
  2. On the recovery, site perform a Failover LIVE on the VPG_ZVM_SQL. It is highly recommended to use the latest checkpoint.
    • Note that reverse protection in the Failover wizard will be disabled.
    • After failover, the VPG_ZVM_SQL that was failed over will be in the state need configuration.
  3. In the Dashboard, Events or Task area, verify that Failover is committed.
  4. Rename the original protected ZVM VM and SQL VM.
  5. Verify that the recovered ZVM VM and SQL VM are up and running, and that the ZVM VM and SQL VM are configured with the same IP as the original protected VMs.
  6. On the recovery site, power off the failed over ZVM VM and the SQL VM.
  7. Then export the failed over:
    • ZVM VM to an OVF/OVA file.
    • SQL VM to an OVF/OVA file.
  8. Copy the exported OVF/OVA files from the recovery site to the protected site.
  9. On the protected site, import both the OVF/OVA files which were exported in Step 7.
  10. On the protected site, power on the ZVM VM, and the SQL VM, which were imported in Step 9.

 

All errors and alerts which will have occurred up to this point are resolved, apart from VPG_ZVM_SQL, which will still be in the state need configuration.
  1. On the recovery site, delete the powered off ZVM VM, and the SQL VM which were failed over in Step 2. Deleting them triggers the deletion of their VPG, and the error goes away.
  2. Delete the ZVM VM and the SQL VM which were renamed in Step 4.
  3. You must now protect the new ZVM VM and SQL VM. To do this, follow the steps in the procedure above, “To Protect a ZVM Machine with an External Exclusive SQL Database:”, on page 4.

Note: After protecting and recovering ZVM, there might be environment changes that could cause inconsistencies in the ZVM recovery process.

 

Protecting and Recovering a ZVM Machine with an External Shared SQL Database

Use these procedures to protect and recover a ZVM machine, when the ZVM database is external, and is one of many databases on the SQL database.
In this procedure, you will protect only the ZVM database. When recovering, you will select the same checkpoint in the SQL VM to which to you Failed over LIVE the ZVM VM.
Before You Begin:
Verify both the following conditions are met at your site:

  • The ZVM database runs on a separate virtual machine to the ZVM machine
    • And -
  • The ZVM database is one of many databases on the SQL Server

Note: The following procedures require using Zerto jFLR (Journal File Level Restore), which is supported from ZVM v4.5 and above. For details, see Zerto Virtual Replication Installation Guide for VMware vSphere.
 

To Protect a ZVM Machine with an External Shared SQL Database:

  1. On the protected site, create a VPG of the protected ZVM VM. In this procedure we will refer to it as VPG_ZVM.
    • There must be no other VMs in this VPG.
  2. On the protected site, create a VPG of the protected SQL VM. In this procedure we will refer to it as VPG_SQL.
    • There must be no other VMs in this VPG.
  3. On VPG_ZVM, which you created in Step 1, configure FAILOVER TEST network with an IP address which is different to the production IP address.
  4. On VPG_ZVM, which you created in Step 1, configure FAILOVER LIVE network with the same network and IP as the

production network and IP address.

  1. Verify that initial sync, which is the result of the creation of both VPGs, is complete.
  2. Best Practice: After configuring the VPG, perform a FAILOVER TEST on the VPG_ZVM. When FAILOVER TEST is successful, the original production ZVM VM is protected.
  3. Stop the FAILOVER TEST.

In the event of a disaster, continue with the next procedure “To Recover a ZVM Machine with an External Shared SQL Database:”, below.

 

To Recover a ZVM Machine with an External Shared SQL Database:

  1. On the protected site, power down the ZVM. As a result, the following occurs:
    • An alert on the recovery site appears: The Zerto Virtual Manager is not connected to site: [name of site where ZVM crashed](IP:port)
    • The VPGs which are protected to the site where the ZVM crashed will be in the state: site disconnection
    • The VPGs which are protected to the recovery site will be in the state: recovery is possible. This includes VPG_ZVM and VPG_SQL.
  2. On the recovery site, perform a Failover LIVE on the VPG_ZVM.
    • Note that reverse protection in the Failover wizard will be disabled.
    • After failover, the VPG_ZVM that was failed over will be in the state need configuration.
  3. Make a note on which checkpoint the Failover LIVE is performed. We highly recommend you use the latest checkpoint.
  4. Rename the original protected ZVM VM.
  5. On the recovery site, using the Zerto jFLR (Journal File Level Restore), restore the database files from the VPG_SQL. To do this:

 

  1. In the main ZVM page at the bottom, click Actions, then select Restore File. The File and Folder Restore wizard appears.
  2. In the VM window, select the SQL VM, which is protected by the VPG_SQL, then click Next.
  3. IMPORTANT! In the Checkpoint window, select the same checkpoint which was noted in Step 3, to which to you will Failover LIVE the ZVM VM, then click Next.
  4. In the Disk window, select the disk, then click Next.
  5. In the Mount window, click Start Mount.
  6. In the Tasks area, click the folder icon  to browse, then select the database files zvm_db.mdf and zvm_db.ldf from the SQL VM database folder where the ZVM database is located, then click Start Download.

Note: zvm_db.mdf and zvm_db.ldf are default database file names. These might have been changed.

  1. On the recovery site, power off the failed over ZVM VM.
  2. On the recovery site, export the failed over ZVM VM to an OVF/OVA.
  3. Copy the exported OVF/OVA file from the recovery site to the protected site.
  4. On the protected site, import the OVF/OVA, which was copied in Step 8.
  5. On the recovery site, verify that the Restore File operation is complete, and that both database files, zvm_db.mdf and

zvm_db.ldf, were downloaded.

  1. In the Tasks area, click Unmount disk, of the disk which you Mounted in Step 5.
  2. Copy the database files zvm_db.mdf and zvm_db.ldf from the recovery site to the protected site.
  3. On the protected site, power down the SQL server service, and overwrite both ZVM database files (zvm_db.mdf and zvm_db.ldf) with the .MDF and .LDF files which you copied to the protected site in Step 12.
  4. Then, power on the SQL server service.
  5. On the protected site, power on the ZVM VM, which was imported in Step 9.

At this point an error appears, informing the user that the VPG needs configuration.
All errors and alerts which will have occurred up to this point are resolved, apart from VPG_ZVM, which will still be in the state need configuration.

  1. On the recovery site, delete the powered off ZVM VM which was failed over in Step 2. Deleting it triggers the deletion of its VPG, and the error goes away.
  2. Delete the ZVM VM that was renamed in Step 3.
  3. You must now protect this new ZVM. To do this, follow the steps in the procedure above,“To Protect a ZVM Machine with an External Shared SQL Database:”, on page 6.

Note: After protecting and recovering ZVM, there might be environment changes that could cause inconsistencies in the ZVM recovery process, please refer to the next chapter.
 

Troubleshooting Protecting and Recovering Zerto Virtual Manager

Protecting and recovering the ZVM involves protecting the ZVM in a VPG, then, in the event of a disaster, recovering the ZVM by performing a failover.

Since the ZVM monitors the site, any change that occurs in the site after failing over to an early checkpoint, is not recognized by the recovered ZVM and may prevent a smooth recovery.
Therefore, the recommendation is to fail over the ZVM to the latest checkpoint available.

Following are a list of environment changes that may cause inconsistencies in the ZVM recovery process, if the ZVM is recovered to a point in time before these changes occurred.

Post recovery, the user must review the environment and check whether any of the problems listed below have occurred. In case of any problem, the VPG/s affected by this inconsistency needs to be deleted and re-created.

User-added image