After a user upgraded their VRAs (Either via 'Auto Upgrade' or 'Manual' within the ZVM GUI); alerts in the ZVM GUI for peer VRA disconnections and intermittent bitmap syncs are observed.
Note: If no VRAs are noted as being '<deleted>' after the VRA upgrades, then the below troubleshooting steps within this KB should be followed.
During a VRA upgrade, the original OVF for the deployed VRA is replaced with a new OVF template. The settings from the old VRA VM (CPU virtual socket count, Memory configuration, MTU settings) are not automatically carried over to the new VRA VM.
Note: This in turn may cause unforeseen sync performance/VRA peer network connectivity issues.
Prior to upgrading VRAs, please note the following VRA settings within SCVMM or vMware:
1] Amount of configured RAM per VRA
2] Number of VRA CPUs per VRA
3] VRA MTU
Note: In the ZVM GUI click on 'Setup' > 'Show/Hide Columns' > 'VRA RAM' > 'Export' to retrieve the amount of memory configured per VRA.
Note: In either SCVMM or vSphere/Web GUI, right click on the VRA VM > 'Edit Settings' > 'Hardware' > 'CPU', to retrieve the number of virtual sockets configured for the VRA.
Note: Starting in 7.0, to check the VRA MTU this you will need to load the 'ssh.ppk' certificate file in 'C:\Program Files\Zerto\Zerto Virtual Replication\Secrets' into PuTTY to securely SSH to the VRA as 'root'. If the version you upgraded to is less than 7.0, please contact support for assistance.
Securely Log into VRAs via SSH:
3.1] Once you have located the 'ssh.ppk' file in the 'C:\Program Files\Zerto\Zerto Virtual Replication\Secrets' folder, open PuTTY (which also exists in the ZVR installation folder) and navigate to the ‘Auth’ menu (Connection -> SSH -> Auth).
3.2] Click on the ‘Browse’ button and load the 'ssh.ppk' key file into PuTTY.
3.3] Navigate to the ‘Session’ menu, enter the VRA’s IP address in the ‘Host Name (or IP address)’ bar, and click on ‘Open’.
3.4] Click 'Yes' on the host’s SSH fingerprint alert.
3.5] Use ‘root’ as the username and press ‘Enter’. You should now be logged in to the VRA as root.
Check VRA MTU (packet fragmentation) between peer VRAs:
3.1] Log into a peer VRA via PuTTy SSH. You may need to follow the previous steps noted in the steps noted in 'Securely Log into VRAs via SSH'.
3.2] Note that the default MTU value for VRAs is 1492. Use the following Linux 'ping' command to determine the proper MTU size with ICMP pings.
ping [-M do] [-s <packet size>] [host]
-M <hint>: Select Path MTU Discovery strategy. <hint> may be either "do" (prohibit fragmentation, even local one), "want" (do PMTU discovery, fragment locally when packet size is large), or "dont" (do not set DF flag).
-s packetsize: Specifies the number of data bytes to be sent. The default is 56, which translates into 64 ICMP data bytes when combined with the 8 bytes of ICMP header data.
Note: [host] is the IP address of the peer VRA.
An example MTU test ICMP ping would resemble the following:
ping -M do -s 1472 192.168.1.15
If you see the message "Packet needs to be fragmented but DF set”, you will need to lower the packet size.
3.3] First, try a lower packet size (for example 1412) until you get a successful ping response instead of the "Packet needs to be fragmented but DF set.” message. Keep changing the packet size until you have found the highest packet size that can be sent in your network without being fragmented.
Add 28 more to this (since you specified ping packet size, not including the IP/ICMP header of 28 bytes) and this is your Maximum MTU. For this example, the maximum packet size is 1412 + 28 = 1440.
3.4] Repeat steps 3.1] thru 3.3] for all peer VRAs to re-establish peer VRA network connectivity.
3.5] In either vSphere, or SCVMM, power off one VRA at a time and change the CPU/Memory configuration before powering on the VM and moving onto the next VRA VM.
Note: In vMware, the number of CPUs is 'virtual socket's and not 'core's'.