Skip to main content

How can we help you?

Druva Documentation

Security updates impacting VMware and Hyper-V workloads

Enterprise Workloads Editions: File:/tick.png Business File:/cross.png Enterprise File:/tick.png Elite

 

Overview

Starting May 17, 2021, Druva has changed the cryptographic library for VMware and Hyper-V workloads to remediate vulnerabilities as part of Druva's Patch Management Program. To equip yourself for this change, we need you to upgrade your VMware backup proxy, {{phoenixagent}}s on the Hyper-V hosts, and Hyper-V FLR proxy from version 4.7.x to version 4.9.4 before May 17, 2021.

You can manually download agent version 4.9.4 for VMware backup proxy, Hyper-V hosts, and Hyper-V FLR proxy from the Downloads page.

To upgrade the agents, follow the instructions in the following articles:

Upgrade VMware backup proxy

Upgrade the agents on Hyper-V hosts

Upgrade Hyper-V FLR proxy

Impacted environments

Workload Configuration
VMware Standalone ESXi Servers and vCenter Servers
Hyper-V SCVMM, Hyper-V cluster, Standalone Hyper-V host Server

All VMware backup proxy servers, Hybrid Workloads agents on the Hyper-V hosts, and Hyper-V FLR proxy servers on versions 4.7.x to 4.8.0( excluding 4.8.0) use an older cryptographic library to securely store the credentials used while protecting VMware and Hyper-V backups. The older cryptographic library is now being replaced with a newer one to enhance security further. 

Proxy server versions 4.8.0 to 4.9.4 have a combination of the old and new cryptographic libraries. Druva can use the older cryptographic library to decrypt the existing credentials and encrypt them with the newer cryptographic library. 

Proxy client version 5.0.0 that goes live on 17 May 2021 will only have the newer cryptographic library and will not be able to decrypt the existing credentials.

Impact if proxies are not upgraded by 17 May 2021

If you are unable to run upgrades before 17 May 2021,  you may need to perform the operations outlined below, to get backups and restores working again.

Note: We recommend letting any ongoing backup or restore jobs finish before initiating the VMware proxy, Hybrid Workloads agent on Hyper-V hosts, or the Hyper-V FLR proxy upgrade.

Impact on VMware Servers

Upon an auto-upgrade from version 4.7.x to version 5.0.0, your VMware proxies will automatically get disconnected if the affected proxy was the only proxy in the backup proxy pool. The backup jobs will go into a queued state and any new restore jobs will not be initiated. If the proxy pool has more than one proxy, and one of those proxies is still connected, your backups and restores will continue seamlessly. 

Before you initiate an upgrade, you need to confirm if you want to proceed with the upgrade if jobs are in progress.

Impact on ongoing jobs 
  • If the VMware backup proxy is upgraded while scheduled backups or ongoing restores are in progress, the jobs will go into a queued state. These jobs will resume once the proxy upgrade is complete.
  • Scheduled backups and ongoing restores for Windows VMs on VMware may fail.
Impact on Backup Now jobs (manual backups)
  • Backup Now jobs will fail
  • If you've initiated a VMware backup proxy upgrade while a Backup Now job is in progress, the job will fail with the PHOENIX189 error.

Impact on Hyper-V Servers

Upon an auto-upgrade of your Hyper-V proxy from v4.7.x to 5.0.0 , the Hyper-V backups and restores will start failing with HYPERV3 error unless the steps under Manual Steps to reconnect the backup proxy section are performed.

The following  manual, scheduled, and ongoing jobs are impacted:

  • Backup now jobs (manual backups) may fail.
  • If you initiated a Hybrid Workloads agent on Hyper-V host or FLR proxy upgrade when Backup Now or FLR restores were in progress, the Backup Now or FLR restores will fail with PHOENIX189.
  • Scheduled backup jobs and ongoing restores for Hyper-V may fail.

Action required 

You are required to upgrade the backup proxy servers from version 4.7.x to 4.9.4 before 17 May 2021. 

If the proxy servers are directly upgraded from version 4.7.x to version 5.0.0, Druva will not be able to decrypt the credentials using the newer cryptographic library. This can cause the backups and restores to fail until a few manual operations are performed.

Manual steps to reconnect the backup proxy

If the backup proxies automatically upgrade from 4.7.x to version 5.0.0 and get disconnected, here are the steps to reconnect the proxy:

Note: Proxy server versions 4.8.0 to 4.9.4 have a combination of the old and new cryptographic libraries. Druva can use the older cryptographic library to decrypt the existing credentials and encrypt them with the newer cryptographic library. If you directly upgrade from 4.8.0 or later to 5.0.0, you don't have to perform the following steps.

VMWare 

  1. Login to the host machine (Standalone ESXi servers or the vCenter Servers) using an account with administrative privileges. 
  2. Issue the following commands in sequence:

NOTE: Enter the password in single quotes. For example, if your password is P@ssw0rd, in the command above, enter ‘P@ssw0rd’.

vCenter credentials set successfully

[root@backup-proxy-21-12-1-1 ~]# service Phoenix status

Druva is running

[root@backup-proxy-21-12-1-1 ~]# service Phoenix restart

Restarting Druva (via systemctl):                        [  OK  ]

[root@backup-proxy-21-12-1-1 ~]# service Phoenix status

Druva is running

[root@backup-proxy-21-12-1-1 ~]# rpm -qa | grep druva

druva-phoenix-vmware-client-5.0.0-113368.x86_64

Ensure that the Hybrid Workloads service is up and running. The backup proxy will show connected again. For more information on the commands, see Use the vCenterDetail utility.

Hyper-V

  1. Login to the host machine (SCVMM, Hyper-V cluster, Standalone Hyper-V host Server) using an account with administrative privileges. 
  2. Issue the following command:
  3. PhoenixHyperVControl.exe set_credential -e (scvmm/cluster) -f scvmm_fqdn (only for scvmm) -u user -p passwd 
    Note: Execute this command from the host with elevated permissions
  4. Restart the Hybrid Workloads Agent Client Service from the Services console.

Ensure that the Hybrid Workloads service is up and running. The backup proxy will show connected again.

Known issues

All of these are legacy issues that existed in the product before these security changes went live.

Known issue Description
VMware
PHN-38500

The VMware backup proxy for vCenter was upgraded while backups for some VMs were in progress. The proxy upgrade and the backup jobs were both successful. Post the backup proxy upgrade, the backup of the same VMs fails with the INTERNAL65535 error.

Workaround:
None
PHN-38516

The VMware backup proxy for standalone ESXi was upgraded. The proxy upgrade was successful. Post the backup proxy upgrade, the backup of the VMs fails with the INTERNAL65535 error however, these backups were successful before the proxy upgrade.

Workaround:
None
PHN-38426

The VMware backup proxy for vCenter upgrade is initiated while the restores for some VMs were in progress. The VM gets into an inconsistent state, and the restore jobs fail. Post the backup proxy upgrade, the backup of the restored VMs fails.

Workaround:
None
HyperV
PHN-17714

If you upgrade the Hybrid Workloads agent on Hyper-V host and the Hybrid Workloads agent restarts while a restore to the original location is in progress, the original source VM is lost. Druva deletes the source VM and then replaces it with the restored VM. The backups (full and incremental) of the source VM are lost too. 

Workaround

Post the proxy upgrade, perform a restore to alternate location and take backups of the restored VM.

PHN-38846

Hybrid Workloads agent upgrades on standalone Hyper-V host from version 4.8.x to 5.0.0 may intermittently fail if jobs are in progress during the proxy upgrade. 

Workaround

Retry the Hybrid Workloads agent upgrade when no backup or restore jobs are in progress.

PHN-38847

If Hybrid Workloads agents on Hyper-V hosts are upgraded while restore jobs are in progress, the restores may fail with the INTERNAL 65535 error.

Workaround

Initiate restores for the VMs once the proxy upgrade is successful

 

  • Was this article helpful?