Skip to main content

 

Druva Documentation

Troubleshoot VMware issues

Phoenix Editions: File:/tick.png Business File:/cross.png Enterprise File:/tick.png Elite

This topic describes common workarounds of issues that you might encounter while backing up or restoring virtual machines. 

Backup fails if there is insufficient space on the data store.

The backup fails when taking a snapshot of a virtual machine and there is insufficient storage space on the data store.

The backup fails when you have attempted to take a snapshot and saved it to a VMware VMFS partition that does not have enough free space to save the snapshot. The log file displays the following error:

An error occurred while saving the snapshot: Unable to open the snapshot file.

Resolution

  • Ensure sufficient free space exist on the data store.
  • Delete unused snapshots and unused virtual disks.
  • Expand the volume by adding extents.
Backup fails because backup proxy cannot take snapshots 

Sometimes, a backup fails because backup proxy cannot take a snapshot of virtual machines that are configured for backup. You might see an error like this in your log file: 

An error occurred while saving the snapshot: Change tracking target file already exists

The log files are created at /var/log/Phoenix

Resolution

  1. Consolidate the snapshots. For more information, see this article in the VMware Knowledge Base.  
  2. Wait for the next scheduled backup, and check the backup status after the backup completes.
    -or-
  3. Start a backup, and check the backup status after the backup completes.
Backup fails because the Small Computer System Interface (SCSI) Controller is in bus sharing mode

A backup fails if the Small Computer System Interface (SCSI) Controller is in sharing mode. This is because VMware does not support snapshots of virtual machines that are in bus-sharing mode. 

The log file displays the following error: 

Virtual machine is configured to use a device that prevents the snapshot operation: Device '' is a SCSI controller engaged in bus-sharing

The log files are created at /var/log/Phoenix

Resolution

Ensure that the SCSI mode is not set to bus-sharing. 

Backup fails if you enabled Changed Block Tracking for virtual machines

Backups might fail if you enabled Changed Block Tracking (CBT) for virtual machines. 

The log files are created at /var/log/Phoenix

Resolution

1. Login to the backup proxy.

2. Disable the CBT of virtual machine using vmcontrol utility.

3. Enable the CBT of virtual machine using vmcontrol utility.

Note: This utility deletes all the snapshots in the virtual machine.

Backup fails if restore of a snapshot is in progress

A backup which starts after the virtual machine has been reverted to some snapshot from the vSphere Client, fails with the following message:

Error caused by file /vmfs/volumes/55708983-391f41a1-eaa8-00224db1d33b/Rhel6.4-MD-SDS-cbtEnable-MixProvision/Rhel6.4-MD-SDS-cbtEnable-MixProvision.vmdk', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [], file = '/vmfs/volumes/55708983-391f41a1-eaa8-00224db1d33b/Rhel6.4-MD-SDS-cbtEnable-MixProvision/Rhel6.4-MD-SDS-cbtEnable-MixProvision.vmdk' }

The log files are created at /var/log/Phoenix

Resolution

You must disable and enable CBT on the associated virtual machines to resolve this issue. 

1. Login to the backup proxy.

2. Disable the CBT of virtual machine using vmcontrol utility.

3. Enable the CBT of virtual machine using vmcontrol utility.

Note: This utility deletes all the snapshots in the virtual machine.

Backup Fails on a Standalone ESX

Backup fails with the following error message:

[2015-07-29 17:21:59,807] [ERROR] Error <class 'agents.vmware.vixDiskLibHelper.vddk_lib.VDDKException'>:Vddk Error [code = 5643587026947, msg = None]. Traceback -Traceback (most recent call last):

File "agents/vmware/vixDiskLibOCManager.py", line 118, in handle_oc_work

File "agents/vmware/vixDiskLibHelper/vddk_lib.py", line 111, in open_disk

File "agents/vmware/vixDiskLibHelper/vddk_lib.py", line 197, in __raise_vddk_error

VDDKException: Vddk Error [code = 5643587026947, msg = None]

Resolution

The password contains ‘@’. Change the password and retry backup. The backup should work fine after this.

Backup fails due to timeouts

Backup fails with the following error message:

[2015-07-20 18:40:20,487] [ERROR] Error <class 'agents.vmware.vixDiskLibHelper.vddk_lib.VDDKException'>:Vddk Error [code = 10041633552057, msg = None]. Traceback -Traceback (most recent call last):

File "agents/vmware/vixDiskLibOCManager.py", line 118, in handle_oc_work

File "agents/vmware/vixDiskLibHelper/vddk_lib.py", line 111, in open_disk

File "agents/vmware/vixDiskLibHelper/vddk_lib.py", line 197, in __raise_vddk_error

VDDKException: Vddk Error [code = 10041633552057, msg = None]

[2015-07-20 18:40:20,487] [ERROR] Failed to open the Disk [local SAS] Win7 VCenter/Win7 VCenter.vmdk

[2015-07-20 18:40:20,487] [ERROR] [VixThread (id=0, pool=vmdk, parent_id=None)] Backup of VMDK Failed. Fault: Vddk Error [code = 10041633552057, msg = None]

[2015-07-20 18:40:20,487] [INFO] [VixThread (id=0, pool=vmdk, parent_id=None)] In finally to close the connection

[2015-07-20 18:40:20,488] [INFO] [VixThread (id=0, pool=vmdk, parent_id=None)] Closed the connection successfully

Resolution

On the backup proxy there is a vddk config file: /etc/Phoenix/vddk.cfg. It is empty by default. In this you can configure NBD timeouts.

The exact config can be read at this link.

Restore Fails

[2015-07-23 12:22:38,491] [ERROR] Exception: (vmodl.fault.ManagedObjectNotFound) {

  dynamicType = <unset>,

  dynamicProperty = (vmodl.DynamicProperty) [],

  msg = 'The object has already been deleted or has not been completely created',

  faultCause = <unset>,

  faultMessage = (vmodl.LocalizableMessage) [],

  obj = 'vim.VirtualMachine:vm-6055'}

Resolution

Sometimes the agent is not able to query the vCenter properly. Retry the restore in some time.

Backup Fails

[2015-07-29 19:24:08,367] [ERROR] (vim.fault.ApplicationQuiesceFault) {

  dynamicType = <unset>,

  dynamicProperty = (vmodl.DynamicProperty) [],

  msg = "An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.",

Resolution

Retry the backup.

Backup and restore Fails for virtual machines that have special characters in its names.

Backup and restore will fail with an Internal error if the virtual machine name has the following special characters.

  • Hash  ("#")
  • Left Curly Brace ("{") 
  • Right Curly Brace ("}") 
  • Vertical Bar/Pipe ("|") 
  • Backslash ("\") 
  • Caret ("^") 
  • Tilde ("~") 
  • Left Square Bracket ("[") 
  • Right Square Bracket ("]") 
  • Grave Accent ("`") 
  • Forward slash/Virgule ("/") 
  • Left Bracket ("(") 
  • Right Bracket (")"

Resolution

Do not add the above listed special characters to the virtual machine name. 

HotAdd Troubleshooting

Test to see if the disk of the virtual machine to be backed up can be manually HotAdded to the proxy

Performing the following steps will help determine if the infrastructure is present to allow the proxy to attach the disks of the virtual machine to be processed in HotAdd.

1.  Create a snapshot on the virtual machine to be processed by backup proxy (Source virtual machine to be backed up).

2.  Within a vSphere client, attach the base disk from the virtual machine in Step 1 to the backup proxy.

  1. Edit the Druva backup proxy that will be processing the virtual machine from Step 1.
  2. Click Add.
  3. Choose “Hard Disk”.
  4. Chose “Use an existing virtual disk”.
  5. Navigate to the location of the base disk from the virtual machine in step 1.
  6. Specify that the disk should be attached Independent - Nonpersistent.
  7. Click Finish.

3.  If this task completes this means the infrastructure is present to allow the Proxy to HotAdd the virtual machines base disk.

4.  Detach the disk that was attached to the backup proxy.

  1. Edit the Druva backup proxy from Step 2.
  2. Highlight the attached disk.
  3. Click Remove (DO NOT change the radio option).
  4. Click Finish.

 5. Remove the snapshot created in Step 1.

Removing Stuck VMDK’s from the backup proxy

Sometimes the VMware API call to remove the VMDK is not received or did not complete properly. So the VMDKs are stuck to the backup proxy. In order to remove those disks perform the following steps.

Note: Kindly perform these steps only when there is no backup going on in the backup proxy.

  1. Through vSphere, you should be able to right-click the Druva proxy virtual machine and select edit settings. Look for the Hard disks attached to the virtual machine. Normally the stuck disks will be listed at the bottom. You should also be able to identify the disks by the name and the datastore they are in under the disk file box (top right).

  1. To remove them, select the disk by clicking on it and select remove at the top right. Under removal options choose “remove from virtual machine”. DO NOT choose the option that deletes it from disk (the second option).

Original restore fails
  • Restore fails, if the vRDM disk is converted to independent vRDM or pRDM.

    Resolution

Revert from the independent vRDM disk/pRDM to vRDM disk. Then try original restore.

  • Restore fails, if the new uuid does not match to old uuid

    Resolution

If the uuid of the new created vmdk could match to the old uuid restore will fail. 

Backup fails of virtual machine having vRDM disks

If the same lun is mapped to two different disks, the backup fails.

Resolution

Unmap the lun from both disks and then map to only one of the disks.

Restore fails on a standalone disk

Restore fails on the standalone disk with the following error message:

[2015-08-21 16:15:54,708] [ERROR] (vim.fault.FileAlreadyExists) {
  dynamicType = <unset>,
  dynamicProperty = (vmodl.DynamicProperty) [],
  msg = 'Cannot complete the operation because the file or folder /vmfs/volumes/5575befd-d1e64fa5-f919-00224db1d2b5/Test-rhel-vm-saranya_5/Test-rhel-vm-saranya_5.vmx already exists',
  faultCause = <unset>,
  faultMessage = (vmodl.LocalizableMessage) [],
  file = '/vmfs/volumes/5575befd-d1e64fa5-f919-00224db1d2b5/Test-rhel-vm-saranya_5/Test-rhel-vm-saranya_5.vmx'
}

Resolution

Rename the folder or vmx file that creates the conflict. Restore work fine after this.

Virtual machine with EFI firmware cannot boot after restore

Perform the following steps if a virtual machine is configured with EFI firmware, and cannot boot after restore.  For more information on EFI, see Using EFI/UEFI firmware in a VMware Virtual Machine.

Note: The following steps are applicable if the backup proxy version 4.6.4 or earlier.

Resolution

  1. Log into vSphere. 
  2. Select the virtual machine that is unable to boot. 
  3. Right-click the virtual machine and select PowerPower Off.
  4. On the Confirm Power Off screen, click Yes.
  5. Right-click the virtual machine and select Edit Settings, the Edit Settings window is displayed.
  6. Under the Options tab, select EFI option.
  7. Click OK.
  8. Right-click the virtual machine and select PowerPower On.
    The virtual machine should boot automatically.​​​​​​

If the virtual machine does not boot, perform the following steps:

  1. Right-click the virtual machine and select Edit Settings, the Edit Settings window is displayed.
  2. Under the Options tab, select The next time the virtual machine boots, force entry into the EFI setup screen option.
  3. Click OK.
  4. Right-click the virtual machine and select Power> Reset.
  5. Right-click the virtual machine and select Open Console. The Boot Maintenance Manager is displayed.
  6. Select Enter setup.
  7. Select Boot from file and then on the next screen highlight the boot partition and press Enter.
  8. Select the EFI > <Operating system> <.efi file>.
    The virtual machine will boot automatically.
Virtual machine not visible in Phoenix UI after configuring backup proxy. 

The issue occurs if the descriptor file for the virtual machine gets corrupted. The following error is received in backup proxy logs.

One of the virtual machine's VMDK have none as UUID. Skipping VM:ARX_FR_DC01 from listing.  

Resolution

Contact VMWare to fix the issue.

See Backup software shows error: Disk does not have a UUID (1033370).

Volume mounting failed.

This error occurs when some volumes are not mounted during File Level Restore.

Resolution

  • Ensure that the Partition ID of a volume is consistent with the actual partition type of the volume. If it is not consistent, then the volume will not get mounted.
  • If the error still persists, contact Druva Support.
Backup proxy upgrade has failed.

This error can occur due to the following reasons:

  1. If you have installed a hotfix on your existing Phoenix version, you cannot automatically upgrade to the next version from the Phoenix user interface.
  2. If the server that hosts the backup proxy restarts during the upgrade.
  3. If the old version is not present on client machine then upgrade rollback will fail.

Resolution

Manually upgrade the backup proxy.

Backup proxy is unable to connect to the Phoenix cloud.

This can happen if both network interface cards are enabled, and:

  • You use one network interface card for public network and the other network interface card for private network

  • The network interface card configured for private network uses DHCP

The backup proxy fails to connect to Phoenix cloud because the DHCP changes routing table sequence. 

Resolution

You can use the following command to manually add a static route entry for the public network in the backup proxy settings. 

nmcli connection modify <connection_name> +ipv4.routes "<ip_range>/<subnet_prefix> <gateway>"

For example: 
nmcli connection modify eth0 +ipv4.routes "192.168.122.0/24 10.10.10.1"