Skip to main content


 

 

druva_logo.pngDruva
|
Documentation

How can we help you?

Trending articles:

 

Druva Documentation

VM backup fails with the VMWARE VDDK65535 error (memory allocation error)

  1. Last updated
  2. Save as PDF
Heads up!

We've transitioned to a new documentation portal to serve you better. Access the latest content by clicking here.

This article applies to:

  • OS: ESX 5.0 and later
  • Product edition: Phoenix

Problem description

VM backup fails with the VMWARE VDDK65535 error (memory allocation error)

Traceback

Phoenix.log :
[2022-03-13 22:15:18,232] [ERROR] DataManager: Error in reading data from the Disk [6000c292-8b6c-4ec5-a71d-fa4c002829ac] at start :0. vix_error ==== 2[Memory allocation failed. Out of memory.]
[2022-03-13 22:15:18,234] [ERROR] Error in reading from the Disk. Error Code: 2. Retrying in 30 Seconds. Retry attempt remaining 2
[2022-03-13 22:15:18,235] [ERROR] 140153062401792 Backup of VMDK Failed. Fault: VMware VDDK: Internal Error (#10005ffff : 2) (Error Code : VMWARE_VDDK65535)

VDDK logs :
2022-03-13T22:14:18.193Z| vthread-12| E110: VixDiskLib: Detected DiskLib error 802 (NBD_ERR_INSUFFICIENT_RESOURCES).
2022-03-13T22:14:18.193Z| vthread-12| E110: VixDiskLib: VixDiskLib_Read: Read 8192 sectors at 0 failed. Error 2 (Memory allocation failed. Out of memory.) (DiskLib error 802: NBD_ERR_INSUFFICIENT_RESOURCES) at 5240

Cause

  • This issue is related to a VMWare decision to limit the NBD protocol to a maximum 32MB read buffer by default on ESXi hosts.
  • There is a limitation with ESX/VC host on NFC session (NBD/NBDSSL Transport) https://pubs.vmware.com/vsphere-50/i...truct.5.5.html 
  • NFC session has a limitation of 32 MB buffer allocation. If it exceeds 32 MB, NFC session fails.
  • The NFS buffer is calculated by the following formula.
  • NFS Buffer Size = Number of VMDKS x Block Size

Example:

In a Phoenix Backup Proxy, if you have set a block size of 4 MB and VM has 11 VMDKs connected using NBDSSL, the NFS buffer size will be 44 MB, which exceeds the limit. 

Note: This issue will occur when a substantial number of backups use the NBD or NBDSSL protocol for backup.

Considerations

  • Since this limitation is not valid for HOTADD transport mode, ensure that the HOTADD options work to avoid this error. 
  • Deploy a proxy in the same cluster that contains vCenter (target VM) to back up the VM.
  • If a backup is not possible via HOTADD, adjust the block size to fix the issue.

Resolution

  1. Log in to the VMWARE Backup proxy with default username root and password.
  2. Ensure there are no jobs running.
  3. Stop the Phoenix Agent service on the backup proxy using the command:
    /etc/init.d/Phoenix stop 
  4. Verify if any Phoenix processes are still running.
    ps -ef | grep -i Phoenix 
  5. If any process is still found running, kill the process with the following command:
    Kill -9 <process id>
  6. Goto to the below path 
    cd /etc/Phoenix/VMWARE 
  7. Edit Phoenix.cfg file using vi editor using below command:
    vi Phoenix.cfg 
    Type ‘ i ‘ to activate insert mode 
    Add below line
    _VM_BACKUP_BLOCK_SIZE = 1048576
    Type esc key followed by :wq to save and exit file.
    *This sets the block size to 1 MB
  8. Verify if above lines has added using below command
    cat Phoenix.cfg
  9. Start the Phoenix Agent service on the backup proxy as follows:
    /etc/init.d/Phoenix start
  10. Trigger the backup or restore