Skip to main content
Druva Documentation

VMware backups in Phoenix fail with Error in reading data from the Disk

 

Problem description

VMware backups fail with error 65535. Phoenix logs have the following error: DataManager: Error in reading data from the Disk: vix_error ==== 3[One of the parameters was invalid.]

Cause

A bad block on the underlying storage or a corrupt file or VMDK causes this error.

Traceback

Phoenix logs

Log locationPhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\Phoenix.<timestamp>

[2020-03-09 22:29:32,330] [ERROR] Error in reading from the Disk. Error Code: 7. Retrying in 30 Seconds. Retry attempt remaining 1

[2020-03-09 22:30:02,355] [ERROR] DataManager: Error in reading data from the Disk [6000C296-2f0e-7d9e-a7f2-463508d1958a] at start :14927. vix_error ==== 3[One of the parameters was invalid.]

[2020-03-09 22:30:02,356] [ERROR] 140412773705472 Backup of VMDK Failed. Fault: VMware VDDK: Internal Error (#10005ffff : 3) (Error Code : VMWARE_VDDK65535)

[2020-03-09 22:30:02,356] [ERROR] Error <class 'inSyncLib.inSyncError.SyncError'>:VMware VDDK: Internal Error (#10005ffff : 3) (Error Code : VMWARE_VDDK65535). Traceback -Traceback (most recent call last):

File "agents/vmware/dataManager.py", line 513, in perform_backup_for_vmdk

SyncError: VMware VDDK: Internal Error (#10005ffff : 3) (Error Code : VMWARE_VDDK65535)

[2020-03-09 22:30:02,357] [INFO] All VMDKs Done enqueing Upload

[2020-03-09 22:30:02,357] [ERROR] DataManager: Error while uploading data, Exiting data manager

VDDK logs

Log location: PhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\VDDK.ZIP

2020-03-09T22:29:02.315-07:00| vthread-6| E110: VixDiskLib: VixDiskLib_Read: Read 6144 sectors at 30570496 failed. Error 7 (A file access error occurred on the host or guest operating system) (DiskLib error 327689: Input/output error) at 5240.
2020-03-09T22:29:32.330-07:00| vthread-6| I125: FileIOErrno2Result: Unexpected errno=5, Input/output error
2020-03-09T22:29:32.330-07:00| vthread-6| I125: DISKLIB-LIB : RWv failed ioId: #1326 (327689) (9) .
2020-03-09T22:29:32.330-07:00| vthread-6| E110: VixDiskLib: Detected DiskLib error 327689 (Input/output error).
2020-03-09T22:29:32.330-07:00| vthread-6| I125: Vix_TranslateErrno: errno = 5
2020-03-09T22:29:32.330-07:00| vthread-6| I125: Foundry operation failed with system error: Input/output error (5)

Resolution

Use vmkfstools to fix a corrupt VMDK

  1. Log in to the ESXi server, which hosts the VM using PuTTy. Locate the volume where the VM folder stores the VMDK : /vmfs/volumes/<DatastoreName>/<VM Name>/
  2. Run the following command to determine if the VMDK is corrupt
    vmkfstools --fix check <diskname.vmdk>
  3. If the disk needs repairing, issue the following command
    vmkfstools --fix repair <diskname.vmdk>

Use the CHKDSK utility to fix bad blocks.

Run the command prompt as Administrator and use the command CHKDSK D: /R – to check, fix, and look for bad sectors on the specific drive.

  • If the VMDK isn't corrupt and the CHKDSK utility doesn't resolve the issue, perform a storage VMotion and retry backups.  A Storage VMotion moves the VMDK to an alternate datastore. Subsequent backup of the VM should not fail because of back blocks with the previous datastore.

Note: A storage VMotion changes the UUID of the VM. Retain the original  UUID to avoid errors arising out of a changed UUID.

  • Restarting the ESX host services like HOSTD and VPXA may also fix issues in rare cases. Use the following commands:

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

  • If the issue persists, create a new VMDK from an older one using vmkfstoos. Use the following command:
    vmkfstools -i disk-001.vmdk disk-new-001.vmdk -d thin
    -i for clone disk
    -d for disk type [thin/thick]