VMware backups in Phoenix fail with Error in reading data from the Disk
Problem description
VMware backups fail with error 65535. Phoenix logs have the following error: DataManager: Error in reading data from the Disk: vix_error ==== 3[One of the parameters was invalid.]
Cause
A bad block on the underlying storage or a corrupt file or VMDK causes this error.
Traceback
Phoenix logs
Log location: PhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\Phoenix.<timestamp>
[2020-03-09 22:29:32,330] [ERROR] Error in reading from the Disk. Error Code: 7. Retrying in 30 Seconds. Retry attempt remaining 1
[2020-03-09 22:30:02,355] [ERROR] DataManager: Error in reading data from the Disk [6000C296-2f0e-7d9e-a7f2-463508d1958a] at start :14927. vix_error ==== 3[One of the parameters was invalid.]
[2020-03-09 22:30:02,356] [ERROR] 140412773705472 Backup of VMDK Failed. Fault: VMware VDDK: Internal Error (#10005ffff : 3) (Error Code : VMWARE_VDDK65535)
[2020-03-09 22:30:02,356] [ERROR] Error <class 'inSyncLib.inSyncError.SyncError'>:VMware VDDK: Internal Error (#10005ffff : 3) (Error Code : VMWARE_VDDK65535). Traceback -Traceback (most recent call last):
File "agents/vmware/dataManager.py", line 513, in perform_backup_for_vmdk
SyncError: VMware VDDK: Internal Error (#10005ffff : 3) (Error Code : VMWARE_VDDK65535)
[2020-03-09 22:30:02,357] [INFO] All VMDKs Done enqueing Upload
[2020-03-09 22:30:02,357] [ERROR] DataManager: Error while uploading data, Exiting data manager
VDDK logs
Log location: PhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\VDDK.ZIP
2020-03-09T22:29:02.315-07:00| vthread-6| E110: VixDiskLib: VixDiskLib_Read: Read 6144 sectors at 30570496 failed. Error 7 (A file access error occurred on the host or guest operating system) (DiskLib error 327689: Input/output error) at 5240.
2020-03-09T22:29:32.330-07:00| vthread-6| I125: FileIOErrno2Result: Unexpected errno=5, Input/output error
2020-03-09T22:29:32.330-07:00| vthread-6| I125: DISKLIB-LIB : RWv failed ioId: #1326 (327689) (9) .
2020-03-09T22:29:32.330-07:00| vthread-6| E110: VixDiskLib: Detected DiskLib error 327689 (Input/output error).
2020-03-09T22:29:32.330-07:00| vthread-6| I125: Vix_TranslateErrno: errno = 5
2020-03-09T22:29:32.330-07:00| vthread-6| I125: Foundry operation failed with system error: Input/output error (5)
Resolution
Use vmkfstools to fix a corrupt VMDK
- Log in to the ESXi server, which hosts the VM using PuTTy. Locate the volume where the VM folder stores the VMDK : /vmfs/volumes/<DatastoreName>/<VM Name>/
- Run the following command to determine if the VMDK is corrupt
vmkfstools --fix check <diskname.vmdk> - If the disk needs repairing, issue the following command
vmkfstools --fix repair <diskname.vmdk>
Use the CHKDSK utility to fix bad blocks.
Run the command prompt as Administrator and use the command CHKDSK D: /R – to check, fix, and look for bad sectors on the specific drive.
- If the VMDK isn't corrupt and the CHKDSK utility doesn't resolve the issue, perform a storage VMotion and retry backups. A Storage VMotion moves the VMDK to an alternate datastore. Subsequent backup of the VM should not fail because of back blocks with the previous datastore.
Note: A storage VMotion changes the UUID of the VM. Retain the original UUID to avoid errors arising out of a changed UUID.
- Restarting the ESX host services like HOSTD and VPXA may also fix issues in rare cases. Use the following commands:
/etc/init.d/hostd restart
/etc/init.d/vpxa restart
- If the issue persists, create a new VMDK from an older one using vmkfstoos. Use the following command:
vmkfstools -i disk-001.vmdk disk-new-001.vmdk -d thin
-i for clone disk
-d for disk type [thin/thick]