VMware backups fail with error VMWARE VDDK1 and VMware logs show 'Failed to connect to peer'. Error: The remote host certificate has these problems.
Problem description
Phoenix VMware Backups fail with Error VMWARE_VDDK1. Phoenix logs shows the VDDK error[13] You do not have access rights to this file.
Cause
This issue can be due to:
- an expired or outdated SSL certificate on the ESX, or
- ESX not being able to retrieve the latest SSL certificate from the vCenter
Traceback
Phoenix Log location: PhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\Phoenix.<timestamp>
[2020-03-25 08:41:14,914] [INFO] In vddk_lib open_disk diskpath [VMWare-5_23] fin15.usa.ccu.clearchannel.com/fin15.usa.ccu.clearchannel.com_2.vmdk
[2020-03-25 08:41:14,915] [INFO] In vddk_lib open_disk openFlag 5
[2020-03-25 08:41:22,663] [ERROR] VDDK error[13] You do not have access rights to this file.
[2020-03-25 08:41:22,664] [ERROR] Error <class 'inSyncLib.inSyncError.SyncError'>:You do not have access rights to one or more vmdk files of the VM. (#100050001 : 13) (Error Code : VMWARE_VDDK1). Traceback -Traceback (most recent call last):
File "agents/vmware/vixDiskLibOCManager.py", line 117, in handle_oc_work
File "agents/vmware/vixDiskLibHelper/vddk_lib.py", line 122, in open_disk
File "agents/vmware/vixDiskLibHelper/vddk_lib.py", line 207, in __raise_vddk_error
SyncError: You do not have access rights to one or more vmdk files of the VM. (#100050001 : 13) (Error Code : VMWARE_VDDK1)
[2020-03-25 08:41:22,665] [ERROR] Failed to open the Disk [VMWare-5_23] fin15.usa.ccu.clearchannel.com/fin15.usa.ccu.clearchannel.com_2.vmdk
VDDK Log location: PhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\VDDK.ZIP
2020-03-25T08:42:00.205-04:00| vthread-7| I125: Opening file [VMWare-5_23] fin15.usa.ccu.clearchannel.com/fin15.usa.ccu.clearchannel.com_2.vmdk (vpxa-nfcssl://[VMWare-5_23] fin15.usa.ccu.clearchannel.com/fin15...-media.com:902)
2020-03-25T08:42:00.325-04:00| vthread-7| W115: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 10 from server: NfcFssrvrOpen: Failed to open '[VMWare-5_23] fin15.usa.ccu.clearchannel.com/fin15.usa.ccu.clearchannel.com_2.vmdk'
2020-03-25T08:42:00.325-04:00| vthread-7| I125: DISKLIB-LINK : "vpxa-nfcssl://[VMWare-5_23] fin15.usa.ccu.clearchannel.com/fin15...-media.com:902" : failed to open (NBD_ERR_GENERIC).
Certificate errors:
2020-03-25T08:42:00.399-04:00| vthread-7| W115: [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: The remote host certificate has these problems:
2020-03-25T08:42:00.458-04:00| vthread-7| I125: SSL Error: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
2020-03-25T08:42:00.458-04:00| vthread-7| I125: NBD_ClientOpen: Couldn't connect to kmcesx11nyc.katz-media.com:902 The remote host certificate has these problems:
2020-03-25T08:42:00.458-04:00| vthread-7| I125+ * unable to get local issuer certificate
Resolution
The VDDK logs indicates an issue with the SSL certificate on the ESX host. A certificate is issued to the vCenter during initial configurations or while adding hosts to the vCenter.
-
To confirm whether the validity of the SSL certificate, migrate the virtual machines to another ESX, and then run a backup. If the backup completes successfully, it confirms that the old ESX has an expired or outdated SSL certificate.
-
Renew the host certificate as suggested in the following screenshot:
-
If the issue persists, migrate all the virtual machines to another ESX; reboot the old ESX to retrieve the latest certificate from the vCenter.
To avoid any further issues on the ESX, ensure that latest SSL certificates are available on ESX.