VM backup fails with VMWARE_VDDK65535 error after ESXi or vCenter certificate update
This article applies to:
- OS: VMware
- Product edition: Phoenix
- Feature category: Configuration
Problem description
When the ESXi or vCenter server certificate is updated, VM backups start failing with the VMWARE_VDDK65535 error.
Cause
This issue occurs when the Phoenix backup proxy tries to connect to the ESXi or vCenter server and starts the back up of the VMs using the SSL certificate provided by the ESXi or vCenter server. The backup proxy stores the certificate thumbprint when the ESXi or vCenter server credentials are configured and uses them every time during backup. Any changes to the certificate are not updated automatically to the thumbprint in the backup proxy. This causes a discrepancy between the certificate present on the ESXi or vCenter server and the one on the backup proxy, resulting in the VMWARE_VDDK65535 error.
Traceback
The failed logs can be downloaded and verified from vixDiskLib as follows:
2017-11-20T21:26:44.416-05:00| vthread-5| I125: 2017-11-20T21:26:44.416-05:00 error -[7FD1FC167700] [Originator@6876 sub=HttpConnectionPool-000000] [ConnectComplete] Connect failed to <cs p:00000000030e5390, TCP:10.10.5.111:443>; cnx: (null), error: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ --> PeerThumbprint: 1C:04:F6:2E:FF:41:AF:95:xx:3C:xx:D7:xx:93:36:A9:2E:CF:89:00
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ --> ExpectedThumbprint: f9:a0:e1:64:a9:xx:4c:xx:2e:xx:20:82:d7:63:9e:xx:cd:c3:f8:e1
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ --> ExpectedPeerName: 10.10.5.111
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ --> The remote host certificate has these problems:
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ -->
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ --> * Host name does not match the subject name(s) in certificate.
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ -->
2017-11-20T21:26:44.416-05:00| vthread-5| I125+ --> * self signed certificate in certificate chain)
Phoenix job logs:
19-04-10 05:13:29,945] [ERROR] [SSL] shutdown while in init (_ssl.c:1941)
[2019-04-10 05:13:29,945] [ERROR] Error <class 'ssl.SSLError'>:[SSL] shutdown while in init (_ssl.c:1941). Traceback -Traceback (most recent call last):
SSLError: [SSL] shutdown while in init (_ssl.c:1941)
VMware VDDK logs:
2019-04-10T05:13:29.666+05:30| vthread-4| E110: 2019-04-10T05:13:29.666+05:30 VixDiskLibVim: Failed to verify SSL certificate: actual thumbprint=7E:E7:B3:9D:1C:8F:D2:5C:2D:25:9C:F0:63:A7:BF:4D:AD:45:10:4E expected=AF:D4:3F:3A:42:31:47:82:0A:17:44:19:35:91:E5:2B:EE:AD:0A:F4
2019-04-10T05:13:29.668+05:30| vthread-4| E110: 2019-04-10T05:13:29.668+05:30 VixDiskLibVim: Error 18000 (listener error GVmomiFaultInvalidResponse).
2019-04-10T05:13:29.668+05:30| vthread-4| W115: 2019-04-10T05:13:29.668+05:30 VixDiskLibVim: Login failure. Callback error 18000 at 2500.
2019-04-10T05:13:29.668+05:30| vthread-4| E110: 2019-04-10T05:13:29.668+05:30 VixDiskLibVim: Failed to find the VM. Error 18000 at 2572.
Resolution for a vCenter server
Update the vCenter credentials from the Management Console and restart the backup proxy as follows:
- All vCenters/ESXi Hosts - > Click three dots - > Update Credentials
- Click on Update once username and password information is enter.
Resolution for a standalone ESXi server
- From the vSphere Client console, click VMs and Templates, and start the backup proxy virtual machine.
- Open a terminal on the virtual machine.
- Log on to the virtual machine.
- To set the vCenter Server or ESXi hypervisor credentials, run the following command.
You can find the vCenterDetail utility at /opt/Druva/Phoenix/bin.
vCenterDetail set <FQDN or IP> <Username> <Password>
To set credentials for VMware setup on cloud, run:
vCenterDetail set vmc <FQDN or IP> <Username> <Password>
In the above command:- <FQDN or IP> is the fully qualified domain name (FQDN) or the IP address of the vCenter Server or ESXi hypervisor.
- <Username> and <Password> are the credentials rquired to log on to the vSphere Client.