Skip to main content
Druva Documentation

VMware backups fail with NFC errors

 

Problem description


VMware backups in Phoenix fail with internal error 65535. NFC errors are visible in VMware logs. Phoenix logs show the error 14009 - The server refused connection.

Cause

Connectivity issues between the VPXd (vCenter) and VPXa (ESXi) are the root cause of NFC errors. In VMware VDDK 6.7.x and later, the issue can occur if VDDK crashes due to an I/O retry after an NBD asynchronous I/O (AIO) error.

Traceback

Phoenix logs

Log location: PhoenixLogs-Job<jobid>\<backupset>\PhoenixJob<jobid>\Phoenix.<timestamp>

Errors:

[2020-01-27 21:28:47,882] [ERROR] Error in reading from the Disk. Error Code: 14009. Retrying in 30 Seconds. Retry attempt remaining 1

[2020-01-27 21:29:17,905] [ERROR] DataManager: Error in reading data from the Disk [6000C299-9c0b-fa25-9933-edebbb1fa634] at start :10218. vix_error ==== 14009[The server refused connection.]

VMware logs

Log location: Highlight VM > Browse Datastore > Browse the VM folder > Download all vmware.log files.

Errors:

2020-01-27T21:27:42.724+01:00| vthread-8| W115: SSL: Unknown SSL Error

2020-01-27T21:27:42.724+01:00| vthread-8| I125: SSL Error: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry

2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcNetTcpWrite: bWritten: -1

2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR

2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcFssrvr_Close: failed to send close message

2020-01-27T21:27:42.724+01:00| vthread-8| W115: SSL: Unknown SSL Error

2020-01-27T21:27:42.724+01:00| vthread-8| I125: SSL Error: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry

2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcNetTcpWrite: bWritten: -1

2020-01-27T21:27:42.724+01:00| vthread-8| W115: [NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR

2020-01-27T21:27:43.791+01:00| vthread-8| I125: VixDiskLib: VixDiskLib_Close: Close disk.

2020-01-27T21:27:43.920+01:00| vthread-4| I125: VixDiskLib: VixDiskLib_Disconnect: Disconnect.

2020-01-28T18:27:18.959+01:00| vthread-6| I125: SSL: syscall error 110: Connection timed out

2020-01-28T18:27:18.959+01:00| vthread-6| W115: [NFC ERROR] NfcNetTcpWrite: bWritten: -1

2020-01-28T18:27:18.959+01:00| vthread-6| W115: [NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR

2020-01-28T18:27:18.959+01:00| vthread-6| W115: [NFC ERROR] NfcFssrvr_IOEx: failed to send io message

2020-01-28T18:27:18.959+01:00| vthread-6| I125: DISKLIB-LIB : RWv failed ioId: #1 (2338) (34)

Resolution

  • Perform a VMotion of the virtual machine. If this resolves the problem, then it is probably a network issue with the original ESX host.  
  • If the issue persists, perform a Storage VMotion of the virtual machine. If this resolves the problem, then it is probably a network issue between the ESX host and the original datastore.
  • If the issue persists despite a VMotion and an SVMotion, reboot the ESX host and the vCenter.
  • For more information on how to resolve NFC error 14009 in VMware VDDK versions 6.7.x and later, see the VMware article VDDK fails with error 14009 while backup application tries to read/write after session is disconnected (71046)