SQL backup failed for only one database with INTERNAL 65535 error
Problem description
- This is a corner case where SQL backups fails for only one database with INTERNAL error and rest all DBs work fine.
- The LDF file gets uploaded but MDF doesn’t upload to the cloud.
- The job failed with an error Aborting SQL data uploader
Cause
Firewall decryption aborting the database upload.
Traceback
level=info ts=2023-05-29T02:06:15.771521+01:00 filename=uploader.go:190 message="File upload started :" path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy110\Databases\DBNAME.mdf
level=info ts=2023-05-29T02:06:15.771521+01:00 filename=uploader.go:188 message="Folder upload started :" path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy109\Logs
level=info ts=2023-05-29T02:06:15.771521+01:00 filename=uploader.go:188 message="Folder upload started :" path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy110\Databases
level=info ts=2023-05-29T02:06:15.771521+01:00 filename=uploader.go:190 message="File upload started :" path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy109\Logs\DBNAME.ldf
level=info ts=2023-05-29T02:06:15.8884148+01:00 filename=uploader.go:277 message="File/Folder Synced." IsFolder=true path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy110\Databases
level=info ts=2023-05-29T02:06:16.0094457+01:00 filename=uploader.go:277 message="File/Folder Synced." IsFolder=true path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy109\Logs
level=info ts=2023-05-29T02:06:17.9499679+01:00 filename=uploader.go:277 message="File/Folder Synced." IsFolder=false path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy109\Logs\DBNAME.ldf
level=error ts=2023-05-29T02:06:28.8413122+01:00 message="Error in call" id=22 error="read tcp 10.100.2.77:56937->35.176.92.194:443: wsarecv: An existing connection was forcibly closed by the remote host."
level=error ts=2023-05-29T02:06:28.8483107+01:00 druvarpc="Retrying RPC Call inSync0114.Data.MPut in 30s" error="Call retry err: read tcp 10.100.2.77:56937->35.176.92.194:443: wsarecv: An existing connection was forcibly closed by the remote host."
In the above logs you can see for the file DBNAME.ldf we received File upload started and File/Folder Synced. but for DBNAME.mdf we only have File upload started and after that we encountered an error Error in call" id=22 error="read tcp 10.100.2.77:56937->35.176.92.194:443
level=error ts=2023-04-24T13:16:53.016474+01:00 filename=uploader.go:273 message="Failed to upload the file" IsFolder=false path=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy21\Databases\DSBNAME.mdf
level=error ts=2023-04-24T13:16:53.0194775+01:00 filename=uploader.go:341 message="Aborting SQL data uploader"
Resolution
Turn off decryption on the firewall for all traffic from the affected server.