FS Backups queued after agent upgrade due to certificate issue
Problem description
File server backups are going in queued state after agent upgrade: 6.1.0-190158 and further.
Cause
This happens due to self signed certificates, if a customer has a self signed extra layer of security then it's blocking to install godaddy certificates. Which is required for the communication to phoenix.druva.com.
Note : This can be an environmental and machine specific issue.
This issue is caused by a bug which is still in open state :
Traceback
Check : C:\ProgramData\Phoenix\FS\backup\<Job ID> as the jobs are queued.
level=error ts=2022-10-07T11:47:08.9398596+01:00 message="Failed to create new connection" error="x509: certificate signed by unknown authority"
level=error ts=2022-10-07T11:47:08.9398596+01:00 filename=rpcs.go:381 message="SyncServer newClient connection failed"
level=error ts=2022-10-07T11:47:08.944862+01:00 layer=main message="Failed to create backup agent" error="x509: certificate signed by unknown authority"
Verification
- Try openssl command on customer environment: Please run below command on customer environment using openssl.
To install openssl please refer: OpenSSL for Windows- Install the Openssl
- Type openssl on command prompt
- Then it'll open open ssl prompt like OpenSSL>
- run command: s_client -showcerts -connect backup-phoenix.druva.com.com:443
- If the above command output shows the below error :
OpenSSL> s_client -showcerts -connect backup-phoenix.druva.com:443
CONNECTED(00000184)
depth=3 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify error:num=19:self signed certificate in certificate chain - This shows that there is a self signed certificate in the chain which is blocking the installation of the Go daddy certificate.
- Check on the server in question whether the godaddy url is accessible.
https://www.godaddy.com/en-in (This point is optional and it is valid if the server has internet access). - Once the above points are verified then check the same steps on any working server in the environment. (As mentioned this can be environmental and machine specific).
- If there is a difference in the certificate chain and the working server has the go daddy certificate installed then proceed with next steps.
Workaround
- Import the Go Daddy certificates from the working machine, then export it to the non-working machine.
Follow the below articles for reference :
https://support.globalsign.com/ssl/s...rosoft-windows https://learn.microsoft.com/en-us/tr...d-certificates - Restart the HybridAgent Workload service after exporting the service.