Nutanix VM Backup fails with Error AHV100
Problem description
- Nutanix VM Backups fail with error AHV 100 - Failed to discover iSCSI targets.
- Error Description: This error may occur when Phoenix fails to discover the iSCSI targets during backup or restore.
Cause
iSCSI Data Service IP is not configured or not reachable
Traceback
level=error ts=2023-03-31T02:44:54.306441158-07:00 filename=iscsi.go:53 message="Failed to discover iSCSI targets on remote machine"
level=error ts=2023-03-31T02:44:54.306558865-07:00 stack="Code: 4\\nOriginal: exit status 4\\niSCSI: connect problem caused operation to fail.\\ngoroutine 13 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\nphoenix/ahv/client/pkg/logutils.(*loggerImpl).Error(0xc0002dc900, {0xf7ada0, 0xc00046c690}, {0xea2391, 0xc00061e200}, {0x0, 0x2, 0x15})\n\t/bld/go/src/phoenix/ahv/client/pkg/logutils/logger.go:80 +0x247\nphoenix/ahv/client/agent/ahv.(*AHVState).loginTargets(0xc00014e140)\n\t/bld/go/src/phoenix/ahv/client/agent/ahv/iscsi.go:53 +0x32f\nphoenix/ahv/client/agent/ahv.(*AHVState).PrepareBackup(0xc00014e140)\n\t/bld/go/src/phoenix/ahv/client/agent/ahv/backup_manager.go:140 +0xfbf\nphoenix/ahv/client/agent/backup.(*Agent).Prepare(0xc000161ef0, {0xf9c5a8, 0xc0002de1e0})\n\t/bld/go/src/phoenix/ahv/client/agent/backup/backup.go:267 +0x3ee\ngit.druva.org/phoenixdev/hyperloop/backup.(*backup).prepare(0xc0002de1e0)\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/prepare.go:12 +0x62\ngit.druva.org/phoenixdev/hyperloop/backup.(*backup).doBackup(0xc0002de1e0)\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/job.go:133 +0xd9\ngit.druva.org/phoenixdev/hyperloop/backup.(*backup).Start.func1()\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/job.go:21 +0x10f\ncreated by git.druva.org/phoenixdev/hyperloop/backup.(*backup).Start\n\t/bld/go/src/git.druva.org/phoenixdev/hyperloop/backup/job.go:14 +0x5b\n"
level=error ts=2023-03-31T02:44:54.308986621-07:00 filename=backup_manager.go:142 message="Failed to login iSCSI targets"
level=error ts=2023-03-31T02:44:54.309098715-07:00 filename=backup.go:269 message="Failed to prepare AHV backup"
level=info ts=2023-03-31T02:44:54.398562949-07:00 filename=job.go:85 message="Abort set" method=Abort error="Failed to prepare backup sources" msg="Code: 4\\nOriginal: exit status 4\\niSCSI: connect problem caused operation to fail."
Resolution
- Connect to the PRISM Element
- Check the Cluster setting and ensure that iSCSI Data Services IP is configured (Ref: Configure Data Services IP)
The iSCSI Data Services IP address is used to provide iSCSI access to the cluster storage. It is primarily used by Nutanix Volumes to present storage to your Guest OS directly via ISCSI. This IP address acts as an iSCSI target discovery portal and initial connection point. For example, Nutanix Volumes utilizes an iSCSI data services IP address to clients for target discovery. This simplifies external iSCSI configuration on clients. - Try to telnet to the Data Services IP on port 443 and 3261 and make sure that it connects successfully
telnet <data_Services_IP> 443
telnet <data_Services_IP> 3261
(Port 3261 is used for storage traffic in Nutanix) - Ensure that the Data Source IP is not in the DHCP scope, it needs to be a static IP address.
- If in DHCP scope, assign a new static IP to it and update the same in Nutanix Cluster settings.
- Restart the Proxy Services
- Trigger a manual backup of the affected server. This time it should complete successfully
Verification
- Observe the next scheduled backup to ensure that it runs successfully without any errors