VMWare FLR Restore getting browse time out
Problem description
Unable to perform the VMWare FLR Restore as Druva UI shows error as "Browse time out" and “Could not load configuration file.”
Cause
FLR Browse request creates a PhoenixStats.cfg file at “/var/Phoenix/VMWARE/stats/browse” location with the browse request number directory and the “PhoenixStats.cfg” file in it. One of the stats files has “null bytes” entry Issues due to which it was unable to load config files.
Traceback
FLR browse logs:
2023-05-30 14:18:11,074] [DEBUG] Error <type 'exceptions.TypeError'>:compile() expected string without null bytes. Traceback -Traceback (most recent call last):
File "agentPartnerProcess.py", line 338, in _load_agent_config
File "roboLib/roboConfigParser.py", line 1534, in load
File "roboLib/roboConfigParser.py", line 174, in load
File "roboLib/roboConfigParser.py", line 200, in _load
File "roboLib/roboConfigParser.py", line 128, in loadstr
File "ast.py", line 37, in parse
TypeError: compile() expected string without null bytes
Mainservice.log:
[2023-05-30 14:22:42,988] [ERROR] Fuse Mount timeout occured
[2023-05-30 14:22:42,988] [ERROR] SyncError: Error in disk browse. Fuse failed.
[2023-05-30 14:22:42,988] [ERROR] Error: Error in disk browse. Fuse failed. (#100070005) (Error Code : FLR5)
[2023-05-30 14:22:42,989] [ERROR] Error <class 'inSyncLib.inSyncError.SyncError'>:Error in disk browse. Fuse failed. (#100070005) (Error Code : FLR5). Traceback -Traceback (most recent call last):
File "client/vmware/vmware_client_rpcs.py", line 280, in vmware_command_disk_browse
File "client/vmware/vmware_client_rpcs.py", line 725, in wait_for_mount
File "inSyncLib/inSyncError.py", line 32, in raise_error
SyncError: Error in disk browse. Fuse failed. (#100070005) (Error Code : FLR5)
[2023-05-30 14:22:42,989] [ERROR] SyncError occured in disk browse. Cleaning up...
Resolution
- Identify and Login to the Backup Proxy server.
- Identify the Proxy (If customer have multiple proxy):
- Log in to the proxy and goto “/var/log/Phoenix/VMWARE/browse” directory.
- Use command “ls -ltr” to find recent updated directory by time and “cd” to the latest modified directory.
- Use “ls -ltr” again to sort and find latest log file to identify if the proxy is being used for browse job.
- Go to “/var/Phoenix/VMWARE/stats/browse”
- Identify the backup set ID which is having issues. (eg 131640)
- Identify the backup set ID:
- Navigate to “/var/log/Phoenix/VMWARE/browse” directory.
- Open the browse job log, It will have the backup set id.
- Eg: u'vm_backupset_id': 182164
- Delete problematic backupset ID directory/s and file available under the “/var/Phoenix/VMWARE/stats/browse” directory.
- Note: Before deletion please take a backup of the /var/Phoenix/VMWARE/stats/browse at some other location in the proxy.
- Restart the Phoenix services
eg:
[root@RJP-BDR-02 browse]# ls
131640 131641
[root@RJP-BDR-02 browse]# rm -r 131640
rm: descend into directory ‘131640’? y
rm: remove regular file ‘131640/PhoenixStats.cfg’? y
rm: remove directory ‘131640’? y
[root@RJP-BDR-02 browse]# service Phoenix restart
Restarting Phoenix (via systemctl): [ OK ]
[root@RJP-BDR-02 browse]# ls
Note: You might need to delete multiple directories if the issue is with multiple backup set.