VMware backups stay in the queued state with IO error Error 28 no space left on device
VMware backups or log requests stay in the queued state. The system does not generate any error messages related to the backup proxy full. The following error is displayed in the logs IOError: [Error 28] No space left on device.
This can happen if you have enabled Phoenix debugging on a backup proxy and are backing up a virtual machine that is fragmented. Backups of fragmented virtual machines cause excessive processing on the proxy and generate large amounts of log data. In this situation, the /var/log volume file gets 100% full and runs out of allocated space.
You can view the error message in the /var/log/messages log
- Determine the size of the following directory
Command: du -sh /var/log/Phoenix/VMWARE/backup/
This gives us the exact size of the directories under the backup directory
- Review and delete the logs that are not required to free up space. Use the following command to delete directories that are not required:
rm -rf “backup log directory name” without quotes
- Use the instructions in the How to enable or disable debug logging for Phoenix backup proxy article to disable VMware logging.
Note: Druva recommends defragmenting virtual machines whose backup can generate a large amount of logs. Contact Druva Technical Support if you need help in determining which folders/logs are safe to delete.