Problem description

Oracle Phoenix Backup Store backup job is stuck at 0% or in a queued state on the Phoenix Management Console.

Resolution

1. Check the UID on the current running job on the Poenix Management Console or detailed logs.
   Progress logs from the Phoenix Management Console
   
   +++Snip+++
   Mar 30, 2021 22:29:11RMAN backup is starting for the Mount:TEST_UPG, Snapshot:UID_715
   +++Snip+++
   
   Phoenix.log
   +++Snip+++
   level=info ts=2021-03-29T07:32:19.985236108Z filename=backup_agent.go:473 message="Waiting for Snapshot to be created"
   level=warn ts=2021-03-29T07:32:19.987170718Z filename=utils.go:93 message="Error occured while querying for snapshot" SnapDetails=Phoenix/PBS_ TEST @UID_715 message="exit status 1"
   +++Snip+++
    
2.  Log in to Phoenix Backup Store  and run the command zfs list -t snapshot to show the list of commands.
   
   +++Snip+++
   root@ip-10-175-209-83:~# zfs list -t snapshot
   NAME USED AVAIL REFER MOUNTPOINT
   Phoenix/PBS_ TEST @UID_717 140M - 9.23T -
   Phoenix/PBS_ TEST @UID_718 659K - 9.23T -
   Phoenix/PBS_ TEST @UID_719 32K - 9.23T –
   +++Snip+++
    
3. Compare the UIDs of the snapshot from phoenix.log and on the Phoenix Backup Store from the above output. From the above example you can see that UIDs for both the snapshots are not the same and there is a  difference of two snapshots, which means that snapshot UID_715 and snapshot UID_716 is deleted from Phoenix Backup Store before backing up to the Druva Cloud.
    
4. Verify the pending snapshots on Phoenix Backup Store on the  Phoenix Management Console.
   
   In this example, the pending snapshots on the Phoenix Management Console shows five snapshots because two snapshots with UIDs as UID_715 and UID_716, are deleted manually and three snapshots are available on Phoenix Backup Store with UIDs as UID_717, UID_718, and UID_719, because Druva Phoenix updates the  successfully-backed up snapshots on the Phoenix Management Console.
   
    
5. Verify if PBS_ TEST is mounted or not during backup which should be mounted during backup using from df -h.
   
   In the following example, you can  see that PBS_DB1 is mounted. It is mounted because the backup is running for this DB but not PBS_ TEST. Backup is triggered but stuck because Druva did not find the snapshot UID_715 on Phoenix Backup Store.
   
   root@ip-10-175-209-83:~# df -h
   Filesystem Size Used Avail Use% Mounted on
   udev 16G 0 16G 0% /dev
   tmpfs 3.1G 916K 3.1G 1% /run
   /dev/nvme0n1p1 30G 12G 18G 39% /
   /dev/loop5 100M 100M 0 100% /snap/core/10908
   tmpfs 3.1G 0 3.1G 0% /run/user/1000
   Phoenix 4.9T 128K 4.9T 1% /Phoenix
   Phoenix/PBS_DB1Mount 12T 6.7T 4.9T 58% /Phoenix/ PBS_DB1
    
6. Cancel the stuck job and restart the Phoenix Backup Store services. Phoenix pickups snapshot with UID_716. Follow this process till it matches snapshot UIDs that are available on Phoenix Backup Store. In this example,  you need to follow twice as two  snapshots are deleted manually before the backup.
   • Run the following command to restart the Phoenix Backup Store service:
     sudo service PhoenixBackupStore restart

Note: The maximum snapshots that will be available on Phoenix Backup Store would be five.