Process workflow of application-aware backups on VMware virtual machines

SQL Server aware backup workflow on VMware virtual machines

• Step 1

  • Virtual machine backup request is initiated based on,

    • Schedule defined in the backup policy.

    • Manual backup

  • Druva forwards the backup request to the backup proxy pool.

  • Based on the load balancing algorithm, Druva automatically identifies the backup proxy that will run the backup request.
  • If the identified backup proxy is busy, the backup request is queued and initiated when the backup proxy becomes free.
     
• Step 2
  • In an environment where virtual machines are deployed on ESXi hosts managed by vCenter server, backup proxy contacts vCenter Server to locate the virtual machine and obtains its configuration.
  • In an environment where virtual machines are deployed on standalone ESXi hosts, backup proxy contacts ESXi host to obtain the virtual machine configuration.
     
• Step 3
  • If the SQL aware backup is enabled for the virtual machine, Druva performs the following steps:
    1. The backup proxy directly connects to the virtual machine using the network infrastructure. 
    2. The backup proxy injects the executables (guestossvc.exe) and (PhoenixSQLGuestPlugin.exe) into the VM.
    3. The backup proxy uses VMware tools to run (guestossvc) process inside the guest OS that starts an API.
    4. The guestossvc process starts the SQL Server executable (PhoenixSQLGuestPlugin) process. The backup proxy uses the opened port 3542 on the VM for communication with the guest OS process (guestossvc).
    5. The guestossvc process uses port 3545 on the Windows Server VM to communicate with PhoenixSQLGuestPlugin process.
    6. The SQL Server executable process(PhoenixSQLGuestPlugin) creates a persistent VSS recovery point of all the SQL Server instances.
    7. The Druva backup proxy performs the following steps:

       • The backup proxy queries the ESXi hypervisor or the vCenter server to create VM recovery points that contain VMDK files and the VMX files of the virtual machines.  Ensure that enough storage is available on the local datastore for the VM recovery point. 

       • The backup proxy uses CBT and Tools Quiescing to create a virtual machine recovery point. For the first automatic backup, the backup proxy creates a recovery point of a full backup. For all subsequent backups, the backup proxy creates recovery points of the incremental backups. 

• Step 4

  1. Backup proxy establishes a VDDK connection to the VM recovery point using a transport mode to read the VM data. 

  2. Reading encrypted VMDK through the NBD mode is not supported. The backup proxy can read encrypted VMDK disks either using NBDSSL or HotAdd. Here are the supported transport modes:

     • Encrypted backup proxy:

       • For encrypted virtual machine the supported transport mode is HotAdd.

       • For unencrypted virtual machine the supported transport mode is HotAdd.

     • Unencrypted backup proxy:

       • For encrypted virtual machine the supported transport mode is NBD-SSL.

• Step 5
  • Backup proxy starts reading the virtual machine recovery point.
    Note: The backup proxy reads the VM recovery point on the local datastore.
     
• Step 6
  • The backup proxy reads the virtual machine recovery point and prepares to send the backup data to CloudCache (if configured) or Druva Cloud.
• Step 7

  • Backup proxy transfers the backup data in a continuous stream to the Druva Cloud. Before transferring the data to the Druva Cloud, the backup proxy performs the de-duplication.

  • After the transfer completes, the backup proxy deletes the recovery point stored on the local datastore. 

• Step 8

  • After the backup proxy uploads the virtual machine backup to the Druva Cloud, it performs the following steps:

    • Stop the SQL Server executable
    • Remove the SQL Server executable files
    • Stops the guest OS process
    • Removes the guest OS process files and executable
    • Removes the VSS recovery point from the guest

Transaction log backup workflow on VMware virtual machines with SQL Server aware backups enabled

In addition to the full SQL Server database backup, Druva supports transaction log backup of the databases to provide a tighter recovery point objective (RPO). The following workflow describes how Druva backs up transaction logs of the SQL Server databases on the virtual machines for which the backup proxy runs SQL Server aware backups. 

• Step 1

  • The backup policy initiates a transaction log backup request. Druva forwards the backup request to the backup proxy pool.

    • Based on the load balancing algorithm, Druva automatically identifies the backup proxy that will run the backup request.
    • If the identified backup proxy is busy, the backup request is queued and initiated when the backup proxy becomes free.
       
• Step 2
  • In an environment where virtual machines are deployed on ESXi hosts managed by vCenter server, backup proxy contacts vCenter Server to locate the virtual machine and obtains its configuration.
  • In an environment where virtual machines are deployed on standalone ESXi hosts, backup proxy contacts ESXi host to obtain the virtual machine configuration.
     
• Step 3
  1. The backup proxy directly connects to the virtual machine using the network infrastructure. 
  2. The backup proxy injects the executables (guestossvc.exe) and (PhoenixSQLGuestPlugin.exe) into the VM.
  3. The backup proxy uses VMware tools to run (guestossvc) process inside the guest OS that starts an API.
  4. The guestossvc process starts the SQL Server executable (PhoenixSQLGuestPlugin) process. The backup proxy uses the opened port 3542 on the VM for communication with the guest OS process (guestossvc).
  5. The guestossvc process uses port 3545 on the Windows Server VM to communicate with PhoenixSQLGuestPlugin process.
  6. The SQL Server executable process (PhoenixSQLGuestPlugin) uses the virtual device interface (VDI) to take log backups of the databases on the virtual machines. It copies the transaction logs to this location: %ProgramData%\Phoenix\VMWare\SQL\TlLogs\. In addition, the SQL Server executable process truncates the transaction logs. 

     Note: Since the SQL Server executable copies the transaction logs to the <PATH> on the virtual machine, the process can cause storage consumption bloat. 

• Step 4
  • The guest OS process (guestossvc) uploads logs to the Druva Cloud through the backup proxy.
     
• Step 5 

  • After the backup proxy uploads the transaction logs to the Druva Cloud, it performs the following steps:

    • Stop the SQL Server executable
    • Remove the SQL Server executable files
    • Stops the guest OS process
    • Removes the guest OS process files and executable
    • Removes the transaction logs on the guest OS

The Druva backup proxy starts from step 1 when the next log backup schedule kicks in. For example, you specified an interval of 30 minutes. If the full virtual machine SQL Server aware backup got completed at 12:00 PM, the first log backup happens at 12:00 PM. If the first log backup completes at 12:20 PM, the next log backup starts according to the schedule at 12:30 PM. If the first log backup runs until 12:45 PM, the next log backup starts immediately as soon as the current log backup ends. The Druva backup proxy continues backing up transaction logs until the next full SQL Server aware VM backup runs.

To know more about virtual machine backup, see Backup and Restore VMware Virtual Machines.

If you observe the below error in your Windows event logs, contact Druva Support:

Event ID: 57 NTFS Warning
The system failed to flush data to the transaction log. Corruption may occur.

Event ID: 137 NTFS Error
The default transaction resource manager on volume \\?\Volume{806289e8-6088-11e0-a168-005056ae003d} encountered a non-retryable error and could not start. The data contains the error code.

Event ID: 140 NTFS Warning
The system failed to flush data to the transaction log. Corruption may occur in VolumeId:<> DeviceName: \Device\HarddiskVolume<>.(A device which does not exist was specified.). 

Few databases might not be recoverable since the VSS service failed to keep the SQL VSS recovery point persistent on the source VM. Druva is working closely with VMware support to address the above problem.