Skip to main content
Druva Documentation

How to address failure of DR failover

This article applies to:

  • OS: NA
  • Product edition: DRaaS v2.1

Overview

This article provides the steps to be performed over the Source VM, AWS proxy and failover instance to troubleshoot and get to the cause of the issue.

Checklist for the source VM

  1. Provide the output of the execution of the following command from the source virtual machine.
    (Importantly ask the antivirus installed on the server)
    wmic product >C:\test 
  2. System events and application events
  3. Free space on the system disk
  4. Virtual memory configuration, including size and location of the pagefile.sys file
  5. Check if Windows update is ongoing
  6. Partition Type of OS disk (NTFS or something else)
  7. Check if server is Domain Controller-
  8. VMchecker script Output (Only to get more information for Source VM): How to run the AWS VMImportChecker script to find the reason for VM import failure

Steps to perform on the source VM (When antivirus is installed) 

  1. If antivirus installed then ask the customer to whitelist the following program/process:
    Ec2Install.exe
    Citrix_xensetup.exe
    systeminfo.exe
    dotnetfx45.exe
    AWSPVDriverSetup8.2.1.msi
    dotNetFx40_Full_x86_x64.exe
    AmazonSSMAgentSetup.exe
    XenGuestAgent.exe
    wic_x86_enu.exe
    wic_x64_enu.exe
    WiXEC2ConfigSetup_64.msi
    C:\Druva\Model\cli.exe
    C:\Druva\Service\rmservice.exe
    C:\Druva\Model\run_model.bat
  2. Run the VM backup.
  3. Update the DR Copy.
  4. Perform the DR failover (with DEBUG_TW = True in the AWS proxy)

Steps to perform on AWS proxy

  1. Login to AWS proxy.
  2. Switch to sudo role.
    sudo su
  3. Stop Phoenix services.
    /etc/init.d/Phoenix stop
  4. Stop all the Phoenix-related process.

    ps -ef| grep -i phoenix
  5. Edit the Phoenix configuration file.

    vi /etc/Phoenix/DR/Phoenix.cfg
  6. Add below parameter ( to avoid the automatic termination of failover instance).

    Note: In case of DR failover job failure, the failover instance is AWS is terminated by default.

    Add DEBUG_TW = True
  7. Save the Phoenix configuration file.
  8. Start the Phoenix services.

    /etc/init.d/Phoenix start
  9. Initiate the DR failover again.

Steps to perform on the failover instance

Prerequisite: Failover instance must be in power on state.

  1. To get the system logs from the failover instance, select the instance >Action > Instance Settings > Get System Log.
  2. To get the instance screenshot from the failover instance, select the instance > Action > Instance Settings > Get Instance Screenshot.
  3. If the failover instance is not passing through 2/2 checks successfully:
    1. Detach the volume from the failover instance.
    2. Mount the system disk from the target to some EC2 test instance running in the same AZ.
    3. Collect the application and system logs from the attached volume.
    4. Collect the following subfolders from the Druva folder on the attached volume.
      <Drive Letter>:\Druva\Logs
      <Drive Letter>:\Druva\Temp
  4. Download Phase-1 and Phase-2 logs from S3 as specified below. (Bucket details can be fetched from the Phoenix Management Console)
    • Phase-1: When EC2 instance is launched using the RM AMI, EBS snapshot and failover settings.
    • Phase-2: When EC2 instance is booted from the DR copy root volume rather than RM root volume.

      Ex: [2019-03-22 18:51:42,789] [INFO] S3client<is_bucket_exists> exist :phoenix-1765-153859650232 Where “phoenix-1765-153859650232” is the s3 bucket.
  5. Collect the DR failover job. Refer the Logs section.