Skip to main content

How can we help you?

Druva Documentation

Data Templates

Overview

As a Salesforce App administrator, you can selectively restore records or metadata from a data set from one environment to another. You can also use data templates to restore archived records. We also understand how critical it is to safeguard the functional production data when copying the data to a non-production instance.

To meet your requirements, we bring to you the following via the use of Data Templates:

  • The granular level of restoration of your Salesforce records and metadata using record selection, and query-based selection of data.
  • The exclusion of objects and associated fields. A good use-case would be ensuring certain personal information of a customer is skipped from a data copy (Sandbox Seeding) process. This is per regulations set by the General Data Protection Regulations (GDPR).
  • The application of data masking and translation to obscure production data into fully functional data. This is to avoid the usage of actual production data in a non-production instance after a data copy or sandbox seeding process.

In a data template, you can selectively configure the data set for Data Copy (Sandbox Seeding) or Data Restore and apply any data transformations, validations, exclusions, and masking.

Step 1: Creating a Data Template

Create a data template for data copy, data restore, or archival restore. For more information, see Creating a Data Template.

Step 2: Configuring a Data Template

Configure the created data template for data selection methods and additional data template settings. 

The following data selection methods are available.

Selection Method Description
Records Selection Record Selection enables you to select individual records and related records from an object to restore or copy.
Query-based Selection Use object queries to select records dynamically. You can add a single object query or multiple object queries.
Data Template Settings

Manage the data template settings, such as:

  • Excluding objects and fields
  • Applying validations, data mask, and data translations
  • Specifying the batch size of records for copying data to a destination organization
  • Discover the child objects of a parent object when working with junction objects

For more information, see Configuring a Data Template.

Creating a Data Template

Ensure that the required profile permissions are set to begin with data template creation. The profile permissions for managing data templates are Manage Data Templates and Perform Data Copy. For more information, see User Profiles.

If you are invited to a project, add the source and destination environments from the Environment Hub page. These environment settings are used in data templates.

For more information, see Adding a Salesforce Environment.

Procedure

  1. On the Salesforce App console, click Data Template.
  2. Click New. The Create Data Template window appears.

Configuring_Data_Templates_Create_a_data_template

3. Assign values to the following fields:

Fields Descriptions
Projects Projects are the organization tools to gather all of the data templates and users associated with one project in one place. Projects can be created as part of the organization settings from the Settings page.
Name Specify a unique template name within the project.
Source Org

Specify the source organization to which you want to apply the data copy or restore template.

Note: Source Org is only applicable if the data template type is Data Copy.

Destination Org Specify the destination organization to which you want to apply the data copy or restore template.

4. Select the data template type, which can be either Data Copy, Data Restore, or Archival Restore.

5. If you have selected Data Restore, the following fields appear.

Fields Descriptions
Backup Definition Specify a backup definition from the dropdown list to link this Data Restore template to a backup definition.
Backup Job Specify a backup job from the dropdown list to link this Data Restore template to a backup job.

6.  If you have selected Archival Restore, the following fields appear.

Fields Descriptions
Archival Set Select the archival set that you want to restore objects from.

7. Click Save

The Data Template fetches the records from the source organization. You can go a step further to configure your data template to filter records using data selection methods. This helps to achieve a granular data copy (Sandbox Seeding) or data restore process. 

Configuring a Data Template

The Data Template page enables users to select their data set for data copy or restore and apply any data transformations. 

The Data Template page is divided into three different views:

  1. Environment Details: The top part of the Data Template or Data Archival Restore Template page displays the source and destination environments for this specific data copy. You can change the source or destination by selecting from the dropdown list. For more information about configuring the environment, see Adding a Salesforce Environment.

data_template.png

Note: For the Archival Restore template, you will additionally see an option to Start Archival Restore. The rest of the workflow remains the same.


2. File and Data Statistics: The center part of the Data Template page displays two progress bars, one for data and the other for files. 

The different color indicators of the progress bars indicate the following:

  • Blue: Existing data and file size within the destination environment.
  • Red: Data and file size selected for data copy or restore.
  • Green: Free space within the destination environment.

Salesforce_App_Data_Template_Data_File_Statistics.png

3. Selection Methods and Template Settings: The bottom part of the Data Template page contains the data selection methods and data template settings. 

The following two ways of selecting data are available:

data_selection_method.png

Record Selection

Record Selection enables you to select individual records and related records from an object to restore or copy.

record_selection.png

Step Description
Step 1

Enter the object name in the search box. 

The table below populates the associated records.

Step 2 Select the checkbox next to the record or records you would like to copy.
Step 3

You can type in a SOQL statement in the search box below the Add/Remove Records button.

Note: You can skip the WHERE keyword in the SOQL statement.

Step 4

Select the level of record relationship to copy from the dropdown list, which can be any of the following.

  • All Child Objects
  • None
  • 1 Level
  • 2 Levels
  • 3 Levels
  • Custom

The related parent records are excluded by default during custom child object selection. Select Include Related Parent Records to include parent records.

For more information, see Understanding Object Relationships.

Step 5

Click Add/Remove Records to discover the records from the source environment.

The Tasks tab on the bottom-right corner displays the status of the record inclusion task. Click on the task entry to know more about the status of this task. This will redirect you to the Tasks page to display additional information about the associated task.

Step 6

The gear icon to the left of the WHERE clause provides control over the table of records. Select among the following options:

Column Selection

This enables you to select the columns that you want to view in the records table. After selecting this option, the Column Selection window appears.

  • Select the field from the list on the left and move it to the list on the right using the directional arrows. 
  • Use the directional arrows to determine the order of the columns.

Note: The table can display a maximum of 20 fields.

Show Selected Rows

This option displays the records for the designated object only, which are selected for migration.

Query-based selection

Select records dynamically by creating an object query. If records are altered in your source environment, the data copy activity continues to run using this method. You can add a single object query or multiple object queries.

Adding Single Object Query

Perform the following steps to query records for a single object.

1. Click Add Query. The Add Object Query window appears.

Salesforce_App_Data_Template_Query Selection.png

2. Enter information for the following fields:

Field Description
Environment Object Select the object to query.
Include All Records Discover all records that meet the search criteria. Uncheck this box to specify a total number of records to include in the Total Numbers to Include text box.
Total Records to Include Discover a specified number of records only that meet the search criteria.
Total Records to Include Per Parent Record

Discover a specified number of child records only per parent record that meet the search criteria. 

For example, if the parent record contains 10 child records, you can limit it to only include 5.

Where Clause - e.g., Id IN ('1232') AND Name = 'Test'

Specify the syntax of the condition expression for a WHERE clause of a SOQL query. For more information, see Condition Expression Syntax.

Note: Skip the WHERE keyword when specifying the condition expression.

Include Related Child Records

Select the level of child records to include from the dropdown list. 

Note: Ensure caution when selecting All Child Objects as the record may be a part of a nested relationship chain. 

For more information, see Understanding Object Relationships.

3. Click Save.

The created queries appear in a table in the Query-based Selection tab.

Adding Multiple Queries

Perform the following steps to discover records across multiple objects.

1. Click Add Multiple Queries. The Add Multiple Queries window appears.

2. Enter information for the following fields:

Field Description
Environment Object Select all objects to query by selecting the objects in the Available Objects list and using the arrows to move them into the Selected Object list.
Include All Records Discover all records that meet the search criteria. Uncheck this box to specify a total number of records to include in the Total Numbers to Include text box.
Total Records to Include Discover a specified number of records only that meet the search criteria.
Where Clause - e.g., Id IN ('1232') AND Name = 'Test'

Specify the syntax of the condition expression for a WHERE clause of a SOQL query.  For more information, see Condition Expression Syntax.

Note: Skip the WHERE keyword when specifying the condition expression.

Include Related Child Records

Select the level of child records to include from the dropdown list. 

Note: Ensure caution when selecting All Child Objects as the record may be a part of a nested relationship chain. 

For more information, see Understanding Object Relationships.

3. Click Save

The created queries appear in a table in the Query-based Selection tab.

Managing Queries

You can manage queries and add them to a data template.

query_based_selection.png

The following table describes the additional actions that you can perform for created queries.

Field Description
Add to Data Template Click Add to Data Templates to discover records from the source environment.
Clear All Queries Click Clear All Queries to delete all queries listed from the bottom table.
Query Row Action

Every query in the bottom table includes a dropdown arrow on the right. This gives additional options to users, which are as follows.

  • Add Sub-Query - A subquery enables you to query child records related to the parent object. For example, you can query cases related to specific accounts.

      Perform the steps from Adding a Single Object Query to add a  subquery.

  • Edit Query - The Add Object Query window appears enabling you to edit the existing query.
  • Delete Query - Deletes the selected query.

Data Template Settings 

You can manage the settings of a data template from the Data Template Settings tab.

Salesforce_App_Data_Template_Settings.png

Overview

Get an overview of the configured data template settings from the Overview tab. You can see the following information for all objects:

Note: When creating a Data Template, some of data template settings are applied by default. For example, the AccountContactRelation, ApexClass, and ApexPage objects are excluded. You can change these settings from the Exclude Objects and Exclude Fields tabs. For a detailed list of exclusions, see Understanding Backup Exclusions.

External ID

Salesforce uses External IDs to create the parent-child relationships between the source and the target object. 

When a field is marked with an External ID, its values can be used to match and associate related records to one another. This is used by the Salesforce App to determine if the record already exists on the destination org. As a result, the Salesforce App can prevent duplicate inserts by performing an update operation in place of an insert operation.

When to create External IDs?

If you have manually created records on the destination org or used another application to load data into the destination org, you must mark fields with external IDs. 

For example, consider a scenario where you are importing data from a .csv file with records that already exist in the destination environment without a Salesforce record ID. To prevent creating a duplicate record, you can mark a field with an external ID to map the source and destination fields for that record.

If the data is populated through a sandbox refresh (Partial or Full Sandbox), or by the Salesforce App, you do not have to define External IDs. In this case, the application automatically tracks the ID mapping between the source and destination.

Note: External IDs support composite keys.

Creating an External ID

  1. Click New External ID from the External Id tab. An External Id Form appears.
  2. Enter the following fields:
Field Description
Environment Object Search the environment object from the search box to apply the external ID.
Source Field Select a field from the source environment.
Destination Field Select a field from the destination environment.
Source RegEx You can apply a regular expression to the source field for its value to match with the destination field.
Source Substitution Specify the value which should replace the regular expression matched value.

3. Click Save.

Perform the following actions to manage external IDs for an object.

  • Click on the object name to see a list of the added external IDs. 
  • Click add_additional_extid.png to add additional external IDs for the same object.
  • Click test_object to test the object’s RegEx. 
  • Click remove_all to remove all object external IDs. 
  • Click update_external_id to update an external ID.

Exclude Objects

Exclude Objects option is used to exclude any objects from data copy into the destination org to consider data storage limitations or to filter sensitive data.

Perform the following steps to exclude an object from a data copy.

  1. Enter the object name or the search criteria to filter from the listed object names.

Note: If an object is not listed, the following object and field-level permissions may not be set to true: 

  • Queryable
  • Creatable
  • Updateable

Set queryable to true and set either creatable or updatable to true for the Salesforce App to perform upsert and update operations.

2. Select the required field and click Exclude or Include.

Note: You cannot exclude objects containing a positive record count. To exclude that object, perform an exclude record operation from the Record Selection tab.

Exclude Fields

Select fields of an object to exclude or include from a data copy. By default, the Salesforce App excludes the following types of fields:

  • Encrypted fields 
  • Fields without a Blob type
  • Fields without a calculation

Change the field settings in your source environment to populate the fields in the Salesforce App.

Perform the following steps to exclude a field.

  1. Enter an object name in the search box. The following search options are available.
Option Description
Search by object Search a field by a specific object name.
Search across objects Search a field across objects.
Show all excluded fields Displays a list of all excluded fields.

2. Select the desired field and click Exclude or Include.

Data Translations

Data translations use regular expressions to transform field values for an object from the source environment and replace them with data to match the requirements of the destination environment. 

Perform the following steps to translate data in the source environment.

  1. Enter an object name in the search box using the search by object option. Alternatively, select search across objects to search using field name.
  2.  Create regular expressions and substitutions to translate data using either of the following options.

Option 1 - Adding manually

  1. Select the desired field names and click Edit. The Add Data Translation window appears with the selected object field.
  2. Enter the regular expression and its substitution pair in the text boxes and click Add. To delete a pair, click Remove.

If multiple object fields are selected, this window displays the common data translations for the selected fields.

  3. Click Save.

Option 2 - Uploading Files

  1. Select the desired field names and click Upload Files.

The file format must be .csv with the following formatting:

Sample File Name Required formatting
myFirstImport.csv

\w,test

\s,fefe

OR
myFirstImport.csv \w,test,\s,eit

2. Click Edit to add a regular expression and a substitution pair for the data translation.

To remove a data translation, select the field name(s) and click Delete on the upper-right corner. For more information, see Using Regular Expressions.

Note: To match a NULL field value, you can use any one of the following regular expression values:

  • (.*)
  • (^$)
  • (^)
  • ($)

Data Mask

You can manage and create data masks for projects, in contrast to creating global data masks from the Settings page.  These data masks are applicable only to the current data template.

Data mask definitions created at the organizational or project level take precedence over those created within the current data template. 

Consider the following example:

  • The administrator of an organization creates a data mask definition to mask Social Security Number fields using a regular expression pattern or substitution values. 
  • A developer also adds a data mask definition to mask Social Security Number fields with a different regular expression pattern or substitution values. 
  • In this scenario, the data mask definition created by the administrator is applied as it is created at an organization level.

For more information about data masking, see Data Masking.

For a step-by-step guide and further information on creating data masks, see Creating a Data Mask.

Validations

Validations in Salesforce evaluate the data to meet defined field guidelines before saving the record.

Salesforce App disables the following Salesforce validation types by default during a data copy:

  • Validation Rules
  • Required Fields
  • Lookup Filters
  • Allow Re-parenting Fields
  • Unique Fields
  • Hidden Fields
  • Read-Only Fields
  • Inaccessible Record Types
  • Triggers
  • Flows
  • Duplicate Rules
  • Workflows

Perform the following steps to disable or enable a validation.

  1. On the Validations tab, select a Validation Type from the dropdown list. A metadata list appears for the selected validation type.
  2. Enter the search criteria in the Search By Name search box to filter from the metadata list.
  3. Click Save Selection.

Some points to consider when enabling or disabling validations:

  • The application does not support disabling certain metadata types for metadata defined in managed packages. 
  • The application supports enabling or disabling validations associated with standard fields.

For example, if a managed package validation rule exists on a Standard Object, the application disables it. Conversely, if a managed package trigger exists, the application cannot disable this trigger as only the managed package vendor can modify it.

For more information about managed packages, see Manage Packages.

  • Triggers are deployed using the Deploy() method of the Metadata API. All other validation types are enabled/disabled using CRUD operations.
  • If the application is inactivating the trigger in Salesforce, the trigger must not have any active Apex Scheduled Jobs (Future methods) running.
  • The application uses the latest API version (i.e. Generally Available (GA)) when working with metadata.
  • If a default record type is not defined for an object, the application automatically sets the first record type as default.
  • You can enable individual validations/validation types during the data copy process. During a data copy or sandbox seeding process, the application disables only the validations related to objects that are part of the data copy process (except Process Builder and Visual Flows objects).

Validation Workflow 

The following diagram illustrates the workflow when performing data copy with validations.

Data_Copy_with_Validations.png

  1. Initiate the Data Copy. A Data Copy Confirmation window appears. For more information about the Data Copy workflow, see Data Copy.
  2. Select Disable destination metadata during data copy to disable the validations during the data copy process.

Note: This option is disabled if the validations on the destination org are already disabled.

  3. If the validations are already disabled for the org, the application does not disable/re-enable the validations.

  4. If validations are not already disabled, the application retrieves the validations associated with the objects, which are applicable for data copy.

  5. The Salesforce App saves a backup of the validations, which will be disabled.

  6. It will disable the validations on the destination org.

  7. The Data Copy operation begins.

  8. The Salesforce App retrieves the validations associated with the objects, which are applicable for data copy.

   9. It enables the validations, which were previously disabled in step 6.

10. It also deletes the backup of the validations taken in step 5.

Manage Validations for an Organization

Perform the following steps to disable, enable, or refresh all validations for an org.

  1. On the Salesforce App console, click Environment Hub
  2. Click the show_det.png icon on the right for an org.
  3. Select among the following options as per your requirements.
Option Description
Disable Validations Disables all validations for an org.
Enable Validations Enables all validations for an org.
Refresh Validations Identifies all validations for an org.

Batch Size

You can specify the batch size of records for copying data to a destination organization. The Salesforce App loads a maximum of 200 records by default.

Perform the following steps to specify the batch size.

  1. On the Batch Size tab, click New. A Batch Size window appears.
  2. Select the object from the Environment Objects dropdown list.
  3. Enter the batch size in the Batch Size text box.
  4. Switch the Process Serial toggle to enable serial processing.
  5. Click Save.

Object Child Discovery

You can discover the child objects of a parent object when working with junction objects from the Object Child Discovery tab. Junction objects are used to create many-to-many relationships between two objects in Salesforce.

For more information about junction objects in Salesforce, see Junction Object.

Perform the following steps to identify all child objects for a parent object.

  1. On the Object Child Discovery tab, enter the search criteria in the search box. Alternatively, select an alphabetic character to filter the object.
  2. Select the desired object and click Save.

Cloning a Data Template

You can clone the data template by making a copy of the original template. Cloning enables you to create a data template quickly with auto-populated configurations.

Procedure

  1. On the Salesforce App console, click Data Template. The Data Template page appears with the data templates list.Percentage_deprecation_final.gif
  2. Select the template you want to clone and click Show Details. The Data Template window appears.

  3. Click Clone Data Template in the top-right corner. The Clone Data Template window appears with the auto-populated fields for Project, Name, Source Org, and Destination Org. For more information about assigning values to these fields, see Configuring a Data Template.clone_template.png

  4. Select the required items in the Items to Clone list:
Field Description
Data Record Data records of the template.
Query Selection Select records dynamically by creating an object query.
External Id External IDs to create the parent-child relationships between the source and the target object. 
Excluded Object Excluded objects from data copy into the destination org to consider data storage limitations or to filter sensitive data.
Excluded Field Excluded fields from a data copy. By default, the Salesforce App excludes the following types of fields:
  • Encrypted fields 
  • Fields without a Blob type
  • Fields without a calculation
Data Translation Transform field values for an object from the source environment and replace them with data to match the requirements of the destination environment. 
Batch Size Batch size of records for copying data to a destination organization.
Validations Validations to evaluate the data to meet defined field guidelines before saving the record.
Data Mask Data masking replaces sensitive information while copying data. The data mask configurations that were created in the original template will be added as a part of the cloned template.
  1. Click OK. The data template is cloned and appears in the data templates list.

Schedule Sandbox Seeding

You can now automate seeding data into your sandboxes using the data template scheduler.  Using the data template scheduler, you can:

  • Copy data to your sandboxes on a daily, weekly, or monthly basis

  • Ensure your sandboxes have adequate test data at all times

  • Accelerate your Salesforce development tasks as operational data is available regularly to your sandboxes.

Let’s see how you can schedule a data copy activity.

Procedure

  1. On the Salesforce App console, click Data Template.
  2. Click Scheduler. The Data Template Scheduling window appears.

Scheduler_Data_Copy_Window.png

   3. Click New Schedule.

   4. Enter the following details in the Data Template Selection area.

Field Description
Project Select the project, which includes the data copy template.
Data Template Select the data copy template using which you want to schedule a data copy job.
Source Org Select the source org from which you want to copy the data.
Destination Org

Select the destination org to which you want to copy the data.

5. Click Add to include the data template in the schedule.

6. Enter the following details in the Scheduling area.

Field Description
Repeat every Specify how often your data copy schedule should execute.
Start Date Select the start date for the data copy schedule.
Start Time Select the start time for the data copy schedule.
End Date Select the end date for the data copy schedule.

7. Specify the following details in the Record Selection area.

Field Description
Run Query-based Selection

This setting is disabled by default. When enabled, any queries defined in the Query-based Selection section within the data template are executed prior to starting the data copy operation.

For more information, see Query-based selection.

Clear existing record selection

This setting is disabled by default. When enabled, any records which are already added to the data template will be cleared. 

You can disable this setting if you have already selected specific records, which you want to copy. You can also optionally use the Run Query-based Selection option to append additional records to the data template.

 8. Configure the following details in the Data Copy Settings area as per your requirements.

  • Disable destination metadata during the data copy
  • Replace references to inactive user IDs with your user ID.
  • Replace inactive and blank record type IDs with the object's default record type ID.
  • Specify the email address to send notifications after the data copy completion.

9. Specify the maximum number of attempts for the data copy activity. This configuration will retry the data copy activity for the specified maximum number of times in case of any failures.

10. Click Save Schedule.

The data template scheduler appears in the list of created schedulers on the left pane and executes as per the configured frequency. You can click on a scheduler and edit or delete it from the right pane.

Scheduler_Data_Copy_Details.png

You can navigate to the Tasks tab from the Salesforce App console to view the following task details:

Task Description

Data Template Scheduler Post

Provides details of the data template scheduler.

Data Copy

Provides details of the data copy task, such as status and reasons for failure, if any.

For more information, see Managing Tasks.

Using Regular Expressions

The following features in the Salesforce App enable you to use regular expressions to match patterns and substitute values.

  • External IDs
  • Data Mask
  • Data Translations

Regular expressions are patterns applied to strings to match character combinations. This is used to apply actions to data.

For example, you may want to change all of the user email addresses in your data set’s suffix from .source to .destination

You can apply the following RegEx pattern to the data set and use ‘destination’ as the substitution.

Original Data Set Applied RegEx Resulting Data Set
john@yourorg.com.source, james@yourorg.com.source, jane@yourorg.com.source (.)@(.) or (\d{3})[^0-9]*(\d{2})[^0-9]*(\d{4}) john@yourorg.com.destination, james@yourorg.com.destination, jane@yourorg.com.destination

To help you construct your regular expression, see Regular Expressions for a detailed guide on how to create regular expressions. Additionally, RegEx Tester is a useful tool to test the created RegEx.

Where to go from here 

Restore Data

Copy Data (Sandbox Seeding)

 

  • Was this article helpful?