Redgate

Fig 02 - Editing a masking rule in Data Masker

Data Masker (as seen in Figure 2) is a rules-based static data masking product that performs on the level of millions of rows per hour. It maintains the credibility of masked data by ensuring that correlated values (for instance, age and date of birth) remain consistent using substitution rules and correlated data sets that contain potential values to substitute. These are available out of the box or can be user defined, and this process can be actioned between databases or even database instances. Data Masker also has the ability to generate synthetic data in a limited capacity, best utilised when production data is either unavailable or incomplete.

Masking in Data Masker is irreversible, always retains relational integrity, and can mask primary or foreign keys without a join operation. It automatically generates reports whenever a masking rule is run, making the masking process fully auditable, and allows you to either apply your masking rules immediately or export them for use elsewhere (in SQL Clone or the SQL Data Catalog, for example).

Data Masker provides some basic sensitive data discovery functions, but in practice the meat of this is in SQL Data Catalog, which allows you to define your own taxonomy of classifications that your data can be matched against using an extensible library of built-in pattern matching rules. You can then attach actions to these tags in order to, for instance, automatically mask data that is classified as sensitive. To that end, compatibility with HIPAA, GDPR, and other regulations is provided out of the box, as is integration with various test data management processes.

SQL Clone allows you to create and centrally manage images and virtualised clones of your production data. Images are complete point-in-time copies of a database, taken from either a live server or a backup. As they are often quite large (since they are complete copies) they are usually stored centrally. You can modify your image during its creation using either SQL scripts or sets of masking rules exported from Data Masker.

Clones, on the other hand, are derived from an image and only store the differences between themselves and the image they were derived from. Due to this, they are small in size (usually less than a hundred megabytes) and can be created very quickly and wherever they are needed. For test data management, this means that your testers can provision a masked clone to their local machines whenever they need test data without having to wait on an administrator. Team management, permissions, and self-service features are provided to this end, as is integration with Git. PowerShell-driven automation is also available for image creation and management. By leveraging SQL Data Catalog, you can even distribute clones (and thus test data) as an automated part of your business processes.

SQL Provision and SQL Data Catalog together provide a complete test data management solution that includes test data discovery, masking, and provisioning (via database clones) and can readily enmesh itself into your existing processes. Cloning in particular offers significant advantages over both data subsetting and synthetic data generation: for example, it effectively guarantees that your test data will be representative.

Redgate also makes distributing that test data fast and easy. This is always a boon, but it is particularly important for enabling high-speed testing that can easily fit into (and keep pace with) a DevOps pipeline. Moreover, access to self-service and team management features help to make life easier for your developers, testers and DBAs alike, and integration with Git allows you to automatically build, validate and deploy an appropriate database clone whenever a pull request is made. In turn, this means that you can review the request against a live environment that implements it, instead of just raw code. This all works together to help facilitate a “shift left” approach to testing, allowing you to test earlier, faster, and altogether more smoothly.

The Bottom Line

SQL Provision and SQL Data Catalog form a capable, comprehensive, and cost-effective test data management solution. If it weren’t for its limited database support – its full range of functionality is only compatible with SQL Server – we would be singing its praises unreservedly. If that’s not a problem for you, Redgate should definitely be on your shortlist.

The core of Test Data Manager is its ability to create images of your production databases, then generate virtualised clones of those images that can be quickly and easily distributed to your testers via self-service. This functionality is provided by Redgate Clone, an advancement of the company’s SQL Clone product to include compatibility with the same databases that Test Data Manager supports, as well as additional functionality such as instance-level clones and support for containerised workflows.

Fig 1 - Creating an image in Redgate Test Data Manager

Images are complete point-in-time copies of a database taken from a backup. They are stored centrally, may contain multiple databases (since that is what testers usually need to work with), and can, if desired, be subsetted and masked during the image creation process (shown in Figure 1). An image preview is also available during this process, which by design is made as simple as possible while working within the product’s user interface. Subsetting and masking options are limited as a result of this: subsetting is restricted to ‘small’ and ‘large’ subsets, with the size of those subsets adjusted automatically according to the size of the original data set, while masking allows for the masking of either all data or only data that has been flagged as sensitive by the product’s data discovery functionality. Both capabilities retain relational integrity, primary/foreign key relationships, and so on, as you would expect.

On the one hand, this limited functionality may feel stifling to advanced users. On the other, it really does make the product very easy to use for everyone who is not so advanced. The former group may prefer to use the command line interface, which offers more advanced options (that we are told will be available through the regular user interface at some point).

Clones are derived from an image and only store the differences between themselves and the image they were derived from. Due to this, they are small in size (typically a couple of hundred megabytes) and can be created very quickly (usually within 20-30 seconds) and wherever they are needed. For test data management, this means that your testers can easily and quickly connect to a masked clone from their local machine whenever they need test data, without having to wait on either an administrator or a lengthy provisioning process. At the same time, the administrative overhead for test data environments is substantially reduced, because they only contain a relatively small number of assets (images) that need to be managed manually. In addition, clones created from the same image are identical in terms of their internal specifications, what database version they’re running, and so on, enforcing an additional measure of standardisation and uniformity in your testing procedures.

The product also features ephemeral servers that can be spun up and down rapidly, as needed, and en masse.

Redgate Test Data Manager provides a test data management solution built around creating and distributing database clones while also offering highly automated data subsetting, data masking, and sensitive data discovery. This offers significant advantages in and of itself: for example, leveraging database cloning effectively guarantees that your test data will be representative, discovery and masking help to ensure compliance, and the automated nature of these features can work to accelerate your test data processes.

In addition, using all of these features is easy due to the simplified approach employed within the product’s user interface, making both test data creation and distribution fast and straightforward, while access to self-service test data makes life easier for your developers, testers, and DBAs alike. At the same time, more advanced functionality is available for those who want or need it via the command line interface, as is integration with existing tools and into existing pipelines and processes.

The bottom line

Redgate Test Data Manager is an effective test data management solution built around database cloning (or virtualisation, if you prefer). While we would not say its database support is extensive, it is significantly broader than its predecessor products, and will be more than enough for many organisations. On top of that, the product’s commitment to ease of use (while still providing advanced functionality, albeit through configuration files) is compelling. In short, if Redgate Test Data Manager is an option for your environment, you should be considering it.