Configuration Migration Tool: Working Smarter with Metadata Entities

If you mostly work with configuring the Microsoft Dynamics CRM solutions via the customization UI with its graphical point & click tools instead of writing custom code in Visual Studio, you may not keep such a close eye on the changes introduced in the CRM SDK. After all, the term SDK stands for “software development kit” and you may not consider yourself as the target audience if you don’t actually develop any new software that would interface with Dynamics CRM.

The What’s New page in the CRM SDK documentation is a place that would be very useful for also non-developers to have a look at whenever updates to the product are released. Apart from the end user guidance material published on CRM Customer Center, the SDK is pretty much the official place where changes to the platform functionality are introduced. With the latest CRM 2013 Service Pack 1 release (a.k.a Leo) now available, there’s a long list of new and improved features to review. One new addition that didn’t receive a mention on that page, though, is the Configuration Migration Tool, which is a very handy tool if you need to transfer metadata entities across different CRM organizations.

What Are “Metadata Entities”?

In theory the CRM solution files should contain all of the configuration data about your CRM organization. This is what is often referred to as metadata, which could be described as “data about data” in short. As an example, the list of possible values for the Relationship Type attribute on your account record would be considered metadata, whereas the actual account records referring to one of these values would be “real” data stored in the business entities. To illustrate the difference, a blank CRM organization containing only your customizations would not contain any records for the business entities (accounts, contacts, opportunities), but it would contain all of the records in the metadata entities that would be necessary for executing the customized business processes, since otherwise the configuration of that environment would be incomplete.

In practice there are often valid reasons for why it makes more sense to store this type of metadata in a custom entity of its own, instead of using option sets. This provides a whole different level of flexibility in terms of what type of metadata and how much information you can store. Maintaining the list of values can be done without the need to update the customizations. Filtering available records to the end user can be done via standard security roles and BU’s. Building dependent lists on the form can be achieved via the filtered lookups feature. You can leverage the new Quick View Forms to show data from related parental records. And so on.

One problem with using these type of metadata entities is that the data cannot be packaged into a CRM solution file. While you can easily export any changes to the standard customization components and move them to a different CRM organization, getting the metadata entity records from one org to another requires a bit more effort. As a result, keeping the different development, test and production environments in sync requires specific attention to be paid on these metadata during the release process.

Introducing the Configuration Migration Tool

The CRM 2013 Service Pack 1 release includes the new Unified Service Desk (USD) application that relies heavily on storing its configuration data into these metadata entities. In order to support the systematic deployment of USD solutions configured for the customer’s environment, Microsoft has introduced new tools in the version 6.1 (a.k.a. CRM 2013 SP1) SDK. These are the Configuration Migration Tool and the Package Deployer tool. While the latter one is surely useful in deploying larger ISV solutions, I believe it’s the Configuration Migration Tool that can benefit a far wider audience. Also, since you don’t need to run Visual Studio to work with the tool, I’m personally more comfortable discussing its features in my blog as a non-developer who generally steers clear from anything that isn’t close to point & click.

In short, what the Configuration Migration Tool (from here on referred to as “CMT”) allows you to do is 1) choose the entities that you want to migrate data from, 2) export all the records from those entities as a .zip file and 3) import it into a different CRM organization. There’s a great illustration available in the SDK article describing CMT, presented below:

“Awesome, now we have a built-in tool for CRM data migrations!” Well, actually, no. While CMT can move a set of records from several entities in one package and maintain their relationships, it has not been designed for the purpose of business data migration. For example, you cannot filter out any of the records to be transferred from the source organization, nor are there any guarantees for the tool’s performance when moving very large sets of data. Here’s what the Dynamics CRM product team has to say about the tool:

Check out this CMT feature overview video on YouTube for more information on the above points.

Moving Configuration Data with CMT

Let’s see how the tool works in practice. To test the features, I created two custom entities for storing my configuration data: “Category” and “Subcategory”. A category can have many subcategories (1:N relationship), but in order to make the experiment more interesting, I also added a lookup on the category form that allows selecting the default subcategory to be used. So, we have both a 1:N and N:1 relationship between our entities, as shown on the Category form below. For the purpose of testing I added category A with subcategories 1*, 2, 3 and 4, then category B with subcategories 1 and 2*, with the asterisk determining the chosen default subcategory (notice the duplicate names in the subcategory data, we’ll get to that later on).

After launching CMT, choosing to define the schema (step 1) and entering the connection information to log into my own development CRM 2013 server via IFD (hint: enter the server URL with the org name as “org.domain.com” and the user name as “user@domain.com, no other connection option worked for me), we’re presented with a screen for choosing the components to be included in the schema. We can pick from entities included in a specific solution. After selecting the entity, we can either choose which fields we want to transfer data from or just be lazy and add the whole entity into our schema definition. Save and Export will give us an XML file with the contents from the included entities, fields and relationships.

The next step is the data export. Here we’ll choose the schema XML we just created and a target file for the data export. Hitting the Export button will start the process, report the progress for each entity in the CMT app window and finally produce a .zip file containing the data for all the components defined in the schema file.

The last step is to connect to a different organization that already has the same schema. So, if you’ve not yet transferred your solution files into the target organization, now’s the time to do that, since CMT will not modify the organization schema, but rather just assume that it is identical to the source organization as far as the selected entities and fields go.

After you’re sure that the source and target schema are the same, start the import process from the .zip file. The CMT window will again show the details of the process and inform you about the number of records imported.

Import process completed! Now we have two CRM orgs with identical data in our chosen configuration entities and fields.

Reviewing the Results

In the CMT UI it looked like all of the data was imported with no errors. When inspecting the results from the target organization, though, something wasn’t quite right. Even though the CMT log showed that 6 Subcategory records had been imported, all I had in my target system was 4 records:

On the Category record form, it seemed like the reference to the default subcategory had been populated (as a result of the “second pass updates” shown in the import process log window above), but the references were actually incorrect. The subcategory value for Category A was actually the Subcategory 1 record that was a child record of Category B in the source system. Hmm, what’s going on here?

CMT provides access to the log file of the latest import job conveniently from the top toolbar, so I opened up the logs and went to review the entries. I found two rows about the subcategory entity, saying “Switching Insert to Update for Subcategory, jkn_name = 1, ID: 590a3faa-5dee-e311-9414-00155d569f46″. So, it appears that because subcategory records with primary field values 1 and 2 had already been inserted into the target system, CMT did not create the last two records but instead updated the contents of the earlier records, thus creating the mismatches discovered.

Had I taken the unconventional RTFM route and actually studied the SDK documentation before the import, I would have already known that this is actually the default behavior of CMT. Unless you specify a uniqueness condition during the schema creation stage, CMT uses the primary field name of the entity to compare against the existing data on the target system. To change this, you should click Tools – Configure Import Settings and then select the fields that determine a unique record in your source data. OK, that’s definitely a good feature to be aware of!

Since there was now a mismatch between the configuration data in different environments, this was a good opportunity to try out the capabilities of CMT in maintaining changes in such data. To make the sample data a bit easier to match, I decided to update the subcategory names (primary fields) with the parent category prefix, such as A1, B2 and so on. This way there would be no duplicate values to be found, even without touching the uniqueness condition.

I then started a new export with the existing schema and created a new .zip file. Then I ran the import process to the target organization again. The question in my mind during this operation was “I wonder what the end result will look like after this”. It turns out that the export-import process was able to correctly update the existing records, create the missing subcategories and link them to one another with no errors. Configuration successfully migrated!

One nice feature of the Configuration Migration Tool is that not only will it carry over the visible record fields with identical content, but it can also preserve the record GUID’s across CRM organizations. This means that any solution components with references to specific GUID values (such as workflow processes) should also transfer from one environment to another without requiring manual reconfiguration tasks. That’s how CMT know how to update subcategory “1″ to “A1″, for example, without creating a brand new record. The only configuration changes that would need manual intervention would be the deletion of configuration records from the target system, as this falls outside of the scope of functionality in CMT.

If you haven’t previously explored the contents of the Dynamics CRM SDK but want to try out the Configuration Migration Tool, then just download the SDK, extract it and navigate to the Tools folder to find the CMT application files.

The post Configuration Migration Tool: Working Smarter with Metadata Entities appeared first on Surviving CRM.