In this mini-series of blog posts, we are looking at data integration design patterns in the context of Dynamics 365 and, when relevant, Azure Service Bus. The five most common data integration patterns are:

  1. Broadcast Pattern
  2. Aggregation Pattern
  3. Bi-directional Synchronization Pattern
  4. Correlation Pattern
  5. Migration Pattern

In a previous article [1], I have already presented the Broadcast Pattern. Now it’s time to look into the next data integration pattern, the Aggregation Pattern.

The Aggregation Pattern

The aggregation integration pattern collects data from multiple systems and copies it into Dynamics 365. By aggregating data in one process, we remove the need for multiple single one-way integrations between source and target systems. This helps to mitigate concerns on data accuracy and consistency, as data is processed in an atomic transaction.

A critical aspect of the aggregation process is merging data into a single entity in Dynamics 365. It is often necessary to implement a custom logic, as part of the integration process, to merge and format data as needed. The Microsoft Common Data Service [2] is an answer in that direction, providing a connection services and a common data model for a variety of systems.

Not less critical is also tracking aggregated records sourced from different systems and stored in one or more entities in Dynamics 365. This reflects in the need for a mapping table that keeps track of records IDs in the different source systems and maps them to the entity (or entities) ID in Dynamics 365. This mapping table is typically persisted in a high transactional repository, like a relational database, and updated by the data merge custom logic during the aggregation process.

Implementation

Implementation of this integration pattern in Dynamics 365 follows the principles described in the Publish / Subscribe pattern in Azure Service Bus [3]. In this scenario, Dynamics 365 is the target system receiving data (i.e. subscribing to a Topic) from other source systems via the ESB. The ESB acts as a broker to guarantee delivery of the message at destination.

As part of the solution, we also need to take care of some (potential) data merge logic and a mapping table to track source record IDs and entity ID at destination.

Within a XRM workflow, we can read a message from the ESB by connecting to a Subscription for a Topic (code example in [3]). After obtaining the initial object wrapped in the brokered message that we have received from the Topic, we have to map it to an XRM entity. If the entity already exists in our Dynamics 365 organization, we update it, otherwise create a new one.

var client = SubscriptionClient.CreateFromConnectionString(connectionString, topicName, subscriptionName);
client.OnMessage(message => {
    Entity entity = EntityMap.Current.MapToEntity<Entity>(message);
    var exists = service.Retrieve(entity.LogicalName, entity.Id, columnSet: new ColumnSet(new[] { "Id" })) != null;
    if (exists)
        service.Update(entity);
    else
        service.Create(entity);
});


The mapping process, managed by the EntityMap class, handles two important aspects:

  1. Creates a map between the object in the external system that has sent the message to the ESB, and the entity in Dynamics 365. This map identifies the external object by its primary key and system name (i.e. “ERP”, “Payroll”, “WebSite”, etc. You may want to be more specific here.). The mapped entity is identified by its type in Dynamics 365 (Contact, Company, Lead, any custom entity, etc.) and its entity ID.
  2. Builds an entity record after the properties of the external object. This is your custom merge logic, which can be as easy as using a library like AutoMapper [4] for, indeed, auto mapping of objects, or build your own solution when more detailed control is necessary.

The Object – Entity map is a dictionary that associates a System Name – Primary Key pair to an Entity Type – Entity Id pair. The combination of System Name and Primary Key identifies uniquely an object in the external system. The combination Entity Type and Entity Id identifies uniquely an entity in Dynamics 365.

The map is populated by retrieving System Name and Primary Key from the properties of the brokered message, and then building an entity with any necessary custom logic. The solution proposed in this example takes an approach based on Type Reflection, by therefore populating entity attributes with values of the external object.

public T MapToEntity<T>(BrokeredMessage message) where T : Entity, new()
{
    string systemName = message.Properties["SystemName"] as string;
    string primaryKey = message.Properties["PrimaryKey"] as string;

    T entity = BuildEntity<T>(message);
    map.Add((systemName, primaryKey), (entity.GetType(), entity.Id));

    return entity;
}

private T BuildEntity<T>(BrokeredMessage message) where T : Entity, new()
{
    T entity = new T();
    object obj = JsonConvert.DeserializeObject(message.GetBody<string>());

    // Use reflection to fill entity attributes with property values of the object
    obj.GetType().GetProperties().ToList().ForEach(p =>
    {
        entity.Attributes[p.Name] = p.GetValue(obj);
    });

    return entity;
}

As you have noticed, System Name and Primary Key of an external object are defined as properties of the brokered message that is retrieved from the Service Bus. This implies that the publisher application set these properties before sending the message to the ESB.

public async Task SendMessageToServiceBus(object obj)
{
    var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
    var message = new BrokeredMessage(JsonConvert.SerializeObject(obj));

    message.Properties["SystemName"] = "Publisher System Name";
    message.Properties["PrimaryKey"] = "Object Primary Key";

    await client.SendAsync(message);
}

The entire solution is available free to download from my GitHub repository [5].

References

[1] Common Data Integration Patterns – The Broadcast Pattern, “XRM and Beyond” blog, Stefano Tempesta

https://community.dynamics.com/enterprise/b/xrmandbeyond/archive/2017/11/25/common-data-integration-patterns-the-broadcast-pattern

[2] Common Data Service, Microsoft Web Site

https://docs.microsoft.com/en-us/common-data-service/


[3] Publish/Subscribe Pattern in Dynamics 365 with Azure Service Bus, “XRM and Beyond” blog, Stefano Tempesta

https://community.dynamics.com/enterprise/b/xrmandbeyond/archive/2017/11/16/publish-subscribe-pattern-in-dynamics-365-with-azure-service-bus

[4] AutoMapper

http://automapper.org/

[5] Dynamics365 repository, GitHub, Stefano Tempesta

https://github.com/stefanotempesta/dynamics365