web
You’re offline. This is a read only version of the page.
close
Skip to main content

Announcements

No record found.

News and Announcements icon
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Microsoft Dynamics AX (Archived) / Data Replication from ...
Microsoft Dynamics AX (Archived)

Data Replication from Dynamics AX to MONGO DB

(0) ShareShare
ReportReport
Posted on by Community Member

we are attempting a solution to replicate data(transactional as well as reference) from microsoft dynamics AX to MongoDB

after reading a lot of blogs and doing brainstorming among my team, i have arrived at following three solutions

1. Creating Document Services over any transfer protocol

 poll through it and obtain the changes that have been done with regards to tables that document service is looking on

Fetch changed keys and subsequently obtain the row records based on changed keys

Pros:

since it is using AX document service, writing custom query or service with data source having multiple tables will make the process easier

Cons: 

each call to document service at an interval of 30 to 60 seconds for 40 services will burden the system and impact performance

2. Enable CreatedDateTime and ModifiedDateTime system columns on tables inside AOT, write custom stored procedures which will run at certain intervals and retrieve rows which have changed or newly created after last retrieval. 

Pros:

simpler approach and removes the dependency on dynamics ax

Cons:

we are unable to handle deleted records taking place in db with this approach

3. Turn on CDC on microsoft SQL Server residing behind Dynamics AX and monitor the change logs to replicate changes in mongo DB via message queues

Cons:

Enabling change tracking will slow down the performance of SQL Server and make it less responsive

is there any method through which we could get real time data push of table changes that a certain business flow has made?

could you suggest on best practices that are employed for replication using microsoft dynamics ax? also are there are any other approaches which we have not considered

*This post is locked for comments

I have the same question (0)
  • Martin Dráb Profile Picture
    Martin Dráb 239,040 Most Valuable Professional on at

    Extracting data directly from database is potentially risky and you would have to replicate logic that already exists in the application (table relations, interpreting enums, dealing with PartitionId and DataAreaId, display methods and so on). While you can get excellent performance, it's generally discouraged. Such a solution also couldn't be upgraded to AX 7, because there you don't have direct access to production databases.

    Note that AIF already uses change tracking under the hood. I've once extended the solution to track deletions and used it for synchronization with an external system (using X++ and .NET, in my case Entity Framework).

    I'm surprised that you didn't consider DIXF. For example, look at how Microsoft utilizes it to populate Entity Store, which is quite a similar scenario.

  • Community Member Profile Picture
    Community Member on at

    Hello Martin,

    Thanks for your valuable inputs and I agree that if we go with Database route, it would be a much efforts to understand all underlying data models. We are not planning to upgrade to Ax7 from Ax 2012 R2.

    I believe we will have explored all data model complexity during migration and that knowledge would be helpful during replication. Do you believe even in this scenario, if we go with "ModifiedDateTime" column and query tables in groups and group is mapped to mongoDb world, is a bad idea ?

    As per my understanding, CDC would create problem during aggregate different tables.

  • Martin Dráb Profile Picture
    Martin Dráb 239,040 Most Valuable Professional on at

    I wouldn't consider building a new solution by myself unless I had requirements that can't be handled by the standard application. If you don't have any, remember that the best code is the one that you don't have to write, test and maintain.

    I'm not sure what you mean by "CDC would create problem during aggregate different tables" and how it's different in case of ModifiedDateTime. Both work on table level, so that's the same, the difference is that the support for change tracking in AX abstracts from that - you enable change tracking for a whole query. If you build a custom solution that includes queries with several tables, you'll have to implement similar logic to what Microsoft has already done there.

  • Community Member Profile Picture
    Community Member on at

    i am not sure how would we enable CDC for whole query, does it mean that we can CDC on a custom query which would subsequently turn CDC on for specified data sources and dependent data source?

    in that case could you tell us or specify link on how to enable CDC in such a scenario.

    i was referring to this link "https://community.dynamics.com/ax/b/k3retailtechnicalblog/archive/2014/11/11/change-tracking-in-ax2012-r3;

    to create jobs which would be running on specified intervals to pull changed data.

    is creating jobs to do CDC an ideal way?is it the best practice?

  • Martin Dráb Profile Picture
    Martin Dráb 239,040 Most Valuable Professional on at

    Both AIF document services and DIXF entities are based on queries, not individual tables, therefore AX supports queries. As I said, CDC is based on tables, therefore this is an abstraction in the application layer. You can see it in the linked article as well (enableChangeTrackingForQuery()).

    I recommend you consider using the Data Import Export Framework (DIXF); it already has support for both incremental export and scheduling.

  • Community Member Profile Picture
    Community Member on at

    Thanks for your inputs and we will certainly take a look on DIXF. As I am novice to DynamicAX and heard DIXF first time, so please excuse if my below questions does not make much sense.

    1. Will we able to use DIXF for migration and replication both ?

    2. Does DIXF exposed itself as service, which we can invoke on certain interval, for e.g.: every one minute, to get replication in place ? Just to be on same page, with "REPLICATION" I meant that we want to push any incremental changes in some selected tables to our mongoDb based new solution.

    3. Will DIXF sit in some environment of Dynamics or it will be hosted service somewhere  else ?

    4. Does DIXF support data transformation on the go ? I am asking because in our case, Dynamics tables for e.g.:  table1, table2, table3, table4 will create single document in mongoDB.

  • Martin Dráb Profile Picture
    Martin Dráb 239,040 Most Valuable Professional on at

    If you're not familiar with DIXF, you should look at Data import/export framework user guide (AX 2012). It will answser some of your questions, such as whether it's a part of AX (it is).

    Regarding 1), I'm not sure how you distinguish between migration and replication.

    Regarding 2), I though you would simply set recurrence inside AX, without any custom development. There some services and you can expose anything you like, if you have a good reason for doing it. But then you may want to use document services instead of DIXF.

    Regarding 4, transformation options are limited, but the fact that one entity consists of several different tables is not a problem at all; it's the usual case and the reason why we have such entities in the first place. If you need some complicated transformation, using AIF may be easier (see About the AIF Pipeline and Transforms), although exporting data through DIXF and then handling transformations separately may be better from the performance perspective.

  • Community Member Profile Picture
    Community Member on at

    Hello Martin,

    Thanks for the information. In our AX implementation, there are one entity per table and relation are defined between other tables. I explored a bit around DIXF and my understanding as following:

    1. Create a Processing group and add all related tables manually in it through Entity tab.
    2. Map these tables to Staging tables with required attributes.
    3. Perform transformation on any column, if required,  through custom function and move data to target tables.  
    4. Staging and target are tables exist in same sql server.

    So flow look like : AxTable --> Staigng Table --> Per field transformation (if required) --> Target Table.

     

    The questions I have with this approach are :

    1.  Still I need to write all custom logic to consolidate different target  tables in a single json document for each record. Do you think we can achieve consolidation into a combined single entity before putting it into target table ?
    2. How do we handle delete record case ?
  • Martin Dráb Profile Picture
    Martin Dráb 239,040 Most Valuable Professional on at

    Create an entity to combine several tables.

    As I briefly mentioned, the support for change tracking doesn't cover deleted records. You must do a full synchronization in such a case. By the way, your original design with ModifiedDateTime wouldn't cover deleted records either.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Introducing the 2026 Season 1 community Super Users

Congratulations to our 2026 Super Stars!

Meet the Microsoft Dynamics 365 Contact Center Champions

We are thrilled to have these Champions in our Community!

Congratulations to the March Top 10 Community Leaders

These are the community rock stars!

Leaderboard > 🔒一 Microsoft Dynamics AX (Archived)

#1
Joris dG Profile Picture

Joris dG 5

#2
Andrew Jones a1x Profile Picture

Andrew Jones a1x 2

#3
GL-01081504-0 Profile Picture

GL-01081504-0 1

Last 30 days Overall leaderboard

Featured topics

Product updates

Dynamics 365 release plans