Batch Integration - D365FO

(0) Share

Report

Posted on by MrT007

We are currently trying to integrate financial information from our bespoke ERP into Microsoft Dynamics 365 FO. The process is very much a batch-oriented and we send approximately 300k-500k lines in a once off daily batch.

The current integration is designed to go through via an API that's developed on Microsoft's Azure's Logic Apps. The high level integration flow is as follows:

1. Generate file

2. POST the data by invoking a Logic Apps RESTful API endpoint.

3. Once the data is received in Logic Apps, the data is processed line by line into D365 FO.

The line by line processing into D365 seems somewhat inefficient, as the number of lines can be fairly large (between 300K - 500K). This could take a while.

Is this the best/recommended approach that Microsoft offers to integrate large subsets of data? Or are there other more "batch-oriented" patterns that can be recommended.

All responses (12)

Answers (4)

Poojith Jain 50 on at

Like (0)

Report

RE: Batch Integration - D365FO

Hi,

We came across the same issue and we resolved the issue using recurring integration:

https://poojithjain.wordpress.com/2020/03/28/d365uo-recurring-integration-to-avoid-dmf-parallel-execution-issue/

Poojith Jain [Blog 1, 2, 3, 4]
Suggested answer

Satish Panwar 14,645 on at

Like (0)

Report

RE: Batch Integration - D365FO

Hello MrT007,

Understand that you have made good progress. Not sure if you have looked at docs article that talks in great details about tips and tricks for basic 'Journal' entity. Take a look and see if it helps you.

docs.microsoft.com/.../tips-tricks-import-general-journal-entity
MrT007 5 on at

Like (0)

Report

RE: Batch Integration - D365FO

Thanks Rahul. Based on my knowledge within the SQL world regarding "set-based" processing, I do think it would aid performance. I do understand there maybe challenges with regards to "composite" entities, but I think our use case is referring to the basic "journal" entity.

Regards
MrT007 5 on at

Like (0)

Report

RE: Batch Integration - D365FO

Hi Martin,

I 100% agree with your suggestion. I think using LogicApps almost like a front or proxy to expose the underlying D365 Data Management API is a good option. And I think invoking an API from the Logic App that processes the entire file, rather than processing a record at a time will definitely be more efficient.

Thanks for sharing your thoughts. Much appreciated.

Regards
Martin Dráb 230,503 Most Valuable Professional on at

Like (0)

Report

RE: Batch Integration - D365FO

Not a problem, moderators can help. I've changed the type to 'Question'.
MrT007 5 on at

Like (0)

Report

RE: Batch Integration - D365FO

Hi Will WU,

Unfortunately I set up this inquiry as a discussion and not a question, which is probably the reason why I can't "verify" this answer.

Apologies.

Regards
Verified answer

Rahul Mohta 21,014 on at

Like (0)

Report

RE: Batch Integration - D365FO

also could use data management/entities and enable set based processing

this would bring all data once to staging in D365FO
Verified answer

Martin Dráb 230,503 Most Valuable Professional on at

Like (0)

Report

RE: Batch Integration - D365FO

Note that you can still use LogicApps as the integration platform - it's often easier than building a direct connection. The app can then call a data management API to process the whole file, rather then processing records on by one.
WillWU 22,350 on at

Like (0)

Report

RE: Batch Integration - D365FO

Hi MrT007,

Glad to hear that, please spare a little time to verified the helpful answer to close this thread.
MrT007 5 on at

Like (0)

Report

RE: Batch Integration - D365FO

@Will and @Kevin,

Thanks for the prompt responses and feedback. I've been through the various links you provided which was very informative.

Just a bit more context on the current design.

1.1. File generated on bespoke ERP.

1.2. Data POSTed by invoking a Logic App RESTful API endpoint.

1.3. At a high level, Logic app does the following

1.3.1. Reads the file line by line (these are journal lines)

1.3.2. Invokes a D365FO Endpoint per line to process the line into D365FO

So, from my understanding on the links that were provided,

2.1. The "Recurring Integrations" capability and the "Data Management API" exist within the D365FO world (application suite). - @Will's Links

2.2. The "Import/Export Strategy - Batch API" and using a "Set based processing approach" also exist within the D365FO world (application suite) @Kevin's links.

Hence, Logic Apps in the current design is merely acting as a "facade"/"proxy" for invoking the core D365FO Endpoints. In theory, we could call those D365FO Endpoints directly. From a performance perspective, these are some of my thoughts.

3.1. We are essentially adding an extra hop to the process (i.e. going via Logic Apps)

3.2. Given some of the journal volumes we looking at (over 100K lines), it would be more efficient to be passing the entire data set to D365FO, and invoking some sort batch oriented utility (as per 2.1 and 2.2), instead of doing the reading(1.3.1.) and invoking(1.3.2) of the D365FO Endpoint on Logic Apps.

Once again, thanks for the advice. Much appreciated.