Announcements
We are currently trying to integrate financial information from our bespoke ERP into Microsoft Dynamics 365 FO. The process is very much a batch-oriented and we send approximately 300k-500k lines in a once off daily batch.
The current integration is designed to go through via an API that's developed on Microsoft's Azure's Logic Apps. The high level integration flow is as follows:
1. Generate file
2. POST the data by invoking a Logic Apps RESTful API endpoint.
3. Once the data is received in Logic Apps, the data is processed line by line into D365 FO.
The line by line processing into D365 seems somewhat inefficient, as the number of lines can be fairly large (between 300K - 500K). This could take a while.
Is this the best/recommended approach that Microsoft offers to integrate large subsets of data? Or are there other more "batch-oriented" patterns that can be recommended.
Hello MrT007,
Understand that you have made good progress. Not sure if you have looked at docs article that talks in great details about tips and tricks for basic 'Journal' entity. Take a look and see if it helps you.
docs.microsoft.com/.../tips-tricks-import-general-journal-entity
Thanks Rahul. Based on my knowledge within the SQL world regarding "set-based" processing, I do think it would aid performance. I do understand there maybe challenges with regards to "composite" entities, but I think our use case is referring to the basic "journal" entity.
Regards
Hi Martin,
I 100% agree with your suggestion. I think using LogicApps almost like a front or proxy to expose the underlying D365 Data Management API is a good option. And I think invoking an API from the Logic App that processes the entire file, rather than processing a record at a time will definitely be more efficient.
Thanks for sharing your thoughts. Much appreciated.
Regards
Not a problem, moderators can help. I've changed the type to 'Question'.
Hi Will WU,
Unfortunately I set up this inquiry as a discussion and not a question, which is probably the reason why I can't "verify" this answer.
Apologies.
Regards
also could use data management/entities and enable set based processing
this would bring all data once to staging in D365FO
Note that you can still use LogicApps as the integration platform - it's often easier than building a direct connection. The app can then call a data management API to process the whole file, rather then processing records on by one.
Hi MrT007,
Glad to hear that, please spare a little time to verified the helpful answer to close this thread.
@Will and @Kevin,
Thanks for the prompt responses and feedback. I've been through the various links you provided which was very informative.
Just a bit more context on the current design.
1.1. File generated on bespoke ERP.
1.2. Data POSTed by invoking a Logic App RESTful API endpoint.
1.3. At a high level, Logic app does the following
1.3.1. Reads the file line by line (these are journal lines)
1.3.2. Invokes a D365FO Endpoint per line to process the line into D365FO
So, from my understanding on the links that were provided,
2.1. The "Recurring Integrations" capability and the "Data Management API" exist within the D365FO world (application suite). - @Will's Links
2.2. The "Import/Export Strategy - Batch API" and using a "Set based processing approach" also exist within the D365FO world (application suite) @Kevin's links.
Hence, Logic Apps in the current design is merely acting as a "facade"/"proxy" for invoking the core D365FO Endpoints. In theory, we could call those D365FO Endpoints directly. From a performance perspective, these are some of my thoughts.
3.1. We are essentially adding an extra hop to the process (i.e. going via Logic Apps)
3.2. Given some of the journal volumes we looking at (over 100K lines), it would be more efficient to be passing the entire data set to D365FO, and invoking some sort batch oriented utility (as per 2.1 and 2.2), instead of doing the reading(1.3.1.) and invoking(1.3.2) of the D365FO Endpoint on Logic Apps.
Once again, thanks for the advice. Much appreciated.
Stay up to date on forum activity by subscribing. You can also customize your in-app and email Notification settings across all subscriptions.
André Arnaud de Cal... 290,802 Super User 2024 Season 2
Martin Dráb 229,133 Most Valuable Professional
nmaenpaa 101,154