Skip to main content

Notifications

Dynamics 365 Community / Forums / Finance forum / Batch Integration - D3...
Finance forum
Answered

Batch Integration - D365FO

Posted on by 5

We are currently trying to integrate financial information from our bespoke ERP into Microsoft Dynamics 365 FO. The process is very much a batch-oriented and we send approximately 300k-500k lines in a once off daily batch.

The current integration is designed to go through via an API that's developed on Microsoft's Azure's Logic Apps. The high level integration flow is as follows: 

1. Generate file 

2. POST the data by invoking a Logic Apps RESTful API endpoint.

3. Once the data is received in Logic Apps, the data is processed line by line into D365 FO. 

The line by line processing into D365 seems somewhat inefficient, as the number of lines can be fairly large (between 300K - 500K). This could take a while.

Is this the best/recommended approach that Microsoft offers to integrate large subsets of data? Or are there other more "batch-oriented" patterns that can be recommended.

  • Poojith Jain Profile Picture
    Poojith Jain 50 on at
    RE: Batch Integration - D365FO

    Hi,

    We came across the same issue and we resolved the issue using recurring integration:

    https://poojithjain.wordpress.com/2020/03/28/d365uo-recurring-integration-to-avoid-dmf-parallel-execution-issue/

    Poojith Jain [Blog   1234]

  • Suggested answer
    Satish Panwar Profile Picture
    Satish Panwar 14,643 on at
    RE: Batch Integration - D365FO

    Hello MrT007,

    Understand that you have made good progress. Not sure if you have looked at docs article that talks in great details about tips and tricks for basic 'Journal' entity. Take a look and see if it helps you.

    docs.microsoft.com/.../tips-tricks-import-general-journal-entity

  • MrT007 Profile Picture
    MrT007 5 on at
    RE: Batch Integration - D365FO

    Thanks Rahul. Based on my knowledge within the SQL world regarding "set-based" processing, I do think it would aid performance. I do understand there maybe challenges with regards to "composite" entities, but I think our use case is referring to the basic "journal" entity.

    Regards

  • MrT007 Profile Picture
    MrT007 5 on at
    RE: Batch Integration - D365FO

    Hi Martin,

    I 100% agree with your suggestion. I think using LogicApps almost like a front or proxy to expose the underlying D365 Data Management API is a good option.  And I think invoking an API from the Logic App that processes the entire file, rather than processing a record at a time will definitely be more efficient.

    Thanks for sharing your thoughts. Much appreciated.

    Regards

  • Martin Dráb Profile Picture
    Martin Dráb 229,133 Most Valuable Professional on at
    RE: Batch Integration - D365FO

    Not a problem, moderators can help. I've changed the type to 'Question'.

  • MrT007 Profile Picture
    MrT007 5 on at
    RE: Batch Integration - D365FO

    Hi Will WU,

    Unfortunately I set up this inquiry as a discussion and not a question, which is probably the reason why I can't "verify" this answer.

    Apologies.

    Regards

  • Verified answer
    Rahul Mohta Profile Picture
    Rahul Mohta 21,014 on at
    RE: Batch Integration - D365FO

    also could use data management/entities and enable set based processing

    this would bring all data once to staging in D365FO

  • Verified answer
    Martin Dráb Profile Picture
    Martin Dráb 229,133 Most Valuable Professional on at
    RE: Batch Integration - D365FO

    Note that you can still use LogicApps as the integration platform - it's often easier than building a direct connection. The app can then call a data management API to process the whole file, rather then processing records on by one.

  • WillWU Profile Picture
    WillWU 22,350 on at
    RE: Batch Integration - D365FO

    Hi MrT007,

    Glad to hear that, please spare a little time to verified the helpful answer to close this thread.

  • MrT007 Profile Picture
    MrT007 5 on at
    RE: Batch Integration - D365FO

    @Will and @Kevin,

    Thanks for the prompt responses and feedback. I've been through the various links you provided which was very informative.

    Just a bit more context on the current design.

    1.1. File generated on bespoke ERP.

    1.2. Data POSTed by invoking a Logic App RESTful API endpoint.

    1.3. At a high level, Logic app does the following

    1.3.1. Reads the file line by line (these are journal lines)

    1.3.2. Invokes a D365FO Endpoint per line to process the line into D365FO

    So, from my understanding on the links that were provided,

    2.1.  The "Recurring Integrations" capability and the "Data Management API" exist within the D365FO world (application suite). - @Will's Links

    2.2.  The "Import/Export Strategy - Batch API" and using a "Set based processing approach" also exist within the D365FO world (application suite) @Kevin's links.

    Hence, Logic Apps in the current design is merely acting as a "facade"/"proxy" for invoking the core D365FO Endpoints. In theory, we could call those D365FO Endpoints directly. From a performance perspective, these are some of my thoughts.

    3.1. We are essentially adding an extra hop to the process (i.e. going via Logic Apps)

    3.2. Given some of the journal volumes we looking at (over 100K lines), it would be more efficient to be passing the entire data set to D365FO, and invoking some sort batch oriented utility (as per 2.1 and 2.2), instead of doing the reading(1.3.1.) and invoking(1.3.2) of the D365FO Endpoint on Logic Apps.

    Once again, thanks for the advice. Much appreciated.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

News and Announcements

Give Back to the Community this Month

Quick Links

Forum Structure Changes Coming on 11/8!

In our never-ending quest to help the Dynamics 365 Community members get answers faster …

Dynamics 365 Community Platform update – Oct 28

Welcome to the next edition of the Community Platform Update. This is a status …

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 290,802 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 229,133 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,154

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans