Skip to main content

Notifications

Announcements

No record found.

Dynamics 365 general forum

Mass Creation of Records on Hierarchy

Posted on by 370

I have a business need where I need to be able to create several thousand records when one single record is created. These records are based on a hierarchy of accounts. Ie when a custom entity is created on the top account it needs to trigger and create a duplicate record for all the accounts under that account in the account hierarchy (literally "under" not just children but all descendants). I need to be able to process at least 7000 records. I have tried the following:

  • I created a plugin but it hit the 2 minute timeout - (Query all accounts under and loop through creating records)
  • I created a cascading down plugin but it still hit the 2 minute timeout because we have certain accounts that have several hundred direct children - (Query all direct children and loop through creating records that then triggers creating records for all their children all the way down the hierarchy) 
  • I created a cascading down and crawling across plugin but it hit the 8 depth and was classified as an infinite loop - (This was complicated but worked on small scale, and like everything else failed when scaled up)
  • I created a Microsoft Flow job that creates the records for all the children and then restarts on the create of the children. I got this to work about half the time, but Flow seems to have issues with not always triggering and sometimes triggering twice creating duplicates. Almost every other time when I run on the large record sets it either doesn't trigger or runs and creates too many records. The other half of the time it works fine. I am working with the same set of data so it is not likely a problem with the flow.

Any ideas on how I could address this problem? I am running out of ideas. I am considering am Azure Function or Logic App, but I have never worked with them and not sure what there capabilities are. 

  • ArdantHammer Profile Picture
    ArdantHammer 370 on at
    RE: Mass Creation of Records on Hierarchy

    Kokulan,

    Good answer, I have been trying to parse out the differences between, all the items you discussed and you put it together nicely. I was really trying to get away from the idea of a batch job and stick to something that would be event driven, but I am coming to the conclusion that may not be possible. As a result if I am going the batch direction, I built a SSIS integration services job using Kingswaysoft as this is what I have worked with extensively for other batch and integration jobs. It has work flawlessly and requires less build time then a console app.

  • Verified answer
    Kokulan Profile Picture
    Kokulan 18,048 on at
    RE: Mass Creation of Records on Hierarchy

    Hi

    I think your question was on a different post as well as the issues seem somewhat similar, anyway please see the below

    Your code is taking more than 2 minutes and that's why it hits the Sandbox execution time limit and then eventually fails - Whether it is a plugin or CWA and whether it is Sycn or Async - it's going to be the same problem. In my opinion, your code should be executed outside of CRM to avoid hitting these limits and to have more control over the process. 

    I know the flow is easy to create and its a low code solution, but I would definitely lean toward an approach like Azure Function, Web Job or Console App based service

    Logic App allows you to do more complex processing but then it does have limits and it costs you more than Flow. Logic Apps does not have any free execution limits like the Flow or Azure Function.

    With Azure Function, you will again hit an execution time limit of 5 minutes, your process may take more than 5 minutes when it's processing in batches on a scheduled basis so this may not be a good choice but Durable Azure Function may work as long as you orchestrate/chain them well.

    Option One (Console App/Windows Service (possibly using TopShelf) - Running onPrem or on a Cloud VM): 

    You could have scheduled job (console app / Windows Service) On-prem running on a scheduled basis and going through accounts that are created sync last processed, and create all the records.  This code could take advantage of ExecuteMultiple request to batch requests in 5k.  With this approach you won't hit any limit, and even if CRM times outs or returns errors like Sandbox not available, your process can re-try again and again until it gets the job done.  This approach would work fine if you are happy with the child records or hierarchy of records not being created instantly. You can certainly schedule your process to run every hour or whatever the interval suits your requirement. For this, you can lift your code out from Plugin and move to console application and you do not have to do a lot of re-work and you have full control and no limits in terms of execution.

    If you host your service on-prem, you won't have to pay for VM or other cloud computing costs but it's your choice depending on your circumstance

    Option Two : (Web Job)

    You could almost all the could that you would normally do in a console app, deploy it as a scheduled web job and it will do that same as a console app except for its running on the cloud and will also cost you more

    Option Three (Durable Function)

    You could create a Durable Azure function to run on a scheduled basis. In Durable functions, you can call other functions and also save state or context. Since your process is going to take more than 5 mins in batch processing, you will have to create functions that do processing in less than 5 mins and then keep calling it until all the records are processed.  You will have to test and find out how well you can chain or loop through to get around the 5 mins limit. 

    Personally, I would think about the first option as it gives full control over the process, no limits and not much extra cost involved.

    Note: There are other options, especially using ServiceBus or event grid or other Azure services, I have only outlined the ones that won't cost much and the ones with less learning curve

    Hope this helps

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

November Spotlight Star - Khushbu Rajvi

Congratulations to a top community star!

Forum Structure Changes Coming on 11/8!

In our never-ending quest to help the Dynamics 365 Community members get answers faster …

Dynamics 365 Community Platform update – Oct 28

Welcome to the next edition of the Community Platform Update. This is a status …

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,280 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,235 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans