Microsoft Dynamics 365 | Integration, Dataverse...

Plugin - Bulk Update 160k+ records best practices

(0) Share

Report

Posted on by EBMRay

879

Hello everyone,

I have some concerns regarding plugins as below:

We have a list of contacts in dataverse with total of records up to 160K+. We want to perform a bulk update for all of those contacts by updating one column value for all of them using a plugin.

- How can we perform a bulk update into batches for all these records?
- Does plugins count towards the API limits if we want to perform this kind of update? If so, does every update operation for a record is counted as one API call?
- What is the best practice to perform this kind of update for a large data set?
- How we can handle the plugins recursion to avoid infinite loop during the update process?
- In case some contacts was not updated during the update process, how we will be able to identify the records that failed?

Could you please provide some helpful resources, a small sample code and documentation that we should follow?

Any help is highly appreciated.

Best regards,
EBMRay

All responses (9)

Answers (0)

Suggested answer

Bipin D365 28,961 Moderator on at

Like (0)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hi,

Please look at below article for Service protection limit.

docs.microsoft.com/.../api-limits

Yes, you can scheule your flow to run nightly to perform your operation.

You could create custom optionset field on your parent entity and set this field value from you ribbon button. Now you fetch records where this field is set to something which you decide from your power automate.

Please mark my answer verified if this is helpful!

Regards,

Bipin Kumar

Follow my Blog: xrmdynamicscrm.wordpress.com/
EBMRay 879 on at

Like (0)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hello Bipin Kumar ,

Did you get the chance to check the above?

Looking forward to your response.

Best regards,

EBMRay
EBMRay 879 on at

Like (0)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hi ,

Thank you for your recommendation.

If we decided to proceed with Power automate we found that we would be tentative to hit the service protection API limit (6k within the 5 minute sliding window) and in that case, some of the updates might fail. I think there are some strategies that we should follow to avoid hitting the above limitation in power automate. Is there any blog post/documentation on that?

In our requirements the flow should not be triggered on a daily basis it will be based on whenever a manager decides to shift the records to a different date for the resources based on a filtered view. We are thinking of adding a button on the ribbon that can call a Power Automate flow as suggested and perform this update process.

It can be an option to trigger the flow daily every X hours but the idea is the workflow should be triggered at night and not during business hours. Therefore, we want to make sure that the flow is efficient and has an optimized performance to process those records before the next working day starts. In case of any failure, we need the flow to generate the failed records for them to be tackled on the next update.

Currently, we don't have those numbers of records to process but we are thinking ahead to design the flow to handle a large data set in the near future.

Please advise what can be the best practice and tool to implement this kind of requirement.

Looking forward to your response.

Thank you in advance!
Suggested answer

Bipin D365 28,961 Moderator on at

Like (1)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hi,

First of all updating 100k+ records should not be Real time and should be updated in background to avoid user blocking and various performance reasons.

I don't think you can achieve your requirement with Custom workflow actvity which is called from OOB On-Deman workflow because custom workflow activity will have 2 minute timeout limitation.

What you can do for your requirement is to run your code on schedule basis for example every 2 hours and so. You can achieve scheduling with the help of Schedule flow. You can reduce that to 1 hour or every 30 mins.

https://docs.microsoft.com/en-us/power-automate/run-scheduled-tasks

So I don't think you will have those many records to process within that range and you can easily update with power automate.

Please mark my answer verified if this is helpful!

Regards,

Bipin Kumar

Follow my Blog: xrmdynamicscrm.wordpress.com/
EBMRay 879 on at

Like (0)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hello Guido Preite Bipin Kumar ,

Thank you for your answer.

I have a concern regarding the below scenario:

If we have a large set of work orders scheduled on a specific date also (100k+) and we want to perform a bulk update to shift the dates to another day based on some conditions and this custom workflow should run on demand by a user on the UI, should we also create it as a custom workflow activity or also a console app?

How we can avoid the user running the workflow multiple times and show the number of processed records on the UI since this process should be triggered from a user on UI?

Like it is in progress (10% of records are completed..)

Can you please provide some resources or documentation to get started on creating a console app?

Looking forward to your response.

Best regards,

EBMRay
EBMRay 879 on at

Like (0)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hi

Thank you so much for providing this amazing tool.

I will check it out and give it a try.

Best regards,

EBMRay
Suggested answer

Bipin D365 28,961 Moderator on at

Like (1)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hi,

It is not a best practise to use plugin in this kind of the requirement where in you want to update 160K records because plugin has 2 minute timeout limitation.

If your requirement is to update 160k records for one time then you could utilize Console app and use executemultiplerequest to bulk update records -

docs.microsoft.com/.../use-executemultiple-improve-performance-bulk-data-load

- How we can handle the plugins recursion to avoid infinite loop during the update process?

You need to use filtering attributes to spepcify the limited fields to avoid infinite loop in your plugin. Also when you should always create new Entity object instead of relying on retrieve Entity object as show below-

Entity childEntity=new Entity("Entity Logical Name");

childEntity.Id=Record GUID;

//map other fields

service.update(childEntity);

This way you are only updating fields which are required.

Please mark my answer verified if this is helpful!

Regards,

Bipin Kumar

Follow my Blog: xrmdynamicscrm.wordpress.com/
Guido Preite 54,075 Moderator on at

Like (1)

Report

RE: Plugin - Bulk Update 160k+ records best practices

in your scenario the plugin may be not the best course to follow (as plugins require a trigger event in order to be called)

the update the can be done with a on-demand workflow (and if the logic requires it, a custom workflow activity) but due to the number of updates (160k+) and the need to identify the records that failed, my suggestion is to write a console application to handle this update.

Any kind of update in your scenario will count against the API limit

hope it helps
Suggested answer

Community Member on at

Like (1)

Report

RE: Plugin - Bulk Update 160k+ records best practices

Hi EBMRay,

The Bulk Workflow Execution is a great tool which allows users to run an On-Demand Workflow against a set of records pulled from a System or Personal View, all in bulk! You can find the "Bulk Workflow Execution" tool in the Plug-in Store using the XRMToolBox application.

How to Bulk Update CRM Records Using the Bulk Workflow Execution Tool | PowerObjects

Note: Download XRM ToolBox. https://www.xrmtoolbox.com/