Optimze perfomance for batch update to 1 million contacts in CRM

(0) Share

Report

Posted on by Community Member

Hello,

We have a business need where we have to update 1 million records in CRM every month. The update requries updation of 11 fields on the Contact entity.

We developed a console app for that and tried to do the Bulk update using Execute Multiple, but then we faced the time out issue. We tried to reduce the batch size from 1000 to 500, but it seems like it will take days to update 1M records.

The maximum time we have to run the batch job is 5 hours. I need to know what is the best practice or solution to perform this operation. Do we need any third party tool, or we can modify and optimize our console app to run the operations fast.

Need expert advise here.

Thanks,

Fahad

*This post is locked for comments

I have the same question (0)

All responses (7)

Answers (0)

JayeshAC 1,157 on at

Like (0)

Report

Hi Fahad,

Which version are you using - CRM online or OnPremise?

Was this reply helpful? Yes No
Friyank 944 on at

Like (0)

Report

instead of Console try powershell script.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

CRM 2016 on-Premise

Was this reply helpful? Yes No
Suggested answer

Guido Preite 54,086 Moderator on at

Like (0)

Report

if you are onpremise you can definitely optimize your console application.

The ideal will be to read first all your CRM records (and because you are onpremise you can use the SQL filtered view, it should speedup the process.

After you compare in memory the actual changes you need to update, you wrote about 11 fields, probably these 11 fields do not change always all and not for all the contacts, so when you update a contact you need to be sure that it requires to be updated (so you don't update a contact that had no changes) and just update the changed fields (for example 3 fields of 11)

hope it helps

Was this reply helpful? Yes No
Suggested answer

Community Member on at

Like (0)

Report

Hi Fahad,

when you have to deal with millions of records you have to forget about standard CRM Service.

Although it is not supported, go straight to SQL Server doing a lot of testing in a cloned environment (using SSIS, console application or commercial software like Scribe).

In the past, when I worked hard on OnPremise environments, I did it without any issue (console application).

Hope it helps.

If you found the answer helpful, please mark as Verified

Thank You & Best Regards

Francesco Picchi

Microsoft Dynamics CRM Consultant, Bologna, ITALY.

Independent Contractor

http://www.francescopicchi.com

Was this reply helpful? Yes No
Suggested answer

JayeshAC 1,157 on at

Like (0)

Report

Hi Fahad,

If you want to update millions of records then better option would be using SQL scripts instead of CRM API.

Though it is not recommended but it will solve your problem and update million of records in few minutes.

Click on below link to know more details about calling SQL script.

community.dynamics.com/.../207020

Regards,

Jayesh

Was this reply helpful? Yes No
Suggested answer

Guido Preite 54,086 Moderator on at

Like (0)

Report

direct update of SQL records is not supported, people should stop to suggest this approach

Was this reply helpful? Yes No