Load large amount of data to CRM, every day.

(1) Share

Report

Posted on by Community Member

Hi,

We need to load a lot of data daily to CRM. Using the SDK it becomes very slow. There any viable way to insert and update data quickly (and supported by Dynamics CRM)?.

Unfortunately much of the information changes, around 90% of the data is updated daily.

We appreciate any help!

*This post is locked for comments

I have the same question (0)

All responses (13)

Answers (0)

alf 17,915 on at

Like (1)

Report

Hi,

There are a number of things you can do to speed up the data load. HOw much is a lot of data?

To start with i'd design the integration to go do things in parallel, cache lookups, and minimize round trips were ever possible. The speed can improve dramatically if you do things in parallel:

www.powerobjects.com/.../crm-data-integration-performance-testing-dynamics-crm

The second item you can look at is the new crm data loader. This is in preview mode. However, APIs are not available yet.

blogs.msdn.com/.../preview-feature-data-loader-for-microsoft-dynamics-crm-online.aspx

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

Thanks Alex.

Average: 1 Million Contact / with approximate 2 records per contact (in new entity, with 50+ fields).

using the SDK, testing takes more than 1 day.

Was this reply helpful? Yes No
Suggested answer

a33ik 84,331 Most Valuable Professional on at

Like (0)

Report

Have you tried SSIS or Scribe? I'm pretty sure that most of data operations can be paralleled.

Was this reply helpful? Yes No
Suggested answer

Community Member on at

Like (0)

Report
It would be useful to know more information about your environment. Namely:

1. What version of CRM are you running?

2. Are you online or on-premise?

3. If you are on-premise do you have any load-balancing?

4. Do you have any synchronous plugins that fire when loading data?

To answer you question there is no supported way to import data that is faster than using best practices with the SDK. Best practices include:

a. Use ExecuteMultiple to process loading in batches

requestWithResults = new ExecuteMultipleRequest() { // Assign settings that define execution behavior: continue on error, return responses. Settings = new ExecuteMultipleSettings() { ContinueOnError = false, ReturnResponses = true }, // Create an empty organization request collection. Requests = new OrganizationRequestCollection() }; // Create several (local, in memory) entities in a collection. EntityCollection input = GetCollectionOfEntitiesToCreate(); // Add a CreateRequest for each entity to the request collection. foreach (var entity in input.Entities) { CreateRequest createRequest = new CreateRequest { Target = entity }; requestWithResults.Requests.Add(createRequest); } // Execute all the requests in the request collection using a single web method call. ExecuteMultipleResponse responseWithResults = (ExecuteMultipleResponse)_serviceProxy.Execute(requestWithResults);

b. Only update attributes to an entity that you want to update. This will reduce the data throughput and also may prevent plugins firing.

c. Try to ensure synchronous plugins are not firing. Check your plugins and deactivate them if not needed, or make them asynchronous if possible.

d. If you are using CRM 2015 UR1 Online or CRM 2016 use the Upsert message for a Create / Update in one operation.

e. If you are using CRM on premises implement Network Load Balancing (NLB) to your front end servers

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

Thanks for your help, Tom.

Answers:

1. Last version. 2016

2. Both versions. It depends on customer selection. (This is a solution for CRM, sold this way).

3. It depends on customer selection.

4. No plug-in for update data.

It is absolutely necessary to update daily 90% of the information (data bank products)

Thanks again.

Was this reply helpful? Yes No
Daniel Cai 935 on at

Like (1)

Report

We have a client using our software to push over 30 millions of records every night, it is done within their time window without affecting their operations during business hours. A couple millions of records is really not that difficult. You will just have to have a proper infrastructure, and you will try to utilize every single option to speed up the load.

Was this reply helpful? Yes No
alf 17,915 on at

Like (1)

Report

We use the KingswaySoft ssis adapter quite a bit. Many of our integration push MM of rows per night.

Was this reply helpful? Yes No
Daniel Cai 935 on at

Like (1)

Report

My suggestion is not to go the SDK route, as it is very costly to do so. At the end of day, you will likely end up with something that's difficult to maintain and there is no way to scale, if you choose to do so. Using a tool like ours, you have many ways to scale, all through visual configuration which requires no coding skills. It is more maintainable, more robust, and more cost effective, you get a significantly better development productivity, which supposedly will make your clients happier.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

Thanks to all !. I try your product tomorrow.

Was this reply helpful? Yes No
Daniel Cai 935 on at

Like (0)

Report

Sounds a good plan. In case you need help, you can talk to one of our partners, such as PowerObjects, who has been successful in helping many enterprise clients. Also, you can talk to us directly if you have a technical question related to our product.

Hope this helps.

Was this reply helpful? Yes No