Load large amount of data to CRM, every day.

(1) Share

Report

Posted on by Community Member

Hi,

We need to load a lot of data daily to CRM. Using the SDK it becomes very slow. There any viable way to insert and update data quickly (and supported by Dynamics CRM)?.

Unfortunately much of the information changes, around 90% of the data is updated daily.

We appreciate any help!

*This post is locked for comments

I have the same question (0)

All responses (13)

Answers (0)

Elijah Stownton 192 on at

Like (0)

Report

RE: Load large amount of data to CRM, every day.

Hi Leonardo,

For fast and convenient uploading of large amounts of data it is recommended to use ETL tools.

They upload larger amounts of data within 1 API call that's why their speed is higher than the speed of SDK.

You can check the G2 top list of ETL tools to find the most suitable solution.

In our company we use Skyvia, because it can upload data automatically both from CSV format and relational databases.

Was this reply helpful? Yes No
Ashay Shingornikar 135 on at

Like (0)

Report

RE: Load large amount of data to CRM, every day.

Hi Leo,

did using Kingswaysoft ssis toolkit helped in resolving your issues?

where did you deployed your ssis package when your source was dynamics 365 / CRM online?

Thanks in Advance.

Ashay

Was this reply helpful? Yes No
Suggested answer

Community Member on at

Like (2)

Report

RE: Load large amount of data to CRM, every day.

Hi Leo,

Thanks for your response. That's good you are using CRM 2016 on-premise / online so you do have the options I described to improve things including: Upsert, ExecuteMultipleRequest and Network Load Balancing (NLB) of front end servers.

I do agree with Alex in his post above the using SSIS with parallelalism would also add extra benefits. SSIS Kingswaysoft plugin for Dynamics CRM has the advantage that it also supports Upsert and executing operations in batches.

Tom

Was this reply helpful? Yes No
Daniel Cai 935 on at

Like (0)

Report

RE: Load large amount of data to CRM, every day.

Sounds a good plan. In case you need help, you can talk to one of our partners, such as PowerObjects, who has been successful in helping many enterprise clients. Also, you can talk to us directly if you have a technical question related to our product.

Hope this helps.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

RE: Load large amount of data to CRM, every day.

Thanks to all !. I try your product tomorrow.

Was this reply helpful? Yes No
Daniel Cai 935 on at

Like (1)

Report

RE: Load large amount of data to CRM, every day.

My suggestion is not to go the SDK route, as it is very costly to do so. At the end of day, you will likely end up with something that's difficult to maintain and there is no way to scale, if you choose to do so. Using a tool like ours, you have many ways to scale, all through visual configuration which requires no coding skills. It is more maintainable, more robust, and more cost effective, you get a significantly better development productivity, which supposedly will make your clients happier.

Was this reply helpful? Yes No
alf 17,915 on at

Like (1)

Report

RE: Load large amount of data to CRM, every day.

We use the KingswaySoft ssis adapter quite a bit. Many of our integration push MM of rows per night.

Was this reply helpful? Yes No
Daniel Cai 935 on at

Like (1)

Report

RE: Load large amount of data to CRM, every day.

We have a client using our software to push over 30 millions of records every night, it is done within their time window without affecting their operations during business hours. A couple millions of records is really not that difficult. You will just have to have a proper infrastructure, and you will try to utilize every single option to speed up the load.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

RE: Load large amount of data to CRM, every day.

Thanks for your help, Tom.

Answers:

1. Last version. 2016

2. Both versions. It depends on customer selection. (This is a solution for CRM, sold this way).

3. It depends on customer selection.

4. No plug-in for update data.

It is absolutely necessary to update daily 90% of the information (data bank products)

Thanks again.

Was this reply helpful? Yes No
Suggested answer

Community Member on at

Like (0)

Report
RE: Load large amount of data to CRM, every day.

It would be useful to know more information about your environment. Namely:

1. What version of CRM are you running?

2. Are you online or on-premise?

3. If you are on-premise do you have any load-balancing?

4. Do you have any synchronous plugins that fire when loading data?

To answer you question there is no supported way to import data that is faster than using best practices with the SDK. Best practices include:

a. Use ExecuteMultiple to process loading in batches

requestWithResults = new ExecuteMultipleRequest() { // Assign settings that define execution behavior: continue on error, return responses. Settings = new ExecuteMultipleSettings() { ContinueOnError = false, ReturnResponses = true }, // Create an empty organization request collection. Requests = new OrganizationRequestCollection() }; // Create several (local, in memory) entities in a collection. EntityCollection input = GetCollectionOfEntitiesToCreate(); // Add a CreateRequest for each entity to the request collection. foreach (var entity in input.Entities) { CreateRequest createRequest = new CreateRequest { Target = entity }; requestWithResults.Requests.Add(createRequest); } // Execute all the requests in the request collection using a single web method call. ExecuteMultipleResponse responseWithResults = (ExecuteMultipleResponse)_serviceProxy.Execute(requestWithResults);

b. Only update attributes to an entity that you want to update. This will reduce the data throughput and also may prevent plugins firing.

c. Try to ensure synchronous plugins are not firing. Check your plugins and deactivate them if not needed, or make them asynchronous if possible.

d. If you are using CRM 2015 UR1 Online or CRM 2016 use the Upsert message for a Create / Update in one operation.

e. If you are using CRM on premises implement Network Load Balancing (NLB) to your front end servers

Was this reply helpful? Yes No