Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Dynamics 365 Community / Forums / Microsoft Dynamics CRM (Archived) / CRM Online mass data u...
Microsoft Dynamics CRM (Archived)

CRM Online mass data update or sync

(0) ShareShare
ReportReport
Posted on by Andreas Cieslik 9,265

Imagine you have to update/enrich CRM data with a very large volume of data on a daily basis.

I am interested on how other companies deal with this requirement?

When I talk from large data updates I do mean like 100.000 to 500.000 (half million).

(making Excel import/exports not an acceptable solution)

First idea was to leverage it with a staging database hosted in Azure.

A custom job will extract the data via custom Web API queries and upsert it to the staging db.

Then some logic will transform, analyse and modify or enrich the data.

Another job would write the data back. All asynchronously.

Data changes from CRM could also be synced to that staging DB with Azure-aware plugins.

What tools/technologies would you use or suggest to accomplish this scenario?

*This post is locked for comments

  • Suggested answer
    prt33k Profile Picture
    prt33k 6,907 on at
    RE: CRM Online mass data update or sync

    The best fit is SSIS here. You can use KingswaySoft adapter to help in data extraction and loading.

    The another option is Scribe - they are pretty good too.

    The custom development can also happen in SSIS if you use Script task - you need to write c# code to handle scenarios.

    The other option will be to go with Azure Web Job. These are just console app hosted on Azure and scheduled using azure scheduler.

    Now coming to viability :

    SSIS requires a VM - so if you have a VM and SQL server with proper hardware then SSIS is the way to go. You can choose custom development of KingswaySoft based on licensing and preferences.

    If you dont have azure VM but have money :) then Scribe is the good option. The licensing cost is there however it very user friendly.

    If you have developers with you and Azure subscription - then go with Azure jobs. You might need to put quite a lot of effort since everything need to be done within .NET framework - but that is also a viable option.

  • David Jennaway Profile Picture
    David Jennaway 14,063 on at
    RE: CRM Online mass data update or sync

    One option is to use the Data Export Service to get the data from CRM into the Azure DB. As for writing back to CRM, then the main performance benefit will be to be able to use ExecuteMultiple to minimise call latency.

    The Data Export Service uses change tracking. One thing you may need to consider is if there's the possibility of conflicting changes in CRM and the staging data. If so, you may find that you don't get as much control as you'd like over any resolution process if you use the Data Export Service

  • ashlega Profile Picture
    ashlega 34,475 on at
    RE: CRM Online mass data update or sync

    Hi Andreas,

     I used to do that on-prem (admittedly, no need for Azure  DB there), but it would be exactly like you described:

    - There would be a local staging DB

    - An SSIS job would pull data from the external system into that DB

    - Then it would pull updates from CRM into that DB

    - It would run a query to "transform" the data

    - And, finally, it would send that data back to CRM

    500000 is not that much and can probably be processed in a few hours, but that's unless you have synchronous workflows/plugins which would kick in and slow down that process to a crawl.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

December Spotlight Star - Muhammad Affan

Congratulations to a top community star!

Top 10 leaders for November!

Congratulations to our November super stars!

Tips for Writing Effective Suggested Answers

Best practices for providing successful forum answers ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,280 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,214 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans