Imagine you have to update/enrich CRM data with a very large volume of data on a daily basis.
I am interested on how other companies deal with this requirement?
When I talk from large data updates I do mean like 100.000 to 500.000 (half million).
(making Excel import/exports not an acceptable solution)
First idea was to leverage it with a staging database hosted in Azure.
A custom job will extract the data via custom Web API queries and upsert it to the staging db.
Then some logic will transform, analyse and modify or enrich the data.
Another job would write the data back. All asynchronously.
Data changes from CRM could also be synced to that staging DB with Azure-aware plugins.
What tools/technologies would you use or suggest to accomplish this scenario?
