My company has been using dynamics CRM on premise for several years. We are, of course, now migrating to the 365 platform.
I am not an office 365 or dynamics 365 specialist. I am a back end developer who specialises in SQL Server and C#. I have been given this task purely because I am the most experienced developer in my organisation and the consultants initially asked to provide a quote for the work quoted something on the order of a 6-9 month contract, which our CIO could not accept as being reasonable.
I am thoroughly confused by the evolution of the dynamics 365 platform that seems to have happened over the past 1-2 years. As a developer, the mechanism for data integration with our on premise dynamics CRM was clear: Data in the CRM system could be read via the Filtered Views in the CRM database. Write operations, on the other hand, went through the XRM SDK, which exposes the CRM data as entities whose data can be manipulated and then written back via service endpoints, ensuring that any business logic written into the application for those entities was executed. So, databases and web services. Simple.
The equivalent application surface area now seems almost deliberately obtuse, possibly due to the evolution into CDM/CDS and now dataverse. In particular, the overwhelming majority of documentation is very much sales-focused-jargon (telling me all about how it is now easy for "citizen developers" or "makers" to build productive business focused value-adding secure scalable blah blah blah), without giving me any actual technical details. This is further complicated by the emergence of "dataverse for teams" which - in true microsoft form - has made specific information extremely difficult to find because of the naming collisions one gets out of a google search, and again, the "documentation" focused on trying to persuade decision makers to move to the platform rather than providing specific information relevant to an actual developer.
As you can tell, I am becoming more and more frustrated by microsoft's opacity when it comes to their azure products.
So, my specific question is this: Suppose I have an existing integration between some application which is NOT in any way related to the dynamics 365 suite of applications, and a "legacy" dynamics CRM installation. This integration synchronises data via a SQL Server integration services package, which reads from some source, and uses a script task to write that data into dynamics CRM via the XRM SDK. Another solution moves data in the opposite direction, reading from Filtered Views in the CRM database, and sending that data out to some unrelated application. These SSIS packages run via SQL Agent job schedules. For the record, each of these integration solutions individually took one to two hours to develop, just to demonstrate the ease and simplicity of the "old" way of doing things.
What is the appropriate architectural pattern, and set of technologies, that one should use to replace the above solution if the dynamics CRM platform has been replaced with a dynamics 365 product? If the answer is to use OData via dataverse, what are the licensing implications? Is the "user account" which connects to dataverse in order to read or write the data considered a licensed user, with an associated cost of, IIRC, about 30 US dollars per month?
Further to this, I am currently watching this video https://www.youtube.com/watch?v=N73riCrC4ws which indicates that dataverse is "not populated automatically" - so I gather that dataverse is *not* merely a layer of indirection that sits on top of our "CRM" data and provides various mechanisms of access. It is apparently an independent repository that must be populated by us in some way. So, if that's the case, how would I access our "CRM" data without first spinning up a dataverse model and wiring that up to our actual data? Or is this not even possible? Is our own data effectively hidden from us *unless* we go through the additional expense, in time and licensing, of creating a dataverse and populating it?
In our existing (on premise) architecture we use the XRM SDK to write to CRM via a script task in SSIS.
In the new architecture we are using the XRM SDK to both write to, and read from, CRM, via script tasks in SSIS.
I am using the Change Tracking feature of CRM and RetrieveEntityChangeRequest instead of SQL Change Data Capture when reading.
I make my requests via the (newish) CrmServiceClient, which (at least according to the documentation) automatically honours the service protection limits so you don't have to write your own custom code to do that.
CrmServiceClient documentation is here: docs.microsoft.com/.../microsoft.xrm.tooling.connector.crmserviceclient
We're about to head down this same road - going from on prem 8.2 to online 9.2 later this year. Did you find ways to query your data directly through SSIS or other means? Also I hoped to use the Dataverse also to query our data, but sounds like this integration won't be as seamless as I hoped.
You properly already solved you issue. But I just want to mention that you might have looked at the kingswaysoft connector for Dynamics 365 CE/CDS/Dataverse. That has been the defacto standard for how to use SSIS in relation to making integration/migration to/from Dateverse.
Excellent answer, thanks for taking the time!
Hi,
Thank you for your queries.
First, Dynamics 365 Customer Engagement is an extension to Dataverse. By default, Dataverse comes with limited set of entities, when you buy a Sales Professional or Enterprise license, you get an enhanced version of Datavserse environment (with more entities, processes and features).
Now, to make it simple, we can still use old methods to Read/Write Dynamics 365 CE (CRM) data:
So to make it simple, you can still use your SSIS packages to read/write data using SDK or Web API, you might need to update the assembly versions in your packages and use CrmSvcClient to call Organization service methods.
However, because it is online and we do not have access to DB, Filtered View won't be accessible. Instead of using Filtered Views, you can use Web API (OData Queries) or SDK (with Query Expression or Fetch XML) to read data.
Yes, there are new ways to write your integration logics, specifically using Power Automate/Microsoft Flow. You can write flows to read data from Dynamics 365 CE (Common Data Service Connector) and write to another system using a connector or web service. Flow licensing has different models, you can read about it here:
Power Apps and Power Automate licensing FAQs - Power Platform | Microsoft Docs
If your data flow moves a large dataset, using SSIS packages is still a good idea. Furthermore, it would be an easier transition (with less effort) if you first upgrade your SSIS packages and move to Flows (if applicable).
You can register a trial instance, and start exploring SDK and Web API.
Set up a Postman environment (Developer Guide for Dynamics 365 Customer Engagement) | Microsoft Docs
Query Data using the Web API (Microsoft Dataverse) - Power Apps | Microsoft Docs
Query Data sample (C#) (Microsoft Dataverse) - Power Apps | Microsoft Docs
For SDK, you need to first get an app id and token to access the Dynamics 365:
Quickstart: Organization service sample (C#) (Microsoft Dataverse) - Power Apps | Microsoft Docs
Stay up to date on forum activity by subscribing. You can also customize your in-app and email Notification settings across all subscriptions.
André Arnaud de Cal... 291,151 Super User 2024 Season 2
Martin Dráb 229,993 Most Valuable Professional
nmaenpaa 101,156