Skip to main content

Notifications

Announcements

No record found.

Microsoft Dynamics 365 | Integration, Dataverse...
Answered

Migrating dynamics CRM SSIS data integration to dynamics 365

(0) ShareShare
ReportReport
Posted on by 30

My company has been using dynamics CRM on premise for several years. We are, of course, now migrating to the 365 platform. 

I am not an office 365 or dynamics 365 specialist. I am a back end developer who specialises in SQL Server and C#. I have been given this task purely because I am the most experienced developer in my organisation and the consultants initially asked to provide a quote for the work quoted something on the order of a 6-9 month contract, which our CIO could not accept as being reasonable. 

I am thoroughly confused by the evolution of the dynamics 365 platform that seems to have happened over the past 1-2 years. As a developer, the mechanism for data integration with our on premise dynamics CRM was clear: Data in the CRM system could be read via the Filtered Views in the CRM database. Write operations, on the other hand, went through the XRM SDK, which exposes the CRM data as entities whose data can be manipulated and then written back via service endpoints, ensuring that any business logic written into the application for those entities was executed. So, databases and web services. Simple.

The equivalent application surface area now seems almost deliberately obtuse, possibly due to the evolution into CDM/CDS and now dataverse. In particular, the overwhelming majority of documentation is very much sales-focused-jargon (telling me all about how it is now easy for "citizen developers" or "makers" to build productive business focused value-adding secure scalable blah blah blah), without giving me any actual technical details. This is further complicated by the emergence of "dataverse for teams" which - in true microsoft form - has made specific information extremely difficult to find because of the naming collisions one gets out of a google search, and again, the "documentation" focused on trying to persuade decision makers to move to the platform rather than providing specific information relevant to an actual developer.

As you can tell, I am becoming more and more frustrated by microsoft's opacity when it comes to their azure products.

So, my specific question is this: Suppose I have an existing integration between some application which is NOT in any way related to the dynamics 365 suite of applications, and a "legacy" dynamics CRM installation. This integration synchronises data via a SQL Server integration services package, which reads from some source, and uses a script task to write that data into dynamics CRM via the XRM SDK. Another solution moves data in the opposite direction, reading from Filtered Views in the CRM database, and sending that data out to some unrelated application. These SSIS packages run via SQL Agent job schedules. For the record, each of these integration solutions individually took one to two hours to develop, just to demonstrate the ease and simplicity of the "old" way of doing things.

What is the appropriate architectural pattern, and set of technologies, that one should use to replace the above solution if the dynamics CRM platform has been replaced with a dynamics 365 product? If the answer is to use OData via dataverse, what are the licensing implications? Is the "user account" which connects to dataverse in order to read or write the data considered a licensed user, with an associated cost of, IIRC, about 30 US dollars per month? 

Further to this, I am currently watching this video https://www.youtube.com/watch?v=N73riCrC4ws which indicates that dataverse is "not populated automatically" - so I gather that dataverse is *not* merely a layer of indirection that sits on top of our "CRM" data and provides various mechanisms of access. It is apparently an independent repository that must be populated by us in some way. So, if that's the case, how would I access our "CRM" data without first spinning up a dataverse model and wiring that up to our actual data? Or is this not even possible? Is our own data effectively hidden from us *unless* we go through the additional expense, in time and licensing, of creating a dataverse and populating it?

  • allmhuran Profile Picture
    allmhuran 30 on at
    RE: Migrating dynamics CRM SSIS data integration to dynamics 365

    In our existing (on premise) architecture we use the XRM SDK to write to CRM via a script task in SSIS.

    In the new architecture we are using the XRM SDK to both write to, and read from, CRM, via script tasks in SSIS.

    I am using the Change Tracking feature of CRM and RetrieveEntityChangeRequest instead of SQL Change Data Capture when reading.

    I make my requests via the (newish) CrmServiceClient, which (at least according to the documentation) automatically honours the service protection limits so you don't have to write your own custom code to do that.

    CrmServiceClient documentation is here: docs.microsoft.com/.../microsoft.xrm.tooling.connector.crmserviceclient

  • samalex1701 Profile Picture
    samalex1701 258 on at
    RE: Migrating dynamics CRM SSIS data integration to dynamics 365

    We're about to head down this same road - going from on prem 8.2 to online 9.2 later this year.  Did you find ways to query your data directly through SSIS or other means?  Also I hoped to use the Dataverse also to query our data, but sounds like this integration won't be as seamless as I hoped.  

  • Thomas Rath Profile Picture
    Thomas Rath 665 on at
    RE: Migrating dynamics CRM SSIS data integration to dynamics 365

    You properly already solved you issue. But I just want to mention that you might have looked at the kingswaysoft connector for Dynamics 365 CE/CDS/Dataverse. That has been the defacto standard for how to use SSIS in relation to making integration/migration to/from Dateverse.

  • allmhuran Profile Picture
    allmhuran 30 on at
    RE: Migrating dynamics CRM SSIS data integration to dynamics 365

    Excellent answer, thanks for taking the time!

  • Verified answer
    Wahaj Rashid Profile Picture
    Wahaj Rashid 11,319 on at
    RE: Migrating dynamics CRM SSIS data integration to dynamics 365

    Hi,

    Thank you for your queries.

    First, Dynamics 365 Customer Engagement is an extension to Dataverse. By default, Dataverse comes with limited set of entities, when you buy a Sales Professional or Enterprise license, you get an enhanced version of Datavserse environment (with more entities, processes and features).

    Now, to make it simple, we can still use old methods to Read/Write Dynamics 365 CE (CRM) data:

    • You can use SDK to Retrieve data same way (using Organization Service and Query Expression/Fetch Expression).
    • Web API (OData), please note Web API was released before Dataverse.

    So to make it simple, you can still use your SSIS packages to read/write data using SDK or Web API, you might need to update the assembly versions in your packages and use CrmSvcClient to call Organization service methods.

    However, because it is online and we do not have access to DB, Filtered View won't be accessible. Instead of using Filtered Views, you can use Web API (OData Queries) or SDK (with Query Expression or Fetch XML) to read data.

    Yes, there are new ways to write your integration logics, specifically using Power Automate/Microsoft Flow. You can write flows to read data from Dynamics 365 CE (Common Data Service Connector) and write to another system using a connector or web service. Flow licensing has different models, you can read about it here:

    Power Apps and Power Automate licensing FAQs - Power Platform | Microsoft Docs

    Tip #1276: Do my Dynamics 365 users need a Microsoft Flow license? | Power Platform & Dynamics CRM Tip Of The Day

    If your data flow moves a large dataset, using SSIS packages is still a good idea. Furthermore, it would be an easier transition (with less effort) if you first upgrade your SSIS packages and move to Flows (if applicable).

    You can register a trial instance, and start exploring SDK and Web API.

    Use Postman with Dynamics 365 Customer Engagement Web API (Developer Guide for Dynamics 365 Customer Engagement) | Microsoft Docs

    Set up a Postman environment (Developer Guide for Dynamics 365 Customer Engagement) | Microsoft Docs

    Query Data using the Web API (Microsoft Dataverse) - Power Apps | Microsoft Docs

    Query Data sample (C#) (Microsoft Dataverse) - Power Apps | Microsoft Docs

    For SDK, you need to first get an app id and token to access the Dynamics 365:

    Tutorial: Register an app with Azure Active Directory (Microsoft Dataverse) - Power Apps | Microsoft Docs

    Quickstart: Organization service sample (C#) (Microsoft Dataverse) - Power Apps | Microsoft Docs

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

December Spotlight Star - Muhammad Affan

Congratulations to a top community star!

Top 10 leaders for November!

Congratulations to our November super stars!

Tips for Writing Effective Verified Answers

Best practices for providing successful forum answers ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,391 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,445 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Product updates

Dynamics 365 release plans