Skip to main content

Notifications

Announcements

June 2024 Dynamics 365 Community Newsletter
Community forum and blog attachments are now available!
News and Announcements icon
Dynamics 365 Community / Forums / Dynamics 365 general forum / Data Export Services (...
Dynamics 365 general forum
Suggested answer

Data Export Services (DES) - Deprecation - Metadata Information

(0) ShareShare
ReportReport
Posted on by Shaun W Harvey 240

Hi,

We have been using Data Export Services (DES) to replicate Dataverse Data to an Azure SQL Database.

DES is being deprecated in November 2022 in favour of Azure Synapse Link For Dataverse.

Do more with data – From Data Export Service to Azure Synapse Link for Dataverse | Microsoft Power Apps

As well using the Azure SQL Database for reporting (as many other people probably do).

We also use the Azure SQL DB as part of a Caching Solution so as to reduce the number of API calls to the Dataverse.

Even with DES not everything that was needed was replicated down. So, we have a scheduled task which retrieves extra infomation (such as Global OptionSet Values) into a custom table we have created in the Azure SQL DB.

We have had a quick look at Azure Synapse Link and there does not seem to be the ability to create our own tables. Probably as they are Data Lake tables rather than an actual SQL instance.

With Azure Synapse Link, Is there a way to still get the data to replicate to an Azure SQL DB where we can easily add our own data if needed?

Best Regards,

Shaun Harvey

Categories:
Dynamics 365 Administration
Customization / Solutions / SDK and API
Other Functionality
  • Community member Profile Picture
    Community member 2 on at
    Data Export Services (DES) - Deprecation - Metadata Information
    What is the cost of the Power Hub solution many of you recommend?
  • Andreas Cieslik Profile Picture
    Andreas Cieslik 9,265 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    Hey there,

    I have already implemented Power Hub in my customers environment successfully.

    With PowerHub I am syncing 140+ tables. Changes in D365 are synced to the SQL DB in about 5-30secs.

    The environment is having about 4000 users from all around the world.

    PowerHub is highly scalable through up-scaling features that you can configure in your Azure resources, which makes it really easy to adjust performance base on the needs of your specific environment.

    If you have questions for technical details or on my experiences with it you can contact me directly (https://www.accloudsolutions.com)

  • Philip Bronsztejn Profile Picture
    Philip Bronsztejn 15 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    Hey everyone,

    If you're having issues with DES service, check out Power Hub. Our product streamlines the process and makes it faster and more efficient. As we have already solved this challenge for several customers, I am confident in its ability to deliver results. Feel free to reach out or check out our website to learn more.

    Best regards,

    Philip Bronsztejn

  • SBR_ Profile Picture
    SBR_ 5 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    We have been working with our CRM implementation partner to build an alternative for DES. Microsoft's offered solution isn't a solution, and Microsoft's conduct around this issue is unbecoming to a service provider.

    If you are looking to implement Synapse Link as your alternative, then be aware that data takes up to an hour to be synchronized. 

    If you are hoping for a reply from Microsoft's DES support email address, don't count on it.

    We give these companies way too much power over us.

  • Andreas Cieslik Profile Picture
    Andreas Cieslik 9,265 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    Hi,

    as you can see below Microsoft gives us a new grace period until April 30th 2023.

    2023_2D00_02_2D00_15-17_5F00_28_5F00_58_5F00_DES.png

  • seimor Profile Picture
    seimor 30 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    It's been almost three months since DES End-of-Life support from Microsoft. Despite successfully setting up Azure Synapse Link, I noticed that DES still exists in our tenant and is active in our CRM as it shows last synchronized (current date). Has this been the case with anyone? 

  • phuaccable Profile Picture
    phuaccable 15 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    Hi drissenb ,

    KingswaySoft just released version 22.1 few days ago that support download metadata, but I have not used the new features.

    My .NET console application download metadata, it has a client version timestamp parameter to full download or incremental download it. It only downloads those needed tables to have better performance. Then it saves metadata per table to my Token table as JSON data. I use a stored procedure in SQL Server database to process the JSON metadata - preserves it to AttributeMetadata table in SQL, identify changed tables. Based on the list of changed tables, we have to refresh KingswaySoft data flow (refresh CRM source, apply changes to SQL table, remap the new columns).

    There is not much you can get for deleted data - only GUID of the record. I store the data into a table and run SQL statement in SSIS component to remove it from database.

    Basically, my solution includes:

    1. .NET console app to download metadata and global option set (now KingswaySoft supports this) and store JSON metadata to SQL database.

    2. SQL stored procedure to process JSON metadata to keep track of history and identify changes.

    3. KingswaySoft (D365 CRM source and Premium ADO Net Destination components) to download changes (new/update/delete).

    Cheers

  • drissenb Profile Picture
    drissenb 130 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    Phu, could you say a bit more about your approach to tracking metadata changes?  We're just starting to evaluate KingswaySoft which looks promising but I'm concerned about catching metadata changes and having to update the SSIS packages manually.  I was also curious about your approach to deletes and am inclined to use the structure of the DeleteLog table in DES rather than what was suggested in the Kingsway blog.

  • phuaccable Profile Picture
    phuaccable 15 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    Hi,

    We have successfully deployed a solution that based on KingswaySoft products. It is quite straightforward:

    * We create SSIS packages that contain multiple sequence containers, one for each table that needs to be exported.

    * We have packages that run on every few minutes schedule, also have packages run hourly and on-demand.

    * Most of SSIS packages use KingswaySoft product to get data changes then upsert to Azure SQL database destination.

    * We could get the last token for each table, therefore our SSIS packages can continue upserting new changes to our database without the need to do full export initially.

    * We create a .NET console application to download metadata changes to identify metadata changes against our Azure SQL database destination in order to refresh our SSIS packages as needed.

    Hope this help.

    Cheers,

    Phu Nguyen   

  • Aaron Bennear Profile Picture
    Aaron Bennear 10 on at
    RE: Data Export Services (DES) - Deprecation - Metadata Information

    I can give some additional detail on the solution described by Jon Meyer.

    Here is an illustration of the solution:

    2022_2D00_10_2D00_14_5F00_15_2D00_07_2D00_13.png

    We have one pipeline per entity. Each pipeline is set to run only one instance at a time.

    File change events on the blob files are written to a storage queue. Then a function app on a timer looks for the file changes recorded in the queue and runs the corresponding pipelines.

    This solution has been working well in our testing over the past several weeks.

Helpful resources

News and Announcements

June 2024 Dynamics 365 Community Newsletter
Community forum and blog attachments are now available!

Quick Links

Community Spotlight of the Month

Kudos to Saurav Dhyani!

Congratulations to the June Top 10 community leaders!

These stars go above and beyond . . .

June 2024 Dynamics 365 Community Newsletter

Catch up on the latest news, events, and plans in the community

more Community News

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 285,407 Super User

#2
Martin Dráb Profile Picture

Martin Dráb 225,446 Super User

#3
nmaenpaa Profile Picture

nmaenpaa 101,148

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans