Skip to main content

Notifications

FastTrack for Dynamics 365 forum
Unanswered

ADF Template to Copy Dataverse data to Azure SQL - Part 2?

Posted on by 7

Hi. I've been using the solution described in the blog post "ADF Template to Copy Dataverse data to Azure SQL - Part 1" since August 2022 when it was first posted. The solution isn't nearly as good as the previous data export service, but it works, for the most part. There are a few problems I've been having though.

I have the Dataverse incremental export set up to export hourly. This is mostly fine as the pipeline takes 10-20 minutes on average to run. The problem is in the rare cases where the pipeline takes over an hour, it automatically "skips" the next update. When this happens, the update is never retried, and any data that changed in the skipped update will not be synced unless the record happens to be changed again in the future.

The other problem is that, for whatever reason, every so often the pipeline runs in a "queued" status for hours on end, causing all other updates during that time to be skipped. In order to process those updates, I can either:

1. Disable the trigger on the pipeline, delete the "skipped" entries in the processing log, and rerun each pipeline one at a time until I've caught up.

2. Ignore the skipped files, remove some of the tables from the dataverse sync, and add them back in to ensure that every record is included in the next update.

Both these solutions require a bit of manual work, so I was wondering if there was a better way. I don't know if it's possible with Synapse, but I'm envisioning some kind of extra check at the end of the "DataverseToSQL_Orchestrator" pipeline that looks at the processing log to see if any updates were skipped while the current one was running, then automatically triggers a re-run of the earliest one as soon as the current update has finished.

There was one other note I had and that is: At the end of the original article "ADF Template to Copy Dataverse data to Azure SQL – Part 1" there was a note that said:
"More details on the ADF template FAQs & Best Practices will be shared in part 2."

Has that been abandoned? I've been keeping an eye out for part 2 since August last year, but haven't seen any mention of it.

Regards,

Jeff.

  • Khan zain Profile Picture
    Khan zain 1 on at
    ADF Template to Copy Dataverse data to Azure SQL - Part 2?
    To address your issues:
    1. Handling Skipped Updates: Implement a mechanism to detect skipped updates in the processing log. Upon completion of a pipeline run, check for skipped updates and trigger a re-run of the earliest skipped update.
    2. Handling Queued Status: Improve pipeline performance or implement a mechanism to automatically resolve queued status. This could involve optimizing pipeline components or utilizing features like parallel execution to prevent bottlenecks.
    Regarding the missing "ADF Template to Copy Dataverse data to Azure SQL - Part 2" and its promised FAQs & Best Practices, unfortunately, if it hasn't been released yet, it's unclear whether it's still in development or has been abandoned. You might consider reaching out to the author or the platform where it was supposed to be published for updates.
  • Suggested answer
    ADF Template to Copy Dataverse data to Azure SQL - Part 2?
    Hey, I ran into this issue as well and my solution was to make the following changes.
     
    DataverseToSQL_Orchestrator
    1. Open Pipeline and click on whitespace to make sure no Activities are selected.
    2. Go to Settings and change Concurrency to 1
    This will ensure that if the pipeline is triggered while another is already running, it will enter a queued state rather than try to run, see that another is running and mark it as skipped.
  • Anisur Profile Picture
    Anisur 17 on at
    RE: ADF Template to Copy Dataverse data to Azure SQL - Part 2?

    Hi Jeff,

    I have the same problem. It is very frustrating !

Helpful resources

Quick Links

Can you answer this forum question?

You could make someone's day!

Community Newsletter - May 2024

Kudos to our community stars!

Community Spotlight of the Month

Kudos to Mohamed Amine Mahmoudi!

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 283,623 Super User

#2
Martin Dráb Profile Picture

Martin Dráb 224,463 Super User

#3
nmaenpaa Profile Picture

nmaenpaa 101,146

Featured topics

Product updates

Dynamics 365 release plans