Share
Report
7Hi. I've been using the solution described in the blog post "ADF Template to Copy Dataverse data to Azure SQL - Part 1" since August 2022 when it was first posted. The solution isn't nearly as good as the previous data export service, but it works, for the most part. There are a few problems I've been having though.
I have the Dataverse incremental export set up to export hourly. This is mostly fine as the pipeline takes 10-20 minutes on average to run. The problem is in the rare cases where the pipeline takes over an hour, it automatically "skips" the next update. When this happens, the update is never retried, and any data that changed in the skipped update will not be synced unless the record happens to be changed again in the future.
The other problem is that, for whatever reason, every so often the pipeline runs in a "queued" status for hours on end, causing all other updates during that time to be skipped. In order to process those updates, I can either:
1. Disable the trigger on the pipeline, delete the "skipped" entries in the processing log, and rerun each pipeline one at a time until I've caught up.
2. Ignore the skipped files, remove some of the tables from the dataverse sync, and add them back in to ensure that every record is included in the next update.
Both these solutions require a bit of manual work, so I was wondering if there was a better way. I don't know if it's possible with Synapse, but I'm envisioning some kind of extra check at the end of the "DataverseToSQL_Orchestrator" pipeline that looks at the processing log to see if any updates were skipped while the current one was running, then automatically triggers a re-run of the earliest one as soon as the current update has finished.
There was one other note I had and that is: At the end of the original article "ADF Template to Copy Dataverse data to Azure SQL – Part 1" there was a note that said:
"More details on the ADF template FAQs & Best Practices will be shared in part 2."
Has that been abandoned? I've been keeping an eye out for part 2 since August last year, but haven't seen any mention of it.
Regards,
Jeff.
All responses (
Answers (
