Hi,
We have got D365 CE data landed in Data Lake.
Now, I need to pull data to Azure SQL DB from DL. As suggested by MS, I am using ADF pipeline.
My pipeline is working fine in full load every time.
I need to run incremental load. How do I get latest rows in the entity(created/modified/deleted) like modifiedon.
Is it possible?
If I am able to get these new/changed rows, I can configure my pipeline to just import changes instead of all rows every time.
Thanks in advance
You will need to add a process called a "Watermark" to lookup or hold the last modified date and then you use this in your select statement in your pipeline. If you google Data Factory using Watermarks there are some very good tutorials.
Stay up to date on forum activity by subscribing. You can also customize your in-app and email Notification settings across all subscriptions.
André Arnaud de Cal... 290,867 Super User 2024 Season 2
Martin Dráb 229,173 Most Valuable Professional
nmaenpaa 101,156