65
Hi,
We have got D365 CE data landed in Data Lake.
Now, I need to pull data to Azure SQL DB from DL. As suggested by MS, I am using ADF pipeline.
My pipeline is working fine in full load every time.
I need to run incremental load. How do I get latest rows in the entity(created/modified/deleted) like modifiedon.
Is it possible?
If I am able to get these new/changed rows, I can configure my pipeline to just import changes instead of all rows every time.
Thanks in advance
You will need to add a process called a "Watermark" to lookup or hold the last modified date and then you use this in your select statement in your pipeline. If you google Data Factory using Watermarks there are some very good tutorials.
#1
André Arnaud de Cal...
291,431
Super User 2024 Season 2
#2
Martin Dráb
230,503
Most Valuable Professional
#3
nmaenpaa
101,156
Share
Report
All responses (
Answers (
