Skip to main content

Notifications

Customer Service forum
Unanswered

Load data from DataLake to Azure SQL DB

Posted on by 65

Hi,

We have got D365 CE data landed in Data Lake.

Now, I need to pull data to Azure SQL DB from DL. As suggested by MS, I am using ADF pipeline.

My pipeline is working fine in full load every time.

I need to run incremental load. How do I get latest rows in the entity(created/modified/deleted) like modifiedon. 

Is it possible?

If I am able to get these new/changed rows, I can configure my pipeline to just import changes instead of all rows every time.

Thanks in advance

  • Tex_01 Profile Picture
    Tex_01 50 on at
    RE: Load data from DataLake to Azure SQL DB

    You will need to add a process called a "Watermark" to lookup or hold the last modified date and then you use this in your select statement in your pipeline.  If you google Data Factory using Watermarks there are some very good tutorials.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

November Spotlight Star - Khushbu Rajvi

Congratulations to a top community star!

Forum Structure Changes Coming on 11/8!

In our never-ending quest to help the Dynamics 365 Community members get answers faster …

Dynamics 365 Community Platform update – Oct 28

Welcome to the next edition of the Community Platform Update. This is a status …

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 290,867 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 229,173 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans