web
You’re offline. This is a read only version of the page.
close
Skip to main content
News and Announcements icon
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Finance | Project Operations, Human Resources, ... / Data Lake - Exporting ...
Finance | Project Operations, Human Resources, ...
Suggested answer

Data Lake - Exporting custom entities

(0) ShareShare
ReportReport
Posted on by AshB 20

Good morning,

 Nor sure if this question is a D365FO or a Azure Data Lake question so apologies if in the wrong place.

Within our test environment we have successfully set up a Gen2 Data lake and I have managed to get 2 entities that I created (which we are already pusihing to a BYOD via the data management) to also start pushing the data into the DL:

pastedimage1623146303609v1.png

Now I understand when the data is in the DL it is stored as CDM e.g.:

pastedimage1623146362629v2.png

But what I am trying to understand is if I wanted to then say have exactly the same entity push from the DL into a DW or report is that easily done or do I need to somehow recreate the entity structure in the DL ?

Any help is appreciated.

Ash

I have the same question (0)
  • SaxMan Profile Picture
    SaxMan 15 on at
    RE: Data Lake - Exporting custom entities

    This is my dilemma as well.  The issue is this is CDM in datalake where the metadata is in different Json files that describe what's in the csv files via the common model manifest.  The PowerBI adapter knows how to navigate this model to access the csv files.  We need the same thing for SSIS.  Datalake adapters won't know about CDM metadata.  You can enable polybase via the whole Synapse view capability and there's a github application you can install to generate those views for you.  Then it's just matter of reading those views via SSIS - hopefully.  That's probably the route we will try.  

  • JAlegre_FF21 Profile Picture
    JAlegre_FF21 5 on at
    RE: Data Lake - Exporting custom entities

    Is Azure Synapse the only way? these are only csv files. isn't there a way to use SSIS and get these csv into a SQL Server database for instance?

  • Suggested answer
    Ajit Profile Picture
    Ajit 8,755 on at
    RE: Data Lake - Exporting custom entities

    You would have to create a view same as entity using base tables you exported in data lake and you can use Azure Synapse and create views there.

  • AshB Profile Picture
    AshB 20 on at
    RE: Data Lake - Exporting custom entities

    Hi,

    Thanks for that info, but my question still is , if I have entity X in D365FO, I select all its underlying tables, etc to go into the DL, how do I then get entity X from out of the data from the DL? Later on we may not need it like that but for now we want to see a mirror of entity extraction via DL

    Ash

  • WillWU Profile Picture
    WillWU 22,359 on at
    RE: Data Lake - Exporting custom entities

    Hi AshB,

    Data Lake is designed to store large amounts of data: hundreds of terabytes (TB) or more. It takes advantage of Azure Blob storage, which is inexpensive storage technology that many underlying services of Azure use.

    Data lakes are designed for big data analytics. You can bring your historical data and earlier data from your own systems into Data Lake. This data can consist of scanned documents in addition to business data. You can then apply machine learning models to the documents to make sense of the content and the opinions that the document authors expressed. You can also collect signals from devices and vehicles or machines on the shop floor, and store the data in data lakes. You can then apply machine learning models to detect anomalies and patterns in the signals, and you can join the results with business data to take proactive action.

    Data Lake has many associated services that enable analytics, data transformation, and the application of AI and machine learning. Azure services from Microsoft, partners, and open-source tools can be used to reason over the data.

    Instead of downloading Finance and Operations data from a data lake into your on-premises data warehouses, you can bring your on-premises data into a data lake. Microsoft refers to this transformation as modernization of data warehouses.

    Refer to:docs.microsoft.com/.../azure-data-lake-overview

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

News and Announcements

Dynamics 365 Community Calls Return
Join experts in person to explore low code and AI agents at the Power Platform Community Conference sponsored by Microsoft
Responsible AI policies for the Community

Quick Links

Responsible AI policies

As AI tools become more common, we’re introducing a Responsible AI Use…

Andrés Arias – Community Spotlight

We are honored to recognize Andrés Arias as our Community Spotlight honoree for…

Congratulations to the August Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > Finance | Project Operations, Human Resources, AX, GP, SL

#1
Sohaib Cheema Profile Picture

Sohaib Cheema 775 User Group Leader

#2
Martin Dráb Profile Picture

Martin Dráb 634 Most Valuable Professional

#3
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 622 Super User 2025 Season 2

Last 30 days Overall leaderboard

Featured topics

End-of-life Announcement for Microsoft Dynamics GP

Product updates

Dynamics 365 release plans