Release Wave 2 – Let’s look at CDM and data integration highlights

In my previous article I shared some general information about Release Waves. How it all works, when they come, how can we prepare etc. I also highlighted some Power Apps features that come with 2020 Release Wave 2. The purpose of this article is to highlight a chapter of the Power Platform Release Plan that contains some Power Platform built-in features for data integration – the chapter with the heading Common Data Model and data integration.

Introduction to CDM and data integration

Let me begin with a reminder of what the Common Data Model (CDM) and the Common Data Service (CDS) are. I will just give you this wonderful quote again, from my MVP friend Jonas Rapp and then feel free to dig into the documentation that I just provided you with.

The CDM is like the map and the CDS is the landscape we walk on.

Jonas Rapp

The CDM and data integration section in the plan covers new and enhanced features for the following areas:

Dual Write – Dynamics 365 Finance and Supply Chain Management data to the CDS
Export to data lake – Data from the CDS to Azure Data Lake Gen 2
Power Platform Connectors – The plan covers mostly the certification process
Power Platform Dataflows – Load data from different sources to the CDS
Power Platform Gateway – Connect to on-premises sources
Power Query Online – For data preparation and transformation 

CDM and data integration – what’s new?

The highlights presented comes from the chapters about Export to Data Lake, Power Platform Connectors, Power Platform Dataflows and Power Query Online. Please note that here are more new features to read about in the plan, just a few are presented here!

There are a lot of features presented for Export to data lake. A few highlights are: Cross-tenant support for when exporting to data lake. Until this wave the Azure data lake gen 2 storage had to be in the same tenant as the CDS. Support for exporting entity Audit data to Azure data lake storage gen 2. Support for exporting attachments – support for the annotation entity. Support for soft delete – it will be possible to delete the data from the source but keep it in the data lake for analysis purpose.

For Power Platform Connectors you can read that partners creating and certifying connectors will get feedback for certified connectors, enhanced version management and we will get more open source connectors from Microsoft to build upon or contribute to. We find these on GitHub, take a look here! The plan also presents that we will get better guidance for when connectors or operation are deprecated, advice for what to use instead. Read more about the certification process here.

What is new for Power Platform Dataflows? First of all, do not forget to read the Power Query Online part – these new features will be applicable for Power Platform Dataflows as well. New administration capabilities are presented. Until now it has been the creator of a dataflow which has been able to see and edit the dataflow and we have had no possibility to change the owner of a dataflow. New administration capabilities are coming to us, we will be able to view all dataflows created in environments and also to reassign dataflows to new owners. The plan also mentions we will be able to “certify” dataflows. Exactly what that means is yet to be found out.

Power Platform Dataflows grow up into Azure Data Factory. We will get an extension possibility – take a Power Platform Dataflow and extend it into an Azure Data Factory Data Flow. This is presented as taking the Power Platform Dataflows from a Citizen developer approach to an ETL Pro developer approach with Azure Data Factory Data Flows.

Read more about Azure Data Factory in the Microsoft Docs and take a look at how to create an Azure Data Factory Data Flow here. You can also read the Azure Blog post announcing GA for ADF Mapping Data Flows here and you can find a lot of other great links here.

I am interested in hearing your thoughts about in what scenarios we might want to extend a Power Platform Dataflow into using Azure Data Factory Data Flows. Please let me know your thoughts!

Here follows some examples of what is new for Power Query Online. Certified connectors (developed by partners) is on the way to Power Query Online. Read more about the different connectors here. The possibility to use Custom Connectors will come to Power Query Online. If you are interested in how to get started with Custom Connectors for Power Query, take a look here. Please note, these are not the same as the connectors we work with when we use Canvas Apps and Power Automate flows.

As already mentioned, the features presented under Power Query Online is applicable also for Power Platform Dataflows. So we will get more possibilities to use new data sources, that is how I read the plan, since we will get certified connectors and Custom Connectors possibilities.

Templates are coming. Import/export Custom Dataflow templates and to define parameters in the templates. Reuse Dataflow definitions in different environments. This feature has been available in Power Query desktop, read about it here. Other interesting things you can read about is a future possibility to enrich data from data sources such as Azure Maps to get smart data transformation, e.g address geocoding.

One thing that crosses my mind when I go through the plan is that there is nothing in it about the “Data Integrator“. The Data Export Service is not mentioned either.

That was the highlights for the CDM and data integration section of the 2020 Release Wave 2 plan. I hope you are looking forward to the new features and can’t wait to try them out!