Dynamics 365 Storage Spotlight: Customer Insights – Data
Overview
Storage Spotlight is a learning series crafted to give you specific insights about how various Dynamics 365 products and applications impact your Power Platform storage consumption and capacity planning needs. In this edition, let’s examine Dynamics 365’s customer data platform (CDP), Customer Insights – Data.
Dynamics 365 Customer Insights - Data is designed to facilitate the data analytics needed to deliver intelligent, tailored customer experiences. It provides a comprehensive view of customers by unifying data from various sources in real-time. The platform empowers marketing, sales, and service professionals with actionable insights through native integrations with both Microsoft and third-party applications, providing 360-degree customer views by connecting transactional, behavioral, and demographic data. That data unification, coupled with AI/ML model-based insights, presents an unprecedented view of customer bases, allowing businesses to meet their customers’ desires and ultimately drive meaningful customer engagement.With all that underlying source data and valuable output data at play, you might be wondering how your Power Platform/Dataverse storage capacity gets impacted. Let’s start with some foundational information. At present (September 2024), per the Dynamics 365 licensing guide, when you purchase Customer Insights – Data, that license grants you the following additional storage entitlements at no extra charge: - 25GB of Dataverse Database storage capacity
- 40GB of Dataverse File storage capacity
- 4GB of Dataverse Log storage capacity
Traditionally, Dataverse storage capacity entitlements are granted from individual user licenses or Dataverse capacity add-on purchases (on top of inherent base storage entitlements granted). With Customer Insights – Data being an inherently data-centric platform, these additional storage entitlements and their respective quantities granted by the CI-D license begin to paint the picture of how Customer Insights – Data manages and allocates your customer data.Managed Data Lake vs. Azure Data Lake Storage Gen2 Customer Insights – Data is a unique application in the Dynamics 365 suite, in that you get some options in terms of how and where you can store aspects of the data involved with it. When you first setup Customer Insights – Data in an environment (note that you can now set it up in as many environments as you would like in your tenant with the new license model) you’re prompted to either store ingested data (and some output data) in Dataverse or you can bring your own Azure Data Lake Storage Gen2 (ADLS) storage account and have the platform use that instead. If you elect to have Customer Insights – Data use Dataverse, behind the scenes that means a Dataverse Managed Data Lake will be used, which falls squarely within the File storage allocation bucket for your environment.
This explains why customers get 40GB of Dataverse File storage capacity entitlements when purchasing Customer Insights – Data. That 40GB is meant to hold the Managed Data Lake’s data for your tenant. For most customers, this is more than sufficient. However, there are rare use cases where this entitlement may not cover the storage needs at play. If customers are ingesting very heavy amounts of data through Power Query, or there are many environments being used for separate Customer Insights – Data needs, each with heavier amounts of ingested Power Query data, then it may be a better option to bring your own ADLS Gen2 account instead. At scale, ADLS Gen2 storage pricing can be more cost effective than having to purchase extra Dataverse File storage if the 40GB entitlement will be exceeded significantly. Additionally, bringing your own Data Lake can be useful for your own specific data isolation or segmentation needs, depending on your specific requirements. To reiterate, however, most customers should be able to safely choose to store their Customer Insights – Data information within the Dataverse Managed Data Lake with the provided 40GB license entitlement, which is why Microsoft provides that entitlement and quantity.
Dataverse Database and Log Storage Considerations
As noted, when customers purchase a Customer Insights – Data license for their tenant, they are granted 25GB of Dataverse database storage capacity entitlements. This is a very important storage entitlement, since database storage is both one of the most critical aspects of Power Platform tenant storage, and the most expensive storage allocation to purchase if additional entitlements are needed.
Customer Insights – Data will store some of the output data within Dataverse Database tables. This includes unified customer profiles, supporting data, and measures created as tables so the customers can leverage this data within the Dataverse platform with more traditional needs like Views, table-level platform automation via Power Automate and other automation tools, and relationships to other Dataverse tables. Like the File entitlement allocations, the 25GB of Dataverse database storage should be sufficient for most customers’ output data. Exceptions to this could be if customers have incredibly high volumes of ingested data that results in heavy amounts of unified customer profile data, or for tenants where multiple instances of Customer Insights – Data are being used in production contexts, but that is not the norm that Microsoft sees.
As a reminder, Customer Insights – Data also grants 4GB of Dataverse log storage entitlements for the licensed tenant. This is an important entitlement, as most log storage entitlements above the base 2GB tenant allotment come from distinct log storage add-on purchases. Auditing is not on for the Dataverse tables that get created from Customer Insights – Data, so this entitlement grant is meant to facilitate any additional audit log consumption from usage of this data with other business processes for existing data commonly enabled for auditing like contacts, accounts, etc.
Guidance and Recommendations
While the storage entitlements for Customer Insights – Data licensing are usually enough for most customers, it is important to consider the following guidelines to most efficiently and effectively manage your storage for your needs.
For data ingestion, consider how much data you’ll be needing to ingest via Power Query-based sources, thereby storing that data within your Dataverse Managed Data Lake and consuming File capacity for the tenant. If you already have data stored in infrastructure sources that are accessible by Customer Insights – Data today, don’t duplicate storage of that data by bringing it into the Dataverse Managed Data Lake – simply use that data as a source instead. Power Query-based ingestion of data should be the pathway used when the source data is in a repository or format that Customer Insights – Data cannot currently connect to as a source. If you have a heavy amount of this kind of data, then it can make more sense to bring your own ADLS Gen2 storage account and use that as the Customer Insights – Data repository instead of the Dataverse Managed Data Lake. Philosophically, that’s not so different from a customer loading all this data into another storage repository in a supported format like Delta Lake and connecting Customer Insights – Data to it as a data source, whether they use the Managed Data Lake or ADLS Gen2. You get the 40GB storage entitlement regardless, so use it as you see fit with knowledge of your data estate. From a database storage perspective, there is not a lot to be done to optimize storage with specific regard to Customer Insights – Data output tables noted earlier. These tables are orchestrated by the platform and are there for your consumption as needed, but should not be manipulated or managed with features like Long Term Retention, bulk deletion, etc. without specific guidance from Microsoft to do so. One aspect of control you do have is with regard to customer measures created as Dataverse tables. These tables are linked to the customer profile table and can be used for specific customer journey orchestration and other in-platform needs. Some good examples of measures as a Dataverse table are briefly listed here, and similar scenarios can be considered, but only when there is a clear distinct need for usage of that data within Dataverse concepts like Customer Journeys or other Dataverse-based automations.
Regarding log storage, the usual principals for log storage considerations apply here as well. Do not audit tables that you don’t have specifically known audit log needs for. Similarly, do not enable auditing on columns that you do not have known audit log needs for within tables you need auditing enabled. Customer Insights – Data tables within Dataverse should not be audited as a general principal, as the data in these tables is managed by Customer Insights – Data automations and job runs, which can lead to significantly high record change volumes being recorded within the audit log.
Summary
Customer Insights – Data, as the name suggests, is an inherently data-centric platform. The biggest takeaways are:
- A Customer Insights – Data license for a given tenant grants 40GB of Dataverse File capacity, 25GB of Dataverse Database capacity, and 4GB of Dataverse Log capacity. These allocations are for the tenant, regardless of how much of those allocations Customer Insights – Data itself consumes, and regardless of how many instances of Customer Insights – Data are installed.
- While 40GB of Dataverse File capacity is granted, customers can elect to use their own Azure Data Lake Storage Gen2 account to hold the ingested/output data that would normally occupy File capacity storage.
- There is also output data that will reside in Dataverse tables themselves and thus occupy Dataverse Database capacity. It is important not to apply traditional storage consumption mitigation strategies against these tables as they are managed by the platform. Features like Long Term Retention, Bulk Delete, and custom offloading strategies should not be applied to these tables.
- Avoid turning on auditing for Customer Insights – Data Dataverse tables as these are system controlled, and transaction volumes can be high.
