Hi all,
we are building a Customer Insights system with customer data mainly coming from Dataverse. We face several issues when using the import method "Dataverse".
Has anybody observed similar issues or is there something wrong with our CI instance? Are these bugs or considered "normal" behaviour?
1. Dataverse data can not be filtered or transformed in any way and is always loaded "full table" with all columns
This leads to loads of data in CI that are not needed and just swamp th system. Even worse, it makes the unification process very difficult. E.g. how can I ignore/filter deactivated D365 contacts during M3?).
What is the recommended best practice to transforem Dataverse data? The PowerQuery Dataverse connector is gone, should we use PowerQuery oData instead (I hope not)?
2. Dataverse "connector" cannot cope with linebreaks
It creates many entries in the contact_Corrupt table. These are most probably caused by linebreaks in text fiels that are not properly handeled by CSV conversion when data travels through the internal MS data lakes.
How can these be fixed? Do we need to ask the data owner to clean up their linebreaks in Dynacics 365? They most probably will not care because it "works for them".
Again this would not be a big issue if there was a practical way to transform Dataverse data before ingesting into CI (see 1).
3. Several (custom) Dataverse tables missing from selection
Several (but not all) custom objects/tables from Dynamics 365 are not displayed for selection. However, these tables are visible and accessible though other connectors (e.g. Power Query from Power BI) - but not in Customer Insight. So we can not use this data at all (!?!)
4. Cannot remove Dateverse tables from selection after initial load
Once a Dataverse table has been selected and is loaded, it can not be removed/deleted anymore (selection checkbox is greyed out) - even if no further processing of the table has ever been configured (M3, Relationships etc.). Is this a bug or a feature?
4. Dataverse refresh/update duration
Updates fo Dataverse data can take very long, especially adding new tables within the connection can take hours.
Great it makes sense.
Please mark as solution. It will make it easier for others to find this solution.
-Francis
Hi Francis, wow - this connector must have been added just yesterday as it was not there before.
Looks like this s exaclty what I need. Thanks a bunch!
Hi Johannes,
Check out my last post on this question.
community.dynamics.com/.../not-able-to-import-dynamics-crm-contacts-into-ci
This is how transformation is done on a dataverse source using the new power platform data flow.
-Francis
Thanks John for this super-fast reply!
How can we cope with Dataverse records (e.g. D365 Contacts) that should not be used in CI (e.g. deactivated or assigned to a business unit/region/division/etc. not relevant for CI).
How can we avoid these become part of our unified "golden" customer data set?
I am currently not aware of any filter option during the MMM process.
Can you please advise?
Thanks for the questions, let me see if I can help with answers.
1. Yes, this is by design and has to do with what is happening behind the scenes. When you use the Import (Power Query) option we actually pull a copy of the data into our data storage on refresh, to keep this as minimal/fast as possible we allow you to transform the data. When using the Dataverse option we do not pull a copy of the data. We will use the settings in Map to determine which fields we need to pull through and use in the unification/M3 process. So, what you should do when using the Dataverse option is to limit the fields in Map to only the ones you need in the unified profile.
2. Yes, this is a known issue and one we are working to resolve. For now the best way to handle this is to not bring those fields into the process and use the individual fields instead.
3. Not sure on this, but I believe there is a setting when you create these custom objects that exposes them.
4. yes, this is by design and has no effect on anything (see #1 response)
5. This again depends on the size of what you are bringing in and the size of your dataverse. We are working on trying to bring this time down.
André Arnaud de Cal...
292,031
Super User 2025 Season 1
Martin Dráb
230,868
Most Valuable Professional
nmaenpaa
101,156