Skip to main content

Notifications

Customer experience | Sales, Customer Insights,...
Answered

Ingest data from Dataverse - several issues

(0) ShareShare
ReportReport
Posted on by 45

Hi all,

we are building a Customer Insights system with customer data mainly coming from Dataverse. We face several issues when using the import method "Dataverse".

Has anybody observed similar issues or is there something wrong with our CI instance? Are these bugs or considered "normal" behaviour?

1. Dataverse data can not be filtered or transformed in any way and is always loaded "full table" with all columns

This leads to loads of data in CI that are not needed and just swamp th system. Even worse, it makes the unification process very difficult. E.g. how can I ignore/filter deactivated D365 contacts during M3?).

What is the recommended best practice to transforem Dataverse data? The PowerQuery Dataverse connector is gone, should we use PowerQuery oData instead (I hope not)?

2. Dataverse "connector" cannot cope with linebreaks

It creates many entries in the contact_Corrupt table. These are most probably caused by linebreaks in text fiels that are not properly handeled by CSV conversion when data travels through the internal MS data lakes.

How can these be fixed? Do we need to ask the data owner to clean up their linebreaks in Dynacics 365? They most probably will not care because it "works for them".

Again this would not be a big issue if there was a practical way to transform Dataverse data before ingesting into CI (see 1).

3. Several (custom) Dataverse tables missing from selection

Several (but not all) custom objects/tables from Dynamics 365 are not displayed for selection. However, these tables are visible and accessible though other connectors (e.g. Power Query from Power BI) - but not in Customer Insight. So we can not use this data at all (!?!)

4. Cannot remove Dateverse tables from selection after initial load

Once a Dataverse table has been selected and is loaded, it can not be removed/deleted anymore (selection checkbox is greyed out) - even if no further processing of the table has ever been configured (M3, Relationships etc.). Is this a bug or a feature?

4. Dataverse refresh/update duration

Updates fo Dataverse data can take very long, especially adding new tables within the connection can take hours. 

  • FrancisRomstad Profile Picture
    FrancisRomstad 205 on at
    RE: Ingest data from Dataverse - several issues

    Great it makes sense.

    Please mark as solution. It will make it easier for others to find this solution.

    -Francis

  • Suggested answer
    Johannes Wagner Profile Picture
    Johannes Wagner 45 on at
    RE: Ingest data from Dataverse - several issues

    Hi Francis, wow - this connector must have been added just yesterday as it was not there before.

    Looks like this s exaclty what I need. Thanks a bunch!

  • Verified answer
    FrancisRomstad Profile Picture
    FrancisRomstad 205 on at
    RE: Ingest data from Dataverse - several issues

    Hi Johannes,

    Check out my last post on this question.

    community.dynamics.com/.../not-able-to-import-dynamics-crm-contacts-into-ci

    This is how transformation is done on a dataverse source using the new power platform data flow.

    -Francis

  • Johannes Wagner Profile Picture
    Johannes Wagner 45 on at
    RE: Ingest data from Dataverse - several issues

    Thanks John for this super-fast reply!

    How can we cope with Dataverse records (e.g. D365 Contacts) that should not be used in CI (e.g. deactivated or assigned to a business unit/region/division/etc. not relevant for CI).

    How can we avoid these become part of our unified "golden" customer data set?

    I am currently not aware of any filter option during the MMM process.

    Can you please advise?

  • Suggested answer
    John Wiese Profile Picture
    John Wiese on at
    RE: Ingest data from Dataverse - several issues

    Thanks for the questions, let me see if I can help with answers.

    1. Yes, this is by design and has to do with what is happening behind the scenes.  When you use the Import (Power Query) option we actually pull a copy of the data into our data storage on refresh, to keep this as minimal/fast as possible we allow you to transform the data.  When using the Dataverse option we do not pull a copy of the data.  We will use the settings in Map to determine which fields we need to pull through and use in the unification/M3 process.  So, what you should do when using the Dataverse option is to limit the fields in Map to only the ones you need in the unified profile.

    2. Yes, this is a known issue and one we are working to resolve.  For now the best way to handle this is to not bring those fields into the process and use the individual fields instead.

    3.  Not sure on this, but I believe there is a setting when you create these custom objects that exposes them.

    4. yes, this is by design and has no effect on anything (see #1 response)

    5. This again depends on the size of what you are bringing in and the size of your dataverse.  We are working on trying to bring this time down.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Announcing Our 2025 Season 1 Super Users!

A new season of Super Users has arrived, and we are so grateful for the daily…

Vahid Ghafarpour – Community Spotlight

We are excited to recognize Vahid Ghafarpour as our February 2025 Community…

Congratulations to the January Top 10 leaders!

Check out the January community rock stars...

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 292,031 Super User 2025 Season 1

#2
Martin Dráb Profile Picture

Martin Dráb 230,868 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans