Skip to main content

Notifications

Customer experience | Sales, Customer Insights,...
Suggested answer

Corrupt entities for import data and "Include all" flag

(0) ShareShare
ReportReport
Posted on by 20

Hi all,
I'd like to better understand how does corrupt entities work and how does "Include all records" flag work in step Match in the Unify process.

About corrupt entities.
We did more than one data import form different data source and we can see corrupt entity for each import source.
In corrupt entity, we found some "strange" records:

- for some records data stored in the ID column does not match to an ID, but we find other data (please see in screenshot below, for example first record) and it's complicated identify which record affected.

- for other record, data stored in the ID column actually match IDs, but I don't understand how to identify which field generated the record log tracking.


About "Include All Records" flag, we did some documentation research and some tests but we cannot understand which is the difference usign it or not. Anyone has tested that flag and can help me please?

Thank you in advance.


MicrosoftTeams_2D00_image.png

  • Mrlyn2 Profile Picture
    Mrlyn2 20 on at
    RE: Corrupt entities for import data and "Include all" flag

    Ok, I understand thank you, but if we use Power Query + Dataverse connector we run into a "Invalid credential" and we are unable to reach dataverse data. That's why we thought about Power-query + Dataverse data lake connector.
    I hope I explained myself because my English is not very good... :) 

    thanks a lot and best regards.

  • John Wiese Profile Picture
    John Wiese on at
    RE: Corrupt entities for import data and "Include all" flag

    No, I have replied on that thread as it is incorrect.  The two import methods are in no way linked.  What I was pointing out in my video is that you should use the Connect to Dataverse as that is the most performant, especially if you need to bring in a lot of entities and a lot of data...however, if you need an entity that has columns with invalid chars in it then you should not bring those in with that import method, you should use the Power Query + Dataverse connector so you can limit the columns and transform the data to remove the carriage returns.  

    I frequently use both import types in the projects I help partners/customers with, sometimes I only use Power Query, and sometimes only Connect to Dataverse.

  • Mrlyn2 Profile Picture
    Mrlyn2 20 on at
    RE: Corrupt entities for import data and "Include all" flag

    Hi John,

    thank you for you suggestion. We've seen your video and it is very useful, only the last question please: do you confirm that you use Dataverse data lake connector (as specified in this other thread: invalid-credentials---authentication-to-dataverse-connectors-power-query-import-mode)?

    Thank you and best regards

  • Suggested answer
    John Wiese Profile Picture
    John Wiese on at
    RE: Corrupt entities for import data and "Include all" flag

    With all the confusion around this I am going to post a video talking through this setup on YouTube as part of my CI In Under 5 series at https://aka.ms/CIUnder5

    S
    hould have the video up this morning with the details you can view there.

  • Suggested answer
    John Wiese Profile Picture
    John Wiese on at
    RE: Corrupt entities for import data and "Include all" flag

    Seems like you are taking more steps than you need to.  You can start from step 4 and use Power Query and the Dataverse connector, clean the data in Power Query, which is what it is meant for, and then use that output in your M3 process.  This would then allow it to run without any manual steps.

  • Mrlyn2 Profile Picture
    Mrlyn2 20 on at
    RE: Corrupt entities for import data and "Include all" flag

    Hi Francis 

    Thank you, your doubt is probably correct :)

    Actually, the first time, I used Power Query with D365 contact records and the cleaned data I uploaded back to D365. According to what you say, I understand that I need to clean the data after it has been imported to Customer Insights.

    So I will summarize the steps performed:

    1) Opened Excel and selected the contacts table from the data source.

    2) Clean data

    3) Saving the cleaned data on excel file on my PC

    4) Create a new Power Query data source to import the cleaned data on CI from excel.

    5) Replacing the table in the 3M process

    We are wondering how this process can be scheduled so that D365 data is loaded directly into CI without manual intervention.

    Thank you very much and regards.

  • FrancisRomstad Profile Picture
    FrancisRomstad 205 on at
    RE: Corrupt entities for import data and "Include all" flag

    Hi

    Just a thought.

    Did you change the entity going into the 3M proces from the raw D365 CE (the one with new line characters) to the cleaned one?

    Regards

  • Mrlyn2 Profile Picture
    Mrlyn2 20 on at
    RE: Corrupt entities for import data and "Include all" flag

    Hi all

    I have some trouble with Power Query.

    Corrupt Entity and Power Query --> I used Power Query on Dyn365 contacts and (it would seem) that I removed all the new line characters. As a matter of fact the warning related to the new lines does not appear anymore after I imported the data on CI, the corrupt entity is "Not Avalilable" but the records are still not considered in the unification process.

    Do you have any other suggestions please? Is there anything else I can check?

    "Include All Records" Flag --> works fine as you explained

    Thank you and best regards

  • Mrlyn2 Profile Picture
    Mrlyn2 20 on at
    RE: Corrupt entities for import data and "Include all" flag

    Thank you to all for your answers.

    Corrupt Entity --> ok thank you we'll try the Francis suggestion.

    Include All Records-flag --> thank you, your explanations are clear but we did some tests as you described and it seems does not work as expected. Sure we'll double-check.

    I'll let you know.

    Best Regards

  • John Wiese Profile Picture
    John Wiese on at
    RE: Corrupt entities for import data and "Include all" flag

    Here is a visual which may help.  Not that the bottom shows which records end up in the unified profiles, the "include all records" ones will only show if checked.

    pastedimage1637773508671v1.png

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Announcing Our 2025 Season 1 Super Users!

A new season of Super Users has arrived, and we are so grateful for the daily…

Announcing Forum Attachment Improvements!

We're excited to announce that attachments for replies in forums and improved…

Vahid Ghafarpour – Community Spotlight

We are excited to recognize Vahid Ghafarpour as our February 2025 Community…

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,979 Super User 2025 Season 1

#2
Martin Dráb Profile Picture

Martin Dráb 230,848 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans