Hi all,
I'd like to better understand how does corrupt entities work and how does "Include all records" flag work in step Match in the Unify process.
About corrupt entities.
We did more than one data import form different data source and we can see corrupt entity for each import source.
In corrupt entity, we found some "strange" records:
- for some records data stored in the ID column does not match to an ID, but we find other data (please see in screenshot below, for example first record) and it's complicated identify which record affected.
- for other record, data stored in the ID column actually match IDs, but I don't understand how to identify which field generated the record log tracking.
About "Include All Records" flag, we did some documentation research and some tests but we cannot understand which is the difference usign it or not. Anyone has tested that flag and can help me please?
Thank you in advance.
Ok, I understand thank you, but if we use Power Query + Dataverse connector we run into a "Invalid credential" and we are unable to reach dataverse data. That's why we thought about Power-query + Dataverse data lake connector.
I hope I explained myself because my English is not very good... :)
thanks a lot and best regards.
No, I have replied on that thread as it is incorrect. The two import methods are in no way linked. What I was pointing out in my video is that you should use the Connect to Dataverse as that is the most performant, especially if you need to bring in a lot of entities and a lot of data...however, if you need an entity that has columns with invalid chars in it then you should not bring those in with that import method, you should use the Power Query + Dataverse connector so you can limit the columns and transform the data to remove the carriage returns.
I frequently use both import types in the projects I help partners/customers with, sometimes I only use Power Query, and sometimes only Connect to Dataverse.
Hi John,
thank you for you suggestion. We've seen your video and it is very useful, only the last question please: do you confirm that you use Dataverse data lake connector (as specified in this other thread: invalid-credentials---authentication-to-dataverse-connectors-power-query-import-mode)?
Thank you and best regards
With all the confusion around this I am going to post a video talking through this setup on YouTube as part of my CI In Under 5 series at https://aka.ms/CIUnder5
Should have the video up this morning with the details you can view there.
Seems like you are taking more steps than you need to. You can start from step 4 and use Power Query and the Dataverse connector, clean the data in Power Query, which is what it is meant for, and then use that output in your M3 process. This would then allow it to run without any manual steps.
Hi Francis
Thank you, your doubt is probably correct :)
Actually, the first time, I used Power Query with D365 contact records and the cleaned data I uploaded back to D365. According to what you say, I understand that I need to clean the data after it has been imported to Customer Insights.
So I will summarize the steps performed:
1) Opened Excel and selected the contacts table from the data source.
2) Clean data
3) Saving the cleaned data on excel file on my PC
4) Create a new Power Query data source to import the cleaned data on CI from excel.
5) Replacing the table in the 3M process
We are wondering how this process can be scheduled so that D365 data is loaded directly into CI without manual intervention.
Thank you very much and regards.
Hi
Just a thought.
Did you change the entity going into the 3M proces from the raw D365 CE (the one with new line characters) to the cleaned one?
Regards
Hi all
I have some trouble with Power Query.
Corrupt Entity and Power Query --> I used Power Query on Dyn365 contacts and (it would seem) that I removed all the new line characters. As a matter of fact the warning related to the new lines does not appear anymore after I imported the data on CI, the corrupt entity is "Not Avalilable" but the records are still not considered in the unification process.
Do you have any other suggestions please? Is there anything else I can check?
"Include All Records" Flag --> works fine as you explained
Thank you and best regards
Thank you to all for your answers.
Corrupt Entity --> ok thank you we'll try the Francis suggestion.
Include All Records-flag --> thank you, your explanations are clear but we did some tests as you described and it seems does not work as expected. Sure we'll double-check.
I'll let you know.
Best Regards
Here is a visual which may help. Not that the bottom shows which records end up in the unified profiles, the "include all records" ones will only show if checked.
André Arnaud de Cal...
291,979
Super User 2025 Season 1
Martin Dráb
230,848
Most Valuable Professional
nmaenpaa
101,156