Skip to main content

Notifications

Customer experience | Sales, Customer Insights,...
Unanswered

Data ingestion issue: "Completed with Warnings", but no error

(0) ShareShare
ReportReport
Posted on by 87

Hi,

Data ingestion was not completed and showed the following error message under "System > Data Preparation > Entity"

pastedimage1642697553728v1.png

Usually, the corrupted records happened due to data type issue (e.g. datetime, nummerical, etc.). However, in my case, there are only 2 data types which are ingested (string and number).

Looking into "Entities > User > Entity", I went through all fields that have number (also String!) data type, but none of them have any error (only missing records). An example:

pastedimage1642697728585v2.png

What could be other reasons that CI could not process these records, therefore labels them as "corrupted" data?

Thank you,

  • rdj Profile Picture
    rdj 87 on at
    RE: Data ingestion issue: "Completed with Warnings", but no error

    Thanks Scott,

    1. Is that your total data set (~7M)?

    correct

    2. Has this ever ingested cleanly?

    Eventually I changed all data types into "String" and it worked for now, because I don't need to utilize the main purpose of using different data types. But it would be great to understand the underlying problem eventually.

    3. What changed between runs?

    I think there was a correlation between the number of records of "non-string" data types with the number of corrupted/error records. But it was hard to pin point exactly which fields that are problematic.

    4. Can you create a test data source with a small number (~100) records, to test?

    Yes, this is a good idea. Will do when I need to use the data types (e.g. birthday for measures).

    5. Did your schema in your input data source change since you originally ingested this data?

    Yes, it changed several times. See my answer to Q2 and Q3

    Thanks. When I have the time, I'd carry out your suggestion on Q4.

  • Paulien Lb Profile Picture
    Paulien Lb 125 on at
    RE: Data ingestion issue: "Completed with Warnings", but no error

    hi rdj,

    which connector are you using? And how is your date-field formatted?

    I've experienced using yyyy/MM/dd HH:mm:ss solved my corrupt records when using CDM-connector.

  • RE: Data ingestion issue: "Completed with Warnings", but no error

    Gotcha.

    It looks like its reporting 100% of records are corrupt (2971761 out of 6971761 corrupt records)
    Is that your total data set (~7M)?

    Has this ever ingested cleanly?
    What changed between runs? 

    Can you create a test data source with a small number (~100) records, to test?  Did your schema in your input data source change since you originally ingested this data?

  • rdj Profile Picture
    rdj 87 on at
    RE: Data ingestion issue: "Completed with Warnings", but no error

    Hi Scott,

    Thanks for your reply.

    I did. In fact I've been fixing previous issues by looking into the corrupt entities (e.g. replacing \n in the data that CI interprets as a new line - you see it in the entity name "no newline").

    But this time, the data in the last corrupt entity looks correct.

    Anywhere else I could check?

  • RE: Data ingestion issue: "Completed with Warnings", but no error

    Hi RDJ

    Did you click Entities, and look for the "KDB_merged_comma_with_datetime_no_newline_nocol1_corrupted" ?

    This should have an indication of what went wrong in your imported data.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Announcing Our 2025 Season 1 Super Users!

A new season of Super Users has arrived, and we are so grateful for the daily…

Vahid Ghafarpour – Community Spotlight

We are excited to recognize Vahid Ghafarpour as our February 2025 Community…

Congratulations to the January Top 10 leaders!

Check out the January community rock stars...

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 292,031 Super User 2025 Season 1

#2
Martin Dráb Profile Picture

Martin Dráb 230,868 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans