Data Import Error (schema mismatch)

(0) Share

Report

Posted on by Johannes Wagner

Hi Community,

We are loading several tables form a Azure data lake Gen2 (CDM folders). The system is in productive use since a few months where we built several (complex) measures and segments.

Recently we recently realized import errors on several tables where the error message says:

Error: Number of columns in source data file does not match the number of attributes: {AttributeCount} defined in the source data schema for the entity '[table name]'

(The data still seems to load correctly)

When I try to review/edit the data source, CI tells me I have to first remove the table from all relationships/measures/segements etc. This would be a massive task and cause a downtime for several days (basically we would have to "rebuild" almost everything).

How can I find out which columns are causing the issue? Is there any other way to analyze this and "reset" the data schema within CI and fix this?

All responses (3)

Answers (0)

Mukesh Pohuja on at

Like (0)

Report

RE: Data Import Error (schema mismatch)

can you reach out to CIHelp@microsoft.com for investigation?
Johannes Wagner 45 on at

Like (0)

Report

RE: Data Import Error (schema mismatch)

Hi Mukesh, thanks for your suggestions

1) No. All ADLS refreshes are complete when CI refreh startes

2) Yes. There are issues with line breaks that cause "corrupt" records (this is yet another problem)

3) Yes. However, this occurs in different entities not related to the schema error
Mukesh Pohuja on at

Like (1)

Report

RE: Data Import Error (schema mismatch)

1. Were the files on the ADLS still in the middle of a refresh, before the Data source refresh started?

2. Any chance there are new lines in the data, thats throwing off the col counts.

3. Under entities, Is there a corrupt entity that you can check for data skew?