Data Transformation in CI for Parquet formatted files

(0) Share

Report

Posted on by MT_IGD

Hi ,

We have ingested Parquet formatted files into Customer Insights using the CI recommended Incremental refresh folder structure. All this works fine. Unfortunately we are unable to do any transformations within the Dynamics 365 Customer Insights upon this data. Is this normal, I would have expected to be able to do some transformation and data type conversions, what am I missing ?

Any help greatly appreciated!

Many thanks,

Mahesh

All responses (3)

Answers (1)

Verified answer

Bill Ryan 803 on at

Like (1)

Report

RE: Data Transformation in CI for Parquet formatted files

Or Power Query - SQL has a pretty rich set of transforms but tbh, I'd probably opt for PQ in general over SQL. They both get you there though.
MT_IGD 27 on at

Like (1)

Report

RE: Data Transformation in CI for Parquet formatted files

Hi Bill,

Thanks for quick reply! It seems that, for transformation, we need to take synapse route then.

Thanks,

Mahesh
Suggested answer

Bill Ryan 803 on at

Like (1)

Report

RE: Data Transformation in CI for Parquet formatted files

Mahesh, if you're doing the CDM mount, the transformations will need to be done in advance. Power Query is the only mechanism right now that'll let you do transformations on ingestion. One other possible way around it is to setup a synapse serverless pool, create views on those parquet files and you can do several of the transforms in the view definition, but if you're already this far down the path it's probably not an option.