Allow for XML schema drift in ADF Copy Data activity (from Fetch XML)

(0) Share

Report

Posted on by kade.hyland

I have observed the same issue as reported in .
Essentially, when you have a column that is sparsely populated (mostly contains NULLs) from a FetchXML query, the column may not be returned at all by your ADF Copy Data action. I believe this occurs because fetchXML does not return the attribute tag in the output when the value for the record is null.

This problem can be resolved by providing a fixed mapping to ADF. However, my issue is that I'd like to allow for schema drift (e.g. new field added to the source table in Fetch XML) when copying data in ADF and I wondered if there is an alternative solution that would address this requirement?

I have the same question (0)

All responses (2)

Answers (1)

kade.hyland 5 on at

Like (0)

Report

RE: Allow for XML schema drift in ADF Copy Data activity (from Fetch XML)

Thanks for the detailed response Daniele. I'll have to do some research to determine the best way to collect the metadata for the mapping, but the process you describe makes sense to me and I really appreciate it.

Was this reply helpful? Yes No
Verified answer

DAnny3211 11,376 on at

Like (1)

Report

RE: Allow for XML schema drift in ADF Copy Data activity (from Fetch XML)

hi

One possible solution to address this requirement is to use a dynamic mapping approach in ADF. This involves using a combination of dynamic content and metadata-driven mappings to handle schema drift.

In this approach, you can use dynamic content to extract the schema information from the source dataset and use metadata-driven mappings to dynamically map the columns based on their names and data types. This way, if a new field is added to the source table in FetchXML, it will automatically be included in the mapping without the need for manual updates.

To implement this approach, you can use the dynamic content feature in ADF to extract the schema information from the source dataset. You can then use this information to dynamically generate the mapping structure using metadata-driven mappings. This can be done using an ADF Data Flow with a metadata-driven mapping transformation, which can dynamically map the columns based on their names and data types.

Overall, this approach can provide a flexible and scalable solution to handle schema drift in FetchXML queries when copying data in ADF.

DAniele

Was this reply helpful? Yes No