DMF Import Validations

(3) Share

Report

Posted on by AF-15101330-0

Hi all,

When using data management framework to import data to a data entity, is it possible to apply an all-or-none type of validation so that if any of the records are invalid then you reject all of the records? My users would prefer if they attempt to import 100 records from an excel file using DMF, if one record has a validation error then none of the records should be created. I was reading the documentation about data entity validations here:

https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/validations-defaults-unmapped-fields

From this, it appears we just have field and record level validations and I didn't see any good examples of how to apply a transaction scope to all of the records being imported so that I could accept or reject all of them as a set. Is there any example in the system that creates a transaction scope around the process that copies the data to the target tables so that I can roll back all of the rows if any one of them are not valid? Or does this type of design go against the spirit or concept of data entities in D365? If we cannot do it at the time of writing to the target tables, is it possible to do this type of validation on the staging tables before any records begin writing to the target? I am struggling to find good examples of either of these approaches and from my experience it seems the staging is getting written to at the same time as the target records. Any examples or direction on a validation like this would be greatly appreciated! Or confirmation that this is not a good design pattern and that I need to adjust my user's expectations on this is also appreciated!

Categories:

Dynamics 365 Finance

I have the same question (1)

All responses (9)

Answers (4)

Sort by

Verified answer

Sohaib Cheema 49,668 Super User 2026 Season 1 on at

Like (1)

Report
Copy link

Link copied!

Hi @AF-15101330-0

it already does that, I have tested on the standard system using a simple entity named Customer Groups.

I added two records to the excel, 1 good(having primary key), second bad record (missing primary key)

As I tried executing my load, it goes to error which makes sense.

none of the two newly added rows came in the system

I have checked it with both (Import Now and Import via batch)

if you look at the back-end classes, those show the transaction scope as well all exception handling

1 people found this reply helpful.

Was this reply helpful? Yes No
AF-15101330-0 35 on at

Like (0)

Report
Copy link

Link copied!

Thank you, @Sohaib Cheema for taking the time to test it out and show me.

Do you know if it's possible to add custom business validations at this level? If I implement the ValidateWrite method in my entity with some silly code like shown below which won't allow lines with a line number that is a multiple of 3, I can see the first two lines are created and the 3rd line is not created and has a warning on it in the staging table

public boolean validateWrite()
{

boolean isValid = super();

if (this.LineNumber mod 3 == 0)
{
isValid = checkFailed('line number divisible by 3, this is illegal');
}

return isValid;
}

I also tried throwing an error in the validateWrite method as a test but I still see the same result. Do you know if there are any useful extension points where we can add our own custom validation and it will fail the entire batch of records? Thanks again for your help here!

Was this reply helpful? Yes No
Verified answer

André Arnaud de Cal... 305,299 Super User 2026 Season 1 on at

Like (1)

Report
Copy link

Link copied!

Hi,

The example from Sohaib shows what happens if there is an issue moving data from the source file to the staging table. In that case, a complete file can indeed be stopped for processing. The result is that zero records are inserted in staging.

In case the data is imported in staging, but has other validations issues when moving from the staging to the target table, then there is no option to roll-back all records. You can't control that with custom coding.

In case you need to have all or nothing, you can consider creating your own integration or check if the data integration solution from Staedean will fit your needs.

1 people found this reply helpful.

Was this reply helpful? Yes No
AF-15101330-0 35 on at

Like (0)

Report
Copy link

Link copied!

Thank you for the response @André Arnaud de Calavon! It makes perfect sense. Is there a documented place that is the correct place to add custom validations at the file level that would allow me to programmatically fail a file in the same way that Microsoft is when for example a file is missing a value for a primary key field as is shown in Sohaib's example?

Was this reply helpful? Yes No
Verified answer

André Arnaud de Cal... 305,299 Super User 2026 Season 1 on at

Like (1)

Report
Copy link

Link copied!

There is no documentation and this is not possible with an extension on the DMF framework. As mentioned, you can consider a custom integration, not using the DMF framework.

FYI, When copying data from the source to the staging, it is executing a SQL Server Integration Services (SSIS) script where it fails in case of data type errors, like no value for a mandatory field or a wrong date format. Copying data from staging to the target is row-by-row with X++ business logic.

2 people found this reply helpful.

Was this reply helpful? Yes No
AF-15101330-0 35 on at

Like (0)

Report
Copy link

Link copied!

Thanks again @André Arnaud de Calavon that makes sense and matches up with what I’m seeing in the debugger. One last idea, I understand that data entities have an option to operate row by row or set based. What if I went with the set-based approach on my custom entity? Is there any point that is a best practice to add a set-based validation in a custom data entity? Thanks again for all the help on this!

Was this reply helpful? Yes No
Verified answer

Sohaib Cheema 49,668 Super User 2026 Season 1 on at

Like (1)

Report
Copy link

Link copied!

code shows that there are not any set-based validations.

set based processing comes into effect after validation is done. and set based processing is only for performance reasons. That checkbox can help with performance but not with validations.

Note: not every entity can be enabled for set based processing https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/sysadmin/optimize-data-migration#enable-set-based-processing

2 people found this reply helpful.

Was this reply helpful? Yes No
AF-15101330-0 35 on at

Like (0)

Report
Copy link

Link copied!

Thanks again @Sohaib Cheema. I was hoping that maybe with set-based processing, I could implement the copyCustomStagingToTarget method and maybe at that place I could add validation and if any issue were found then I could throw an exception and keep records from going to target. Is this not possible or a bad practice to use the copyCustomStagingToTarget method in this manner?

Was this reply helpful? Yes No
Sohaib Cheema 49,668 Super User 2026 Season 1 on at

Like (0)

Report
Copy link

Link copied!

Hi @AF-15101330-0

I haven't looked at the aspect of this getting customized and whether it is correct place to write the code, the one that you are suggesting, therefore I would not say if that is possible or not. My views were on the standard application behaviour.

1 people found this reply helpful.

Was this reply helpful? Yes No