Best tools to bulk insert massive data into AX in "batch"?

(0) Share

Report

Posted on by Community Member

Hi experts,

I need suggestion, i have tried DIXF and AIF inbound with file system based, but with only small-medium data. My client now going to need me to load some quite massive data, so i'm thinking that i need a faster and better tools to bulk insert data in batch in AX, right? I haven't explore everything yet, because i have no time. So, i really need any suggestion from experienced one. Let's say this scenario :

1. I have to load massive data into existing tables of AX that has lots of AX business logic,

2. I have to load massive data into custom tables of AX.

What kind of tools should i use for these two scenario?

Thank you.

Regards,

Anisa.

*This post is locked for comments

I have the same question (0)

All responses (10)

Answers (2)

Suggested answer

Mea_ 60,284 on at

Like (0)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

Hi Anisa,

I think DIXF is a best option here. It supports multi threading (via batch) and could be quite robust.

Please refer ax.help.dynamics.com/.../migrating-data-using-the-data-importexport-framework-dixf-dmf for more details.

Also it is preferable way over AIF because its really easy to create new entities fro custom tables comparing to AIF.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

Hi Ievgen,

Have you had any experience with SSIS when loading massive data to AX? What do you think about it?

Was this reply helpful? Yes No
Mea_ 60,284 on at

Like (0)

Report
RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

It's a bad idea :)

DIXF uses SSIS under the hood. There are 2 stages:

From file\database to AX staging (done by pure SSIS)

From staging to Target (AX, could be done in multi threading via batch)

It is done this way because you need to invoke tons of business logic in AX and do bunch of different conversions and validation. It is just impossible without code that is written in AX in general case.

Each table in AX has system fields like RecId, DataAreaId, Partition and you definitely don't want to generate all of them from SQL.

Generally DIXF was created for data migration so I don't see any reason to invent your own bicycle :)

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

It makes sense. Okay, i know how to use DIXF, but i have no idea how to make multi-threading Batch, usually i use batch and add tasks. How can i make a multi-threading batch?

But before here is my requirement : client will put their .txt file in X file path, i need AX to read that file and insert it to AX everyday at 3 AM. Is this multi-threading batch will help?

Was this reply helpful? Yes No
Mea_ 60,284 on at

Like (0)

Report
RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

As per ax.help.dynamics.com/.../migrating-data-using-the-data-importexport-framework-dixf-dmf

Open Data import export framework > Common > Processing group. Select the processing group to work with.

Click Copy data to target. The Select a job ID to run dialog box opens.

Select a job. By default, the processing group is run for all rows. To limit the rows, in the Run for field, select Criteria, and then select which rows to run the job for. You can select Rows with previous errors, Rows selected by the user, or both.

Click OK. The Target data execution form opens.

To improve performance, consider setting the Number of batch tasks field. The total number of records is then split into the specified number of batch tasks, and those tasks are run at the same time, if resources are available.

To run the job immediately, click Run. To run the job as part of a batch, click Batch processing. The data is copied to the target entities.

As you can read in point number 5 you can specify number of tasks, e.g. 10 and it will split your data per 10 bundles and create batch task for each.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

What i want to know whether i can run the processing group every 3 AM. Or do i have to run it manually every 3 AM?

Was this reply helpful? Yes No
Verified answer

Mea_ 60,284 on at

Like (1)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

You can do setup recurrent execution of batch job with interval 1 day and start time 3 AM

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

Sorry to ask again, but what task (what class) should i added into the batch?

Was this reply helpful? Yes No
Verified answer

Mea_ 60,284 on at

Like (0)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

You are looking at wrong place :)

When you are doing "get staging data" there is an option to do it in batch and option to set recurrence.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

RE: ASK: Best tools to bulk insert massive data into AX in "batch"?

Sorry my bad. Thank you so much for the help!

Was this reply helpful? Yes No