Batch Job Execution Parallelism

(0) Share

Report

Posted on by AngelosP

Hello Community,

I am trying to understand how the batch job is running against all available AOS instances, as the time of an export to BYOD was double in one week since 22/08.

Currently I have setup an export of entities to BYOD through data management and associate it with a recurrent job connected a specific batch group. That doesn't implies that will be run parallel, through the assigned AOS servers of that groups, because my batch job has one only task. Correct?

What are the steps that I should follow to get the cause of that? The batch job history clean-up runs once per week & the monitor tool of the environment doesn't seems promising to find the issue.

Is there any way to schedule for run in sequence? By manual this can be achieved from data management -> Select project -> Export Options (top menu) -> Export in Batch -> Parameters Run Sequentially =Yes

"To improve throughput and reduce overall execution time, you can define a batch job as many tasks and then use a batch server to run the tasks against all available AOS instances." MS Batch processing and batch servers

I have the same question (0)

All responses (9)

Answers (1)

Suggested answer

André Arnaud de Cal... 301,148 Super User 2025 Season 2 on at

Like (0)

Report

Hi Angelos,

During the batch execution, you can monitor what happens. Depending on your setup it will create one or more runtime batch tasks which you can also check via the batch history.

Can you tell how many and which entities you scheduled for BYOD? Are all tables part of one DMF project? Are you using an incremental export or a full push of the data? What was the execution time before and now? Have you also checked the number of records exported?

The Run sequentially setting is only used for inbound integrations.

Was this reply helpful? Yes No
AngelosP 5 on at

Like (0)

Report

Hi Ande,

Thank you for your response

At the moment there are 2 executing tasks. One with task description-class:[Integration activity batch task]-[SysIntegrationActivityBatchTask] and the 2nd [Integration activity batch]-[SysIntegrationActivityBatch]. Can I create manual more of those batch tasks, so I can increase the parallelism execution?

Below the answers,

- 169 entities are scheduled for BYOD

- My knowledge is limited so I don't know by DMF project what you mean.

- 5 on full load, 164 on incremental

- On 22/08/2021 04:10 -> 00:10:43

22/08/2021 04:30 -> 00:15:57

22/08/2021 05:03 -> 00:18:41

30/08/2021 08:45 -> 00:19:49

-No I haven't check the number, because overall I am expecting to be close to 10 min up to 15 min. If it was a comparison of 2-3 days I would agree with you, but now the comparison is for 7 days for a job that runs every 15 min. So I assume that the number of export is not that issue.

- I don't have the option to rebuild/reorganize indexes, right? Neither to restart the database, correct?

Also for the entities I can choose the Export in Batch and run sequential.

Was this reply helpful? Yes No
Suggested answer

Sheikh Sohail 6,125 on at

Like (0)

Report

Hi Angelos

Hope I will address your question properly.

Answer-1

For Export OOTB, Create an only signal task, Or In simple words, BYOD does not export the data in multiple threads.

Answer-2

There is an option of sequence in DMF while you are entering the Entity in the project, By default, project execution follows the sequence you define in the project.

Was this reply helpful? Yes No
AngelosP 5 on at

Like (0)

Report

Hi Sohail,

Thank you for your respond. Could you be so kind to please provide a link that clearly mention BYOD does not export the data in multiple threads (By threads you mean AOS servers I guess)?.

I need still to find the reason why that batch job in 7 days is executed on double processed time. Are there any steps that I could follow?

Was this reply helpful? Yes No
Suggested answer

Sheikh Sohail 6,125 on at

Like (0)

Report

Hi Angelos

One AOS on prod has 8 threads.

In the Microsoft documentation, you will find only "Parallel imports" and nothing is defined as "Parallel Export".

Microsoft documentation link:-

docs.microsoft.com/.../data-import-export-job

I didn't find Here is another community link:

community.dynamics.com/.../dixf---export-job-with-multi-threading

Was this reply helpful? Yes No
Verified answer

André Arnaud de Cal... 301,148 Super User 2025 Season 2 on at

Like (0)

Report

Hi Angelos,

When you published all your entities to your BYOD, you can create multiple export projects and using this approach have some parallel operation. In my demo environment, e.g. I have one recurring export job for vendor related entities and another for customer related entities. You can use the same approach. Instead of having one DMF export project, you can create multiple. Ensure you will do a calculation for available number of threads and then decide how many export projects you want to have in your environment. For each recurring integration, you will need to setup the batch job. Note that one entity per export project also doesn't make sense.

I do hope this will help you.

Was this reply helpful? Yes No
AngelosP 5 on at

Like (0)

Report

Hello Andre,

The calculation should be based on entity per thread, right? So for 68 threads and 169 entities, would it make sense to have 6 export projects.

Yes you helped me, I will point us an answer if someone has the same question.

Kind regards,

Was this reply helpful? Yes No
André Arnaud de Cal... 301,148 Super User 2025 Season 2 on at

Like (0)

Report

Hi AngelosP,

As mentioned above, one export project will use only one single thread. So, 6 export projects will utilize 6*2=12 threads (export and monitor batch job). However, it might be a good start to check if the performance will be acceptable. Note that utilizing too many threads might also cause issues when more batch jobs needs to be executed on the same time.

Was this reply helpful? Yes No
TheRealHansH 5 on at

Like (0)

Report

Stumbled on this thread when I had the same issue, however, on my dev export to byod ran in parallel (put the entities in project on a different execution unit) However on test it didn't. When comparing I saw the only difference was that in dev the project category was set to Project on the dmf export project. On the test it was set to integration. Switch to Project in the test environment and my export to byod started to ran in parallel...

Was this reply helpful? Yes No