Finance | Project Operations, Human Resources, ...

Answered

Transaction Scope Across Batch Tasks

(0) Share

Report

Posted on by Greg's Mom

Hello,

I have a question about batch parallelism and transaction scope. Is it possible to commit the changes made in a batch job with multiple asynchronous tasks only if there are no errors in any of the tasks and roll all changes back by the batch job run if any of the tasks contained an error?

For context, my company has a weekly batch job that performs posting of approx.400 project hours journals sequentially. This process is taking a long time and so I've created a proof of concept using the top picking technique described here:

https://learn.microsoft.com/en-us/archive/blogs/axperf/batch-parallelism-in-ax-part-iii

This is working great, I've cut the processing time significantly but my company has a requirement that if any error occurs on any of the journals, they would like all of the postings that occurred during the batch to roll back. This was easy when it was one big sequential process. Now that I've implemented this with multiple batch tasks, each journal is posting and committing individually and if there is an error, only that one journal fails to post and the rest are left posted.

I wasn't sure if there is a change to the way I am creating the batch tasks that would indicate to fail and roll back the entire batch job?

I have the same question (0)

All responses (5)

Answers (1)

Greg's Mom on at

Like (0)

Report

RE: Understanding Batch Transaction Scope

In case it helps, here's the controller for my batch job:

public class ABCPayrollPostController extends SysOperationServiceController
{
	List taskList;

	BatchGroupId                    groupId;
	SysRecurrenceData               recurrenceData;
	utcDateTime                     startDateTime;
}

public Batch doBatch()
{
	// add all needed tasks
	this.addAdditionalTasks();

	// add the normal info - the super() call is suppressed - the task is added in the method above
	info(strFmt("@SYS73254", this.caption()));

	return null;
}

public void addAdditionalTasks()
{
	// create the list of tasks to be added
	this.buildTaskBatchList();

	// add the tasks to the batch
	this.addTasksBatchHeader();
}

private void buildTaskBatchList()
{
	taskList = new List(Types::Class);

	// add the task 5x - hard coded for now, will parameterize later
	taskList.addEnd(ABCPayrollPostTaskController::construct());
	taskList.addEnd(ABCPayrollPostTaskController::construct());
	taskList.addEnd(ABCPayrollPostTaskController::construct());
	taskList.addEnd(ABCPayrollPostTaskController::construct());
	taskList.addEnd(ABCPayrollPostTaskController::construct());
}

private void addTasksBatchHeader()
{
	BatchHeader                     header;
	ListEnumerator                  taskListEnumerator;
	SysOperationServiceController   batchTask;
	SysOperationServiceController   prevBatchTask;
	BatchInfo                       taskBatchInfo;

	header = this.batchInfo().parmBatchHeader();

	if (recurrenceData != conNull())
	{
		header.parmRecurrenceData(recurrenceData);
	}

	if (!groupId)
	{
		groupId = this.batchInfo().parmGroupId();
	}

	if (header != null &&
		taskList != null)
	{
		taskListEnumerator = taskList.getEnumerator();
		taskListEnumerator.reset();

		while (taskListEnumerator.moveNext())
		{
			batchTask = taskListEnumerator.current();

			taskBatchInfo = batchTask.batchInfo();
			taskBatchInfo.parmGroupId(groupId);

			header.addTask(batchTask);

			prevBatchTask = batchTask;
		}

		header.parmCaption(this.caption());
		header.save();
	}
}

public static void main(Args args)
{
	ABCPayrollPostController::construct().startOperation();
}

public static ABCPayrollPostController construct()
{
	ABCPayrollPostController  serviceOperation;

	serviceOperation = new ABCPayrollPostController();

	serviceOperation.parmClassName(classStr(ABCPayrollPostController));
	serviceOperation.parmMethodName(methodStr(ABCPayrollPostController, runOp));
	serviceOperation.parmExecutionMode(SysOperationExecutionMode::Synchronous);

	serviceOperation.parmDialogCaption("Parallel Payroll Post");

	return serviceOperation;
}

public void setRecurrenceData(SysRecurrenceData _recurrenceData)
{
	recurrenceData = _recurrenceData;
	startDateTime = SysRecurrence::getRecurrenceStartDateTime(recurrenceData);
}

public void runOp(ABCPayrollPostDataContract _dataContract)
{
	ABCPayrollPostService::newFromDataContract(_dataContract).run();
}

BatchGroupId parmGroupId(BatchGroupId g = groupId)
{
	groupId = g;
	return groupId;
}

Here's the service code that each of the tasks runs. Ideally, I would like to get rid of the try/catch here and allow everything to roll back if any error occurred during the posting.

public void run()
{
    ABCPayrollImportJournals importJournals;
    JournalId currentJournalId;

    importJournals.readPast(true);

    do
    {
        try 
        {        
            ttsBegin;
            select firstOnly pessimisticLock importJournals
            index hint ABCPayrollPostingStatusId
            where importJournals.ABCPayrollPostingStatusId == ABCPayrollPostingStatus::Ready;
                        
            info(strFmt("Batch %1 processing journal %2", processId, importJournals.JournalId));
            
            if (importJournals)
            {
                currentJournalId = importJournals.JournalId;                
                importJournals.ABCPayrollPostingStatusId = ABCPayrollPostingStatus::Processing;
                importJournals.update();
            }
            ttsCommit;
            
            ttsBegin;
       
			ABCPayrollCheckPost::PostJournal(importJournals, true, true);

            if(importJournals)
            {
                importJournals.ABCPayrollPostingStatusId = ABCPayrollPostingStatus::Posted;
                importJournals.update();
            }
			
            ttsCommit;
        }
        catch
        {
            if (importJournals && currentJournalId)
            {
                ttsBegin;
                select forUpdate importJournals
                where importJournals.JournalId == currentJournalId;
                if (importJournals)
                {
                    importJournals.ABCPayrollPostingStatusId = ABCPayrollPostingStatus::Error;
                    importJournals.update();
                }
                ttsCommit;
            }
            
        }
    }
    while(importJournals);
}

Was this reply helpful? Yes No

Greg's Mom on at

Like (0)

Report

RE: Understanding Batch Transaction Scope

Maybe I'm thinking about the problem incorrectly. Is there another way to perform asynchronous operations but keep them within the same database transaction scope? Any help to point to an example that exists in the system or documentation that explains how to do something like this would be greatly appreciated!

Was this reply helpful? Yes No
Greg's Mom on at

Like (0)

Report

RE: Transaction Scope Across Batch Tasks

After more research and review, I believe that it's not possible for the tasks in a batch job to share a transaction scope. I'm not sure if I'm right but I believe each batch task is probably running independently and has it's own transaction scope. I was hoping to identify some approach to creating the batch tasks or some configuration on the batch header that would allow me to roll back the entirety of the batch job if a single error occurred but it doesn't look like that is easily achievable. I really don't like this problem because I believe transaction scope should be kept smaller and not large like this. Each journal should post and then the system should move to the next. That is working great. Unfortunately, my business has what I believe to be a poor design in the journals that this batch job is posting where some of the journals are loosely related to other journals. They would like the posting of these journals to be all or none so that in the case of an error, they can delete all of the journals and reimport them if needed. I think in absence of a straight forward way to roll the entire batch back, I will investigate the problem up stream and look at how my business can react if errors do occur by either re-importing the journals that need to be re-imported or reversing the ones that posted with invalid values.

This whole topic did get my curiosity though and I'm still curious if anyone is familiar with a pattern that has been done in AX where processes are done in parallel and but the results of the process are committed all-or-none? I started down the path of a batch task control table that would maintain the status of the batch tasks and once the batch tasks were done with their work they would only commit their transaction if all other batch tasks also completed their work and would throw an error and abort their transaction if any batch task reported an error status. This design feels fragile and I'm not very comfortable with it. I worry about the possibility of infinite loops or deadlocking.

Was this reply helpful? Yes No
Verified answer

Martin Dráb 236,503 Most Valuable Professional on at

Like (1)

Report

RE: Transaction Scope Across Batch Tasks

If you want a transaction, you must give up parallelization. If you run several batch tasks at once, they can be executed on different computers (using independent DB connections). Therefore "Transaction Scope Across Batch Tasks" can't be done, IMHO.

You'd need to approach your problem differently, ideally starting from the solution design. The idea of having separate journals that can be posted independently, but they shouldn't, looks strange to me. Also, I wouldn't merge data import and journal posting in a single step, if possible. If import succeeds, an unposted journal is saved. Then you try to post it, and if it fails, you can try it again (if it fails because of a DB error, for example), or you can correct data before subsequent postings. When you first import the data and only then run the posting, you can then identify related journals and post them together (e.g. creating a batch task with the "batch").

But of course, I'm just speculating without knowing the actual requirements.

Was this reply helpful? Yes No
Greg's Mom on at

Like (0)

Report

RE: Transaction Scope Across Batch Tasks

Thanks, Martin! I agree with what you've written here. We do have the import and posting process as separate batch jobs so we are good there but I believe I need to investigate more on the process that generates the journals and work with the business to create these journals that can be posted independently without being dependent on one another. Either that or give the business a convenient way of correcting whatever issues may be caused by posting some of the journals but not others.

Was this reply helpful? Yes No

Join the conversation

Community site session details

Transaction Scope Across Batch Tasks

Helpful resources

News and Announcements

Quick Links

Responsible AI policies

Abhilash Warrier – Community Spotlight

Congratulations to the September Top 10 Community Leaders!

Leaderboard > Finance | Project Operations, Human Resources, AX, GP, SL

Featured topics

Product updates

Community site session details

Transaction Scope Across Batch Tasks

Helpful resources

News and Announcements

Quick Links

Responsible AI policies

Abhilash Warrier – Community Spotlight

Congratulations to the September Top 10 Community Leaders!

Subscribe to this forum!

Select categories

Leaderboard > Finance | Project Operations, Human Resources, AX, GP, SL

Featured topics

Product updates