Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Dynamics 365 Community / Forums / Microsoft Dynamics AX (Archived) / All batch jobs suddenl...
Microsoft Dynamics AX (Archived)

All batch jobs suddenly stopped exectuting

(1) ShareShare
ReportReport
Posted on by jorlille 5

Hi,

In one of our customers test environment, which is the environment they are working today since they have not gone live yet, we have a problem with batch jobs that we need to get resolved. They use a lot of batch jobs to import/export data and acceptance test is just a few weeks away.

 

Current version: AX 2012 R3 CU9

We are in the process of upgrading development environment to CU12, but not sure if we will updates test environment to CU12 before this issue is resolved since there are pre/post-synchronization scripts that needs to run and these are executed in batch.

 

On November 23, all batch jobs stopped running. The last job ran from 11:04:29 to 11:26:06 and the job that was supposed to start 11:26:38 (32 seconds later) did not start, and no jobs have started since.

6457.BatchOverview.png

 

There are no batch jobs or tasks with status “Executing”

All batch jobs have status “Waiting”.

Tasks are a mix of “Waiting” and “Ready”

 

The server configuration looks correct as far as I can tell.

0172.ServerConfig.png

 

I have tried to uncheck and recheck “is batch server” to see of that helps

I have tried to create new batch group which was used on new batch job, that did not execute either

I have tried to remove all batch groups from server instance in server configuration and later add them again

I have tried to restart the AOS service and also the server

I have done a full compilation, full CIL compilation, deleted XppIL and CIL compiled again (with restarts of AOS service in-between)

I have deleted history (170 000+ records for both Batch and BatchJobs) using BatchHistoryDelete

I have tried to set ServerId in table Batch on all records without ServerId using SQL

None of these attempts have solved the issue

 

I have checked remaining disk space on the SQL Server – Several GB left on all drives

I have checked disk space on the AOS (over 30 GB left)

I have checked remaining space in the log and data files on the SQL Server – There is room left and all files are set to auto grow

I have checked for locks in the database – and there are none.

I have tried to check the event viewer, but don't know what to look for...

Feel that I have checked all of the most common reasons and all the obvious problems, but hope someone has a suggestion that I have not thought of that can help me fix this issue.

Thanks for any input that can help resolve this issue

Best regards,

Jørgen Lillestøl

*This post is locked for comments

  • jorlille Profile Picture
    jorlille 5 on at
    RE: All batch jobs suddenly stopped exectuting

    I got to admit that I felt that many of the suggestions seemed like variations of what I had already tried and could not see any reason why they should help when what I had tried already had not fixed the issue - but that's AX for you.... Do the same thing enough times and suddenly it will start working without any clear reason.

    No matter the reason, the problem seems to have been resolved :)

    This evening I started with setting all batch jobs to Withold (luckily all I had to do was change the property of the menu item in the form to multi select, since the class for changing status was already ready for multi select)

    I then checked in SQL that all batch jobs and task had the correct status and that there were no hidden data causing problems

    Since I was in SQL, I also truncated  both SysBreakPoints and SysBreakPointList

    I then removed all batch groups from the server instance in Server Configuration and disabled "is batch server"

    I then restarted the service

    I then enabled "is batch server" on the instance and added all batch groups to this server instance, and restarted the service again

    I then change a couple of the batch jobs to "Waiting" and they actually started!

    As I wrote in the original message, I had tried most of this already - and could not see any reason why just changing status to withhold and back to waiting should do anything when newly created jobs did not start either. I can understand that it might help on old jobs, but when new jobs did not start either it looked like it must be a deeper issue.

    Vilmos:

    No modifications of any of the classes you mentioned.

    We have, luckily, quite few modifications in the solution - but, unluckily, a whole bunch of 3rd party solutions (but still, no modifications on these classes in any layers) that causes quite a lot of problems when we want to install CUs and hotfixes since they take forever to release new versions and seldom are able to do so before a new CU is already out...

    I restarted the whole server 2 days ago when I started to look at the issue, and VS would have been restarted also and we would not have any debuggers attached to the AOS process - We have not enabled remote debugging, so it would have to be on the AOS Server.

    No hard-coded breakpoints or Debug::Asserts either.

    I agree that service accounts should be used to execute a lot of the batch jobs for, for instance, importing/exporting data.

    I think Viesturs most likely had the solution (and most complete answer), so will mark his answer as "the answer"

    Thanks you all for your assistance

  • feelyd Profile Picture
    feelyd 260 on at
    RE: All batch jobs suddenly stopped exectuting

    Hi,

    We had the same issue previously and similar to your situation I tried a number of steps to try and resolve.  Nothing would work.

    The only solution we had was to delete the existing batch jobs and recreate a new one.

  • Verified answer
    Vilmos Kintera Profile Picture
    Vilmos Kintera 46,149 on at
    RE: All batch jobs suddenly stopped exectuting

    Please check if you have any customization on classes that start with Batch* and SysRecurrence*, I have a feeling that something is broken in there if you have customized it.

    Also debuggers i.e. attaching to the AX AOS process from Visual Studio could suspend executing batches or AIF, if a breakpoint or error is thrown. For tha stop your AX AOS instances and delete all your breakpoints in the SysBreakpoints and SysBreakpointList tables in SQL (call truncate on them), then start the AX AOS.

    Also check if you have any hard breakpoints with the "breakpoint;" text in critical classes like the above mentioned Batch*, or Info, or Application, since hard breakpoints or Debug::Assert calls are evil.

    Verify if the users with the batch execution have System User role (and other roles required to execute the batch process), and they are enabled in AX Users list. To be honest I would use a dedicated Service account for all scheduled batch jobs to avoid such problems of deactivating users.

  • Verified answer
    Community Member Profile Picture
    Community Member Microsoft Employee on at
    RE: All batch jobs suddenly stopped exectuting

    Hi Jorgen,

    I had similar issue not while ago on our test environment all the batch jobs suddenly stopped at status executing or waiting. The way I solved it: rebooted AOS, rechecked that each batch group has batch server selected (system administration - Setup - Batch groups). Correct AOS instance and batch server is selected (Server configuration). Updated recurrence for each batch job with status waiting. Cancelled the ones that stopped with status executing and recreate those.

    Hope that helps

    V

  • jorlille Profile Picture
    jorlille 5 on at
    RE: All batch jobs suddenly stopped exectuting

    Thanks for you answer Crispin,

    By "Force stop all jobs and restarting them again" you are thinking that I should change the status of all jobs from Waiting to Withold and then back again?

    I can of course try that, but not sure what good it will do since newly created batch jobs are not executed either...

    But I am out of possible solutions myself, so I will create a job to change the status off all jobs when I log on to the customer the next time and see if it helps.

    Best regards,

    Jørgen

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

December Spotlight Star - Muhammad Affan

Congratulations to a top community star!

Top 10 leaders for November!

Congratulations to our November super stars!

Tips for Writing Effective Suggested Answers

Best practices for providing successful forum answers ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,280 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,235 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans