Skip to main content

Notifications

Announcements

No record found.

Microsoft Dynamics AX (Archived)

Workflow message processing job error at every 10th min (recursively).

Posted on by 140

Hi All,

I am facing one issue with the workflow message processing batch job. The job throwing an error after every 10 mins (approx.), and if any workflow action is in progress during that time (close to 10th min), that activity could not be processed and the WF got stuck in the system. This is happening continuously.

Experts (@ André Arnaud de Calavon) & all, Could you please share some tips, how to resolve this if anyone face the same issue in past.

Your pointers are highly appreciated!! Great thanks in advance.

Thanks!

*This post is locked for comments

  • thomas_philipose Profile Picture
    thomas_philipose 15 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi all,

    i went through the thread above. Very useful and informative. Thanks to you guys, who spend time here.

    i would like to add some thing here. We faced the same issue. Workflow message processing batch getting stuck...

    I recalled all the WF which were in PENDING status. While recalling i was unable to recall a particular WF. Then i had to delete it from the WORKFLOWTRACKINGSTATUSTABLE. (I feel that this one was the cause of all issues...)

    Meaning I was left with no single WF having pending status. I did a PHYSICAL RESTART of AOS and Database machine. 

    Later when the System was up, i accessed the "Workflow message processing batch"  changed the status to ENDED. Then to WAITING. Immediately system started processing the WF. Everything went on smoothly.

    Thanks

  • PK Axapta Profile Picture
    PK Axapta 140 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi Michele,

    Thanks for your response on this.....

    I have also solved this issue in the same way, you mentioned in your response.

  • Verified answer
    Michele Gozzi Profile Picture
    Michele Gozzi 75 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi,

    I've just fixed the same problem in my environment.

    I found that there was and orphan record in SYSWORKFLOWMESSAGETABLE without a matching record in SYSWORKFLOWTABLE.

    The problem sits in the method \Classes\SysWorkflowQueue\dispatch. the method iterates the messages from the queue one by one. If the message doesn't have a matching workflow, the method just throws an exception and leaves the remaining messages in queue unprocessed.

    In fact you can verify the workflows that are stuck with the query below:

    select
    a.RECID, b.INSTANCEID, b.INSTANCENUMBER, a.BATCHAFFINITY, a.MESSAGETYPE, a.*, b.*
    from SYSWORKFLOWMESSAGETABLE a
    join SYSWORKFLOWTABLE b on a.PARTITION = b.PARTITION and a.ROOTCORRELATIONID = b.ROOTCORRELATIONID
    where a.MESSAGELIFECYCLESTATE = 1
    order by a.RECID asc

    In my case the message was linked to a claim that was already completed (weird!).

    You can use the query below to find the orphans.

    SELECT
    b.INSTANCENUMBER -- this should be null for orphans
    , a.RECID -- recid or the orphan message
    , a.WORKFLOWCONTEXTTABLEID
    , a.WORKFLOWCONTEXTRECID

    --, e.EXPNUMBER
    , a.*
    from SYSWORKFLOWMESSAGETABLE a
    left join SYSWORKFLOWTABLE b on a.PARTITION = b.PARTITION and a.ROOTCORRELATIONID = b.ROOTCORRELATIONID
    --left join TRVEXPTABLE e
    --ON a.PARTITION = e.PARTITION
    --AND a.WORKFLOWCONTEXTRECID = e.RECID
    where a.MESSAGELIFECYCLESTATE = 1
    AND b.recid IS NULL
    --AND a.WORKFLOWCONTEXTTABLEID = 484

    Then just delete the messages returned by the query above using the RECID:

    DELETE FROM SYSWORKFLOWMESSAGETABLE WHERE RECID = <add your recid here>

    Hope this help!

  • Verified answer
    André Arnaud de Calavon Profile Picture
    André Arnaud de Cal... 291,280 Super User 2024 Season 2 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi Pankaj,

    You can first try to see if you can find a source document like purchase order or project timesheet based on the context table and record Id. If there is a record with an invalid reference, the document might have been deleted somehow, without taking care of running workflow instances. In that case you can also delete the record in the workflow message table. If not, you can indeed change the modified date time for all of them to a future date and then try to set one by one to a date/time of at least 10 minutes ago to see if it gets processed correctly or not. In this way, you should be able to find the record which is causing the issue.

  • PK Axapta Profile Picture
    PK Axapta 140 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi Andre,

    Thanks for your prompt response...

    No, all 50 records are not pointing to the same purchase order. There are couple of records which don't have PO in purchtable table (when i am checking with WorkflowContextRecId). Also to be more precise, there are 10/50 records which are pointing to some other customized WF as well.

    So can i check record by record as you advised or there is no harm in updating to future date (i have tested on lower environment, future date is not poping up WF message error).

  • André Arnaud de Calavon Profile Picture
    André Arnaud de Cal... 291,280 Super User 2024 Season 2 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi Pankaj,

    Are all 50 records referring to the same purchase order? You can indeed change the modified date/time to a date in future (using SQL scripting/editing). When it is running OK, then try to revert it back record by record and then see which ones will be processed or not. At the end, there might be one left which has the actual issue.

  • PK Axapta Profile Picture
    PK Axapta 140 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Many thanks, Andre for your reply and sorry for not replying on this sooner, as I have to deliver other multiple tasks.

    As per my analysis, I have found around 50 records with same batch GUID and MessageLifeCycleState=1. All these 50 records have source document records based on the filed WorkflowContextRecId and in my case source document records are in PurchTable which I can't delete. So the only solution left with me is to update the modified date time field of sysworkflowmessagetable to the future date, so the batch job will not pick these corrupted records after every 10 mins. As per your experience on this, Is it advisable to update the modified date time field by taking all due diligence while updating on the live environment.

    Can you please advise on this further. Great thanks for all your help!!  

  • Suggested answer
    André Arnaud de Calavon Profile Picture
    André Arnaud de Cal... 291,280 Super User 2024 Season 2 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi Pankaj,

    There is no need to tag my name in posts. It will not notify me.

    About your question: There is probaby a corrupt workflow message in your environment. I had seen this before. Only to find that one message is a bit cumbersome.

    At a customer I used some SQL scripts to find the error. Currently, I enjoy vacation and don't have access to all my media. So, will try to help you.

    When the workflow batch job starts, it will mark all new messages with a batch guid ID. then also, all pending messages with a batch guid and change time of 10 minutes and older will be included. This is the reason that it comes back every 10 minutes. Be aware that the problem message will cause longer and longer queues.

    You can find all messages in a table. I forgot the exact name, but probably SysWorkflowMessageTable. You can filter on status enum value 1 and all that do have a batch guid. Then  you would possibly find a larger group with the same batch guid. Try to see if the one with the lowest record ID is your issue. This can be done trying to find the original document record. This one might have been deleted. If so, you can also delete the workflow message record. If not, you can try to edit the modified date time in SQL. What I did before I hd some SQL scripts was changing half of the records of a certain batch guid to have a time longer than 10 minutes ago. Then a lot of pending messages should be processed. Then continue it until there is one message left. Or just try to set the message with oldest record id (within the batch guid) to a future date, so it will be excluded from the retry mechanism.

    Hope this will help you already. If not, update the post with your findings. Then I will try to help further.

  • PK Axapta Profile Picture
    PK Axapta 140 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Thanks, Sefa for your reply!

    Below is the screenshot of error message. I am not getting any error in event viewer log.

    Do let me know how I can tackle this.

    WF-error.jpg

  • Sefa Duman Profile Picture
    Sefa Duman 1,167 on at
    RE: Workflow message processing job error at every 10th min (recursively).

    Hi Pankaj,

    Could you please share the error message? It would be good if you look at the errors from eventviewer on server which AOS runs.

    Error messages in windows event logs will make easier to understand what is wrong.

    Best regards,

    Sefa

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

December Spotlight Star - Muhammad Affan

Congratulations to a top community star!

Top 10 leaders for November!

Congratulations to our November super stars!

Tips for Writing Effective Suggested Answers

Best practices for providing successful forum answers ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,280 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,235 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans