Views:

Applies to Product - FnO Platform


What’s happening?
The Statement Batch job has been failing for an extended period, specifically noted over the last six hours.


Reason:
The failure of the batch job was traced back to a potential stack overflow issue, indicated by the error message stating "Potential Stack Overflow in this session, the calling depth is over max depth." This is attributed to excessive usage of the stack, often due to recursion in customization or overloaded usage within the batch. Numerous batch crashes were identified, linked to stack overflow errors.


Resolution:

  1. A temporary workaround for stack overflow prevention has been enabled. It may take 30 minutes to an hour for this change to reflect.
  2. A batch restart was performed based on customer consent.
  3. Monitor the backend telemetry to observe if the stack overflow crashes reduce.
  4. For a permanent fix, collaborate with the code owner responsible for the recursive calls to prevent future stack overflow issues.
  5. Batch Framework Team don't own this, please reach out to FinOps Runtime Crash Team for further mitigation.