I've got an interesting situation I need to resolve. High volume usage situation and lots of async workflows.
I have an Async Workflow that triggers when a particular record is saved with a certain set of values.
The workflow deletes the record, and the reason it runs Async is because I need to ensure that several other workflows that run off of the same trigger complete successfully before the record is deleted. If that does not happen, there will be significant data integrity issues that arise.
Sometimes the system is being used heavily and the async process takes "a little longer than usual".
The problem is users get antsy and start re-updating the record many times in rapid succession to try and force the deletion. As a result there are now 100+ async processes to delete that record.
As a result we had an outage situation several days ago because 3000+ users all started rapidly re-editing the records to try and "force" it instead of waiting and trusting the system.
We had 20,000+ suspended processes and 20,000 waiting for resources/waiting workflow executions because of so many trying to "force" it.
99% of the backlog was the async delete workflow executions. And 99% of them ended up suspended because the record they were trying to delete did not exist, since the first time the async workflow ran it was deleted.
How can I build in a check so that when the workflow runs, it checks to make sure the record that triggered it still exists? If the record doesn't exist, I want to cause the workflow to stop/end. If the record DOES exist, then the workflow can run and delete it. That way, if someone happens to get click happy and triggers the workflow 10000 times, it only actually attempts to delete the record one time, instead of chewing up resources attempting to delete a record that's already been deleted, and thus ending up suspended 10000 times.