Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Dynamics 365 Community / Forums / Microsoft Dynamics CRM (Archived) / Starting lots of workf...
Microsoft Dynamics CRM (Archived)

Starting lots of workflows causes problems with sandbox process

(0) ShareShare
ReportReport
Posted on by Žarko Radevic 264

Hi, we are in a process of upgrading our CRM all the way from 2011 to Dynamics 365 online. As a preparation we are configuring our organization on current 2011 version to be supported in every aspect for online version. I edited all the workflows and plugins to be able to function in sandbox.

We have a routine of generating lots of renewal orders once a month and we do it in a workflow which has a custom workflow activity. Activity calls a wcf service which starts a job in a new thread and returns immediately, so wf activity is not waiting for the service method to finish. If we start this wf on 250 sales orders at the same time, we get an error, but not just on this wf, some other system jobs can also fail because of this:

Unhandled Exception: Microsoft.Crm.CrmException: System.ServiceModel.CommunicationObjectFaultedException: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #FB9C7DBD  at Microsoft.Crm.Sandbox.SandboxCodeUnit.Execute(IExecutionContext context)  at Microsoft.Crm.Workflow.Services.ProxyCustomActivity.Execute(CodeActivityContext executionContext)

We DO NOT get the same problem if we run the workflow activity outside sandbox. Running wf on 20 after 20 orders, I could not reproduce the error. I added tracing in wf activity and don't see my trace text, so I guess activity could not be started at all.

Any suggestion is appreciated

*This post is locked for comments

  • Žarko Radevic Profile Picture
    Žarko Radevic 264 on at
    RE: Starting lots of workflows causes problems with sandbox process

    This is the way we are calling the WCF web service:

    using (var client = new CrmExternal.MyClient(binding, endPointAddress))
    {
     var success = false;
        try
        {
            var request = new MyRequest
            {
                OpportunityId = opp.Id,
                AuthenticatedUserId = context.InitiatingUserId
            };
            client.MyMethod(request);

            if (client.State != CommunicationState.Faulted){
                client.Close();
                success = true;
            }
        }
        finally{
            if (!success)
                client.Abort();
                       
        }
    }

    It is a service that functions as a proxy, it sends forward requests to other services, but we are keeping it the only one service which is accessed from CRM code. Again, if the error was in calling the service, why don't I see my trace text which I create in the beginning of the wf activity?

  • Daniel Wikell Profile Picture
    Daniel Wikell 2,360 on at
    RE: Starting lots of workflows causes problems with sandbox process

    Ah my bad, I missed the part about the external service being the one spawning the threads. Yeah then you can disregard my previous reply for sure.

    What else could cause problems like this then... Possibly some kind of memory leak? How are you setting up the WCF-call from the WF? Is the Connection disposed properly after each call?

    Not sure why you see such different results when running in sandbox mode and in none-isolated mode though. Sandbox does impose a lot of security limitations so I guess it could block the connection classes from reading some WCF setting that is important for connection stability?

    Another problem we've ran into in the past when calling WCF-services from CRM is the connection being randomly force-closed before it is properly established. We've mitigated that particular problem by simply retrying the connection attempt up to 3 times which seems to work fine.

  • Žarko Radevic Profile Picture
    Žarko Radevic 264 on at
    RE: Starting lots of workflows causes problems with sandbox process

    Thank you for your answer, but please read once again, it is in the WCF service where I create a new thread, workflow activity doesn't know anything about it. WF activity just calls the wcf service in a synchronous way, not async method versions you can get when generating proxy in Visual Studio. Thanks for explaining how it works, I think the problem could be in wcf service becoming unresponsive, but it is strange that I can't see my tracing and error says nothing that could indicate a timeout exception.

  • Daniel Wikell Profile Picture
    Daniel Wikell 2,360 on at
    RE: Starting lots of workflows causes problems with sandbox process

    I would bet that the problem lies in the fact that you spawn multiple custom threads. When a plugin or workflow is running in sandbox mode, it is handled by the sandwork host process that in turn spawns the neccessary amount of sandbox worker processes for the actual plugin/workflow execution. From time to time you will see that new worker processes are spawned even though there were already existing idle ones. I guess the sandbox host determines that a sandbox worker is no longer needed and then terminates it. If you are immediately returning from the workflow process it could be that the sandbox host then sees the workflow as completed and that the sandbox worker can be safely terminated. This could explain why you don't see a problem for small amount of records (your threads finish in time before the default termination timer reaches its limit) and why you do see a problem with large amounts (your custom threads are still running when the worker process is terminated).

    What happens if you try an approach where you do all the work in the current workflow thread instead of creating new ones?

  • Žarko Radevic Profile Picture
    Žarko Radevic 264 on at
    RE: Starting lots of workflows causes problems with sandbox process

    Do you know where do I find config files for sandbox service? The only config file I found is CrmAsyncService.exe.config is Server\bin folder and it contains only one assemblyBinding element.

    What kind of limitations are you talking about? You think that the problem is that I am calling an external WCF service?

  • David Jennaway Profile Picture
    David Jennaway 14,063 on at
    RE: Starting lots of workflows causes problems with sandbox process

    You may have reached some limits on the WCF configuration. This is controlled by the .config settings in the respective process, so it's worth checking for any differences between these settings in the respective config files (for the sandbox process, the async process and web.config (for synchronous, non-sandboxed))

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

December Spotlight Star - Muhammad Affan

Congratulations to a top community star!

Top 10 leaders for November!

Congratulations to our November super stars!

Tips for Writing Effective Suggested Answers

Best practices for providing successful forum answers ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,269 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,198 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans