web
You’re offline. This is a read only version of the page.
close
Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Microsoft Dynamics CRM (Archived) / CRM 2016 On-Premise - ...
Microsoft Dynamics CRM (Archived)

CRM 2016 On-Premise - 100% CPU, bug in internal Dictionary

(0) ShareShare
ReportReport
Posted on by Alex Marshall 30

Version 1612 (8.2.0.749) (DB 8.2.0.749) on-premises

Our On-Premise CRM instance will occasionally spike to 100% constant CPU usage, usually around once a day.  CRM still responds, but incredibly slowly.  100% CPU usage continues until the IIS App Pool is restarted or iisreset.

I've dumped the IIS process into a DMP file and DebugDiag reveals that there are multiple threads stuck FindEntry/TryGetValue within a .NET Dictionary.  From the stack trace this appears to be accessed in parallel as part of the 'MultiEntitySearchService' which I suspect causes some sort of infinite loop caused by concurrent read/writes into the dictionary across multiple threads, causing the 100% CPU usage.  See here for a further explanation of this issue: https://blogs.msdn.microsoft.com/tess/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary/ - the stack trace of my DMP files exhibit the classic symptoms of this problem.  A quick decompilation of reveals a potential source of the problem within Microsoft.Crm.CrmCache.AggregateCacheCount.

As a workaround, is there a way to disable the parallel processing of Multi Entity search?

Sample stack trace below:

mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)+ab 

mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].TryGetValue(System.__Canon, System.__Canon ByRef)+14
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].AggregateCacheCount(Boolean)+e7
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].TryLookupEntryWithExactKey(System.String, System.Object ByRef)+4c
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].InternalTryLookupEntryAsObject(System.String, System.Object ByRef, Microsoft.Crm.IOrganizationContext)+9f
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].InternalTryLookupEntry(System.String, System.__Canon ByRef, Microsoft.Crm.IOrganizationContext)+25
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].TryLookupEntry(System.String, System.__Canon ByRef, Microsoft.Crm.IOrganizationContext)+44
Microsoft.Crm.Caching.BasicCrmCache`2[[System.Guid, mscorlib],[System.__Canon, mscorlib]].LookupEntry(System.Guid, Microsoft.Crm.IOrganizationContext)+78
Microsoft.Crm.Metadata.MetadataForMetadata.GetInstance(Microsoft.Crm.IOrganizationContext)+10d
Microsoft.Crm.Query.CrmSqlSelectCommandStrategy+QuickFindConditionBuilder..ctor(Microsoft.Xrm.Sdk.Query.LogicalOperator, System.String, Microsoft.Crm.Metadata.EntityMetadata, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.ExecutionContext)+186
Microsoft.Crm.Query.CrmSqlSelectCommandStrategy+QuickFindPredicateBuilder.CreateConditionBuilder(Microsoft.Xrm.Sdk.Query.LogicalOperator, Microsoft.Crm.Metadata.EntityMetadata, System.String, Microsoft.Crm.Platform.DatabaseQueryTarget)+80
Microsoft.Crm.Query.CrmSqlSelectCommandStrategy+PredicateBuilder.BeginWritingFilter(Microsoft.Xrm.Sdk.Query.LogicalOperator, Microsoft.Crm.Metadata.EntityMetadata, System.String, Microsoft.Crm.Platform.DatabaseQueryTarget)+1b
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+69
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+11d
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+11d
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+11d
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.EntityExpression)+303
Microsoft.Crm.Query.SelectPlan.GetSql()+1b
Microsoft.Crm.Query.QueryPlan.get_Command()+2e
Microsoft.Crm.BusinessEntities.BusinessProcessObject.DoRetrieveMultiple(Microsoft.Crm.BusinessEntities.BusinessEntityCollection, Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.PagingHelper, Microsoft.Crm.BusinessEntities.ExecutionContext, Boolean, Boolean)+1d0
Microsoft.Crm.BusinessEntities.BusinessProcessObject.DoRetrieveMultiple(Microsoft.Crm.BusinessEntities.BusinessEntityCollection, Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.ExecutionContext, Boolean, Boolean)+142
Microsoft.Crm.BusinessEntities.BusinessProcessObject.RetrieveMultiple(Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.ExecutionContext)+1d6
Microsoft.Crm.BusinessEntities.BusinessProcessObject.RetrieveMultiple(Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.BusinessEntities.ExecutionContext)+17
Microsoft.Crm.ObjectModel.MultiEntitySearchService.ExecuteEntitySearch(Microsoft.Crm.Caching.EntitySearchConfiguration, Microsoft.Crm.BusinessEntities.ExecutionContext, Microsoft.Xrm.Sdk.QuickFindResultCollection, Microsoft.Crm.ObjectModel.MultiEntitySearchSettings)+114
Microsoft.Crm.ObjectModel.MultiEntitySearchService.ExecuteSearches(Microsoft.Crm.Caching.EntitySearchConfiguration, System.Threading.Tasks.ParallelLoopState, Microsoft.Xrm.Sdk.QuickFindResultCollection, Microsoft.Crm.ObjectModel.MultiEntitySearchSettings, System.String, Microsoft.Crm.BusinessEntities.ExecutionContext, System.Web.HttpContext)+1f8
Microsoft.Crm.ObjectModel.MultiEntitySearchService+<>c__DisplayClass8.b__6(Microsoft.Crm.Caching.EntitySearchConfiguration, System.Threading.Tasks.ParallelLoopState)+3f
mscorlib_ni!System.Threading.Tasks.Parallel+<>c__DisplayClass42_0`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].b__1()+344
mscorlib_ni!System.Threading.Tasks.Task.InnerInvokeWithArg(System.Threading.Tasks.Task)+20
mscorlib_ni!System.Threading.Tasks.Task+<>c__DisplayClass176_0.b__0(System.Object)+115
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+15e
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+17
mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+21d
mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+98
mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)+34
mscorlib_ni!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)+1bb
mscorlib_ni!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean)+aa
mscorlib_ni!System.Threading.Tasks.Parallel.PartitionerForEachWorker[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Collections.Concurrent.Partitioner`1, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Action`3, System.Func`4, System.Func`5, System.Func`1, System.Action`1)+473
mscorlib_ni!System.Threading.Tasks.Parallel.ForEachWorker[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Action`3, System.Func`4, System.Func`5, System.Func`1, System.Action`1)+236
mscorlib_ni!System.Threading.Tasks.Parallel.ForEach[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Threading.Tasks.ParallelOptions, System.Action`2)+95
Microsoft.Crm.ObjectModel.MultiEntitySearchService.ExecuteParallelSearches(Microsoft.Crm.Caching.MultiEntitySearchConfigurationCollection, System.Threading.Tasks.ParallelOptions, System.String, Microsoft.Xrm.Sdk.QuickFindResultCollection, Microsoft.Crm.ObjectModel.MultiEntitySearchSettings, Microsoft.Crm.BusinessEntities.ExecutionContext)+106
Microsoft.Crm.ObjectModel.MultiEntitySearchService.MultiEntitySearch(System.String, System.String, System.String[], Microsoft.Crm.BusinessEntities.ExecutionContext)+2af
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+80
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+92
System_Web_Services_ni!System.Web.Services.Protocols.LogicalMethodInfo.Invoke(System.Object, System.Object[])+c0
Microsoft.Crm.Extensibility.InternalOperationPlugin.Execute(System.IServiceProvider)+e6
Microsoft.Crm.Extensibility.V5PluginProxyStep.ExecuteInternal(Microsoft.Crm.Extensibility.PipelineExecutionContext)+5b9
Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+19d
Microsoft.Crm.Extensibility.PipelineInstrumentationHelper.Execute(Boolean, System.String, ExecuteWithInstrumentation, Microsoft.Crm.Extensibility.PipelineExecutionContext)+206
Microsoft.Crm.Extensibility.Pipeline.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+28b
Microsoft.Crm.Extensibility.PipelineInstrumentationHelper.Execute(Boolean, System.String, ExecuteWithInstrumentation, Microsoft.Crm.Extensibility.PipelineExecutionContext)+80
Microsoft.Crm.Extensibility.MessageProcessor.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+504
Microsoft.Crm.Extensibility.InternalMessageDispatcher.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+117
Microsoft.Crm.Extensibility.ExternalMessageDispatcher.ExecuteInternal(Microsoft.Crm.Extensibility.IInProcessOrganizationServiceFactory, Microsoft.Crm.Extensibility.IPlatformMessageDispatcherFactory, System.String, System.String, Int32, Int32, Microsoft.Xrm.Sdk.ParameterCollection, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.BusinessEntities.UserAuth, System.Guid, System.Guid, Microsoft.Crm.Core.Security.Identity.UserType, System.Guid, Int32, System.Nullable`1, System.Version)+7e4
Microsoft.Crm.Extensibility.ExternalMessageDispatcher.Execute(Microsoft.Crm.Extensibility.IInProcessOrganizationServiceFactory, Microsoft.Crm.Extensibility.IPlatformMessageDispatcherFactory, System.String, System.String, Int32, Int32, Microsoft.Xrm.Sdk.ParameterCollection, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.BusinessEntities.UserAuth, System.Guid, System.Guid, Microsoft.Crm.Core.Security.Identity.UserType, System.Guid, Int32, System.Nullable`1)+c7
Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequestRequestWithInstrumentation(Microsoft.Xrm.Sdk.OrganizationRequest, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.Performance.WebServiceType, Microsoft.Crm.BusinessEntities.UserAuth, System.Guid, Microsoft.Crm.OrganizationContext, Boolean, Boolean, System.Object, Microsoft.Crm.Core.Security.Identity.UserType)+859
Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequest(Microsoft.Xrm.Sdk.OrganizationRequest, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.Performance.WebServiceType, Boolean, Microsoft.Crm.BusinessEntities.ExecutionContext)+1a4
Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.Execute(Microsoft.Xrm.Sdk.OrganizationRequest, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.Performance.WebServiceType, Boolean, Microsoft.Crm.BusinessEntities.ExecutionContext)+72
Microsoft.Crm.Extensibility.OrganizationSdkService.Execute(Microsoft.Xrm.Sdk.OrganizationRequest)+48
DynamicClass.SyncInvokeExecute(System.Object, System.Object[], System.Object[])+59
System_ServiceModel_ni!System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(System.Object, System.Object[], System.Object[] ByRef)+4c7
System_ServiceModel_ni!System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(System.ServiceModel.Dispatcher.MessageRpc ByRef)+26f
System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(System.ServiceModel.Dispatcher.MessageRpc ByRef)+228
System_ServiceModel_ni!System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean)+dc
System_ServiceModel_ni!System.ServiceModel.Dispatcher.MessageRpc+Wrapper.Resume(Boolean ByRef)+43
System_ServiceModel_ni!System.ServiceModel.Dispatcher.ThreadBehavior.ResumeProcessing(System.ServiceModel.Dispatcher.IResumeMessageRpc)+28
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+15e
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+17
mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()+70
mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+160
[[DebuggerU2MCatchHandlerFrame]]
[[ContextTransitionFrame]]
[[DebuggerU2MCatchHandlerFrame]]

*This post is locked for comments

I have the same question (0)
  • Suggested answer
    Community Member Profile Picture
    Community Member on at

    Normally CPU is hardly boost to 100% usage... unless you are running some super super heavy jobs.

    Check one thing, after you IIS reset, observe the CPU usage, then perform a search then observe the CPU usage again.

    Is there any other feature installed in this Server? This is all in one server or it just host the APP?

    How fast/slow is the search performance? Have you tuned the DB by putting index too see if it helps?

    Any IIS settings causing the CPU usage? HTTP compression?

  • Alex Marshall Profile Picture
    Alex Marshall 30 on at

    Hi Melvin Fong, I'm not sure you understand the issue, this is an infinite loop/deadlock condition within the CRM assembly causing constant 100% CPU usage once the deadlock is triggered so any other features installed on the server are largely irrelevant in this case.

    See https://blogs.msdn.microsoft.com/tess/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary/

  • Community Member Profile Picture
    Community Member on at

    So tell me how are you going to troubleshoot if you don't explore some other possibility? Do you able to find out what is causing the deadlock?

  • Alex Marshall Profile Picture
    Alex Marshall 30 on at

    MÑ‘LvìN, I have already troubleshooted the issue.  The problem is caused by the internal workings of the MultiEntitySearchService.  Every dump I have taken points to this process consuming 100% CPU on multiple threads within System.Collections.Generic.Dictionary.

  • joman Profile Picture
    joman 617 on at

    Is it happens at the same time daily?

    If "Yes", then you need to search some daily job in CRM.

  • Community Member Profile Picture
    Community Member on at

    Is this a single box CRM installation or you have separate box for APP and DB?

    Not sure this can help but there is a setting in SQL you can play with it.

    docs.microsoft.com/.../configure-the-max-degree-of-parallelism-server-configuration-option

    www.simple-talk.com/.../understanding-and-using-parallelism-in-sql-server

  • Simon Tooley Profile Picture
    Simon Tooley 60 on at

    Hi Alex,

    We're experiencing exactly the same issue randomly on 3 web farmed instances of CRM 365.  Our on-premise version is a heavily loaded system and causes us major problems when having to restart the AppPool to fix.

    Did you ever find any resolution to the issue?

    Regards,

    Simon Tooley

  • MikeWalker74 Profile Picture
    MikeWalker74 on at

    Hi, We are also getting similar scenarios in our production environment and I believe they are similar in effect and cause. Did you log a ticket with Microsoft or get any further advice on this.

  • Simon Tooley Profile Picture
    Simon Tooley 60 on at

    Hi, Yes we've opened a ticket with Microsoft and are still working with them. There initial solution is to change the configuration of the IIS app pool to set it to recycle if the application pool runs at a certain percentage or above for so many minutes.

    We currently have the CRM application pool configured as follows:

    CPU – Limit (percent) = 90

    CPU – Limit Action = KillW3wp

    CPU – Limit Interval (minutes) = 2

    This option hides the option rather than finding the cause of the issue, so we are still working with Microsoft to solve the issue.

  • MikeWalker74 Profile Picture
    MikeWalker74 on at

    I agree, this is a work around to what could be causing further performance impact. Please post any further responses as would be good to know. 

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Responsible AI policies

As AI tools become more common, we’re introducing a Responsible AI Use…

Neeraj Kumar – Community Spotlight

We are honored to recognize Neeraj Kumar as our Community Spotlight honoree for…

Congratulations to the November Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > 🔒一 Microsoft Dynamics CRM (Archived)

#1
SA-08121319-0 Profile Picture

SA-08121319-0 4

#1
Calum MacFarlane Profile Picture

Calum MacFarlane 4

#3
Alex Fun Wei Jie Profile Picture

Alex Fun Wei Jie 2

Last 30 days Overall leaderboard

Featured topics

Product updates

Dynamics 365 release plans