You’re offline. This is a read only version of the page.
30
Version 1612 (8.2.0.749) (DB 8.2.0.749) on-premises
Our On-Premise CRM instance will occasionally spike to 100% constant CPU usage, usually around once a day. CRM still responds, but incredibly slowly. 100% CPU usage continues until the IIS App Pool is restarted or iisreset.
I've dumped the IIS process into a DMP file and DebugDiag reveals that there are multiple threads stuck FindEntry/TryGetValue within a .NET Dictionary. From the stack trace this appears to be accessed in parallel as part of the 'MultiEntitySearchService' which I suspect causes some sort of infinite loop caused by concurrent read/writes into the dictionary across multiple threads, causing the 100% CPU usage. See here for a further explanation of this issue: https://blogs.msdn.microsoft.com/tess/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary/ - the stack trace of my DMP files exhibit the classic symptoms of this problem. A quick decompilation of reveals a potential source of the problem within Microsoft.Crm.CrmCache.AggregateCacheCount.
As a workaround, is there a way to disable the parallel processing of Multi Entity search?
Sample stack trace below:
mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)+ab
mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].TryGetValue(System.__Canon, System.__Canon ByRef)+14
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].AggregateCacheCount(Boolean)+e7
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].TryLookupEntryWithExactKey(System.String, System.Object ByRef)+4c
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].InternalTryLookupEntryAsObject(System.String, System.Object ByRef, Microsoft.Crm.IOrganizationContext)+9f
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].InternalTryLookupEntry(System.String, System.__Canon ByRef, Microsoft.Crm.IOrganizationContext)+25
Microsoft.Crm.CrmCache`1[[System.__Canon, mscorlib]].TryLookupEntry(System.String, System.__Canon ByRef, Microsoft.Crm.IOrganizationContext)+44
Microsoft.Crm.Caching.BasicCrmCache`2[[System.Guid, mscorlib],[System.__Canon, mscorlib]].LookupEntry(System.Guid, Microsoft.Crm.IOrganizationContext)+78
Microsoft.Crm.Metadata.MetadataForMetadata.GetInstance(Microsoft.Crm.IOrganizationContext)+10d
Microsoft.Crm.Query.CrmSqlSelectCommandStrategy+QuickFindConditionBuilder..ctor(Microsoft.Xrm.Sdk.Query.LogicalOperator, System.String, Microsoft.Crm.Metadata.EntityMetadata, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.ExecutionContext)+186
Microsoft.Crm.Query.CrmSqlSelectCommandStrategy+QuickFindPredicateBuilder.CreateConditionBuilder(Microsoft.Xrm.Sdk.Query.LogicalOperator, Microsoft.Crm.Metadata.EntityMetadata, System.String, Microsoft.Crm.Platform.DatabaseQueryTarget)+80
Microsoft.Crm.Query.CrmSqlSelectCommandStrategy+PredicateBuilder.BeginWritingFilter(Microsoft.Xrm.Sdk.Query.LogicalOperator, Microsoft.Crm.Metadata.EntityMetadata, System.String, Microsoft.Crm.Platform.DatabaseQueryTarget)+1b
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+69
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+11d
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+11d
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.FilterExpression)+11d
Microsoft.Crm.Query.SelectVisitor.Visit(Microsoft.Crm.Query.EntityExpression)+303
Microsoft.Crm.Query.SelectPlan.GetSql()+1b
Microsoft.Crm.Query.QueryPlan.get_Command()+2e
Microsoft.Crm.BusinessEntities.BusinessProcessObject.DoRetrieveMultiple(Microsoft.Crm.BusinessEntities.BusinessEntityCollection, Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.PagingHelper, Microsoft.Crm.BusinessEntities.ExecutionContext, Boolean, Boolean)+1d0
Microsoft.Crm.BusinessEntities.BusinessProcessObject.DoRetrieveMultiple(Microsoft.Crm.BusinessEntities.BusinessEntityCollection, Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.ExecutionContext, Boolean, Boolean)+142
Microsoft.Crm.BusinessEntities.BusinessProcessObject.RetrieveMultiple(Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.Platform.DatabaseQueryTarget, Microsoft.Crm.BusinessEntities.ExecutionContext)+1d6
Microsoft.Crm.BusinessEntities.BusinessProcessObject.RetrieveMultiple(Microsoft.Crm.Query.EntityExpression, Microsoft.Crm.BusinessEntities.ExecutionContext)+17
Microsoft.Crm.ObjectModel.MultiEntitySearchService.ExecuteEntitySearch(Microsoft.Crm.Caching.EntitySearchConfiguration, Microsoft.Crm.BusinessEntities.ExecutionContext, Microsoft.Xrm.Sdk.QuickFindResultCollection, Microsoft.Crm.ObjectModel.MultiEntitySearchSettings)+114
Microsoft.Crm.ObjectModel.MultiEntitySearchService.ExecuteSearches(Microsoft.Crm.Caching.EntitySearchConfiguration, System.Threading.Tasks.ParallelLoopState, Microsoft.Xrm.Sdk.QuickFindResultCollection, Microsoft.Crm.ObjectModel.MultiEntitySearchSettings, System.String, Microsoft.Crm.BusinessEntities.ExecutionContext, System.Web.HttpContext)+1f8
Microsoft.Crm.ObjectModel.MultiEntitySearchService+<>c__DisplayClass8.b__6(Microsoft.Crm.Caching.EntitySearchConfiguration, System.Threading.Tasks.ParallelLoopState)+3f
mscorlib_ni!System.Threading.Tasks.Parallel+<>c__DisplayClass42_0`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].b__1()+344
mscorlib_ni!System.Threading.Tasks.Task.InnerInvokeWithArg(System.Threading.Tasks.Task)+20
mscorlib_ni!System.Threading.Tasks.Task+<>c__DisplayClass176_0.b__0(System.Object)+115
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+15e
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+17
mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+21d
mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+98
mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)+34
mscorlib_ni!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)+1bb
mscorlib_ni!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean)+aa
mscorlib_ni!System.Threading.Tasks.Parallel.PartitionerForEachWorker[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Collections.Concurrent.Partitioner`1, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Action`3, System.Func`4, System.Func`5, System.Func`1, System.Action`1)+473
mscorlib_ni!System.Threading.Tasks.Parallel.ForEachWorker[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Action`3, System.Func`4, System.Func`5, System.Func`1, System.Action`1)+236
mscorlib_ni!System.Threading.Tasks.Parallel.ForEach[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Threading.Tasks.ParallelOptions, System.Action`2)+95
Microsoft.Crm.ObjectModel.MultiEntitySearchService.ExecuteParallelSearches(Microsoft.Crm.Caching.MultiEntitySearchConfigurationCollection, System.Threading.Tasks.ParallelOptions, System.String, Microsoft.Xrm.Sdk.QuickFindResultCollection, Microsoft.Crm.ObjectModel.MultiEntitySearchSettings, Microsoft.Crm.BusinessEntities.ExecutionContext)+106
Microsoft.Crm.ObjectModel.MultiEntitySearchService.MultiEntitySearch(System.String, System.String, System.String[], Microsoft.Crm.BusinessEntities.ExecutionContext)+2af
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+80
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+92
System_Web_Services_ni!System.Web.Services.Protocols.LogicalMethodInfo.Invoke(System.Object, System.Object[])+c0
Microsoft.Crm.Extensibility.InternalOperationPlugin.Execute(System.IServiceProvider)+e6
Microsoft.Crm.Extensibility.V5PluginProxyStep.ExecuteInternal(Microsoft.Crm.Extensibility.PipelineExecutionContext)+5b9
Microsoft.Crm.Extensibility.VersionedPluginProxyStepBase.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+19d
Microsoft.Crm.Extensibility.PipelineInstrumentationHelper.Execute(Boolean, System.String, ExecuteWithInstrumentation, Microsoft.Crm.Extensibility.PipelineExecutionContext)+206
Microsoft.Crm.Extensibility.Pipeline.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+28b
Microsoft.Crm.Extensibility.PipelineInstrumentationHelper.Execute(Boolean, System.String, ExecuteWithInstrumentation, Microsoft.Crm.Extensibility.PipelineExecutionContext)+80
Microsoft.Crm.Extensibility.MessageProcessor.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+504
Microsoft.Crm.Extensibility.InternalMessageDispatcher.Execute(Microsoft.Crm.Extensibility.PipelineExecutionContext)+117
Microsoft.Crm.Extensibility.ExternalMessageDispatcher.ExecuteInternal(Microsoft.Crm.Extensibility.IInProcessOrganizationServiceFactory, Microsoft.Crm.Extensibility.IPlatformMessageDispatcherFactory, System.String, System.String, Int32, Int32, Microsoft.Xrm.Sdk.ParameterCollection, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.BusinessEntities.UserAuth, System.Guid, System.Guid, Microsoft.Crm.Core.Security.Identity.UserType, System.Guid, Int32, System.Nullable`1, System.Version)+7e4
Microsoft.Crm.Extensibility.ExternalMessageDispatcher.Execute(Microsoft.Crm.Extensibility.IInProcessOrganizationServiceFactory, Microsoft.Crm.Extensibility.IPlatformMessageDispatcherFactory, System.String, System.String, Int32, Int32, Microsoft.Xrm.Sdk.ParameterCollection, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.BusinessEntities.UserAuth, System.Guid, System.Guid, Microsoft.Crm.Core.Security.Identity.UserType, System.Guid, Int32, System.Nullable`1)+c7
Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequestRequestWithInstrumentation(Microsoft.Xrm.Sdk.OrganizationRequest, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.Performance.WebServiceType, Microsoft.Crm.BusinessEntities.UserAuth, System.Guid, Microsoft.Crm.OrganizationContext, Boolean, Boolean, System.Object, Microsoft.Crm.Core.Security.Identity.UserType)+859
Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.ExecuteRequest(Microsoft.Xrm.Sdk.OrganizationRequest, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.Performance.WebServiceType, Boolean, Microsoft.Crm.BusinessEntities.ExecutionContext)+1a4
Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.Execute(Microsoft.Xrm.Sdk.OrganizationRequest, Microsoft.Crm.Extensibility.CorrelationToken, Microsoft.Crm.Sdk.CallerOriginToken, Microsoft.Crm.Performance.WebServiceType, Boolean, Microsoft.Crm.BusinessEntities.ExecutionContext)+72
Microsoft.Crm.Extensibility.OrganizationSdkService.Execute(Microsoft.Xrm.Sdk.OrganizationRequest)+48
DynamicClass.SyncInvokeExecute(System.Object, System.Object[], System.Object[])+59
System_ServiceModel_ni!System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(System.Object, System.Object[], System.Object[] ByRef)+4c7
System_ServiceModel_ni!System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(System.ServiceModel.Dispatcher.MessageRpc ByRef)+26f
System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(System.ServiceModel.Dispatcher.MessageRpc ByRef)+228
System_ServiceModel_ni!System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean)+dc
System_ServiceModel_ni!System.ServiceModel.Dispatcher.MessageRpc+Wrapper.Resume(Boolean ByRef)+43
System_ServiceModel_ni!System.ServiceModel.Dispatcher.ThreadBehavior.ResumeProcessing(System.ServiceModel.Dispatcher.IResumeMessageRpc)+28
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+15e
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+17
mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()+70
mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+160
[[DebuggerU2MCatchHandlerFrame]]
[[ContextTransitionFrame]]
[[DebuggerU2MCatchHandlerFrame]]
*This post is locked for comments
I didn't log this with Microsoft, based on past experience their support has been poor. We are still experiencing this problem. Killing the IIS App Pool is the temporary fix. MS need to update their code to lock whilst accessing the non-threadsafe Dictionary.
Hi,
Unfortunately, we haven't been in a position to install this upgrade into our production environment to actually test this. The issue is quite intermittent so may be a while once we've installed it to actually confirm it does fix the issue.
Regards,
Simon
Hello,
We have the same issue here. The call stack also points to
System.Collections.Generic.Dictionary`2[[System.Guid, mscorlib],[System.Nullable`1[[Microsoft.Crm.Sdk.PrivilegeDepth, Microsoft.Crm.Sdk]], mscorlib]].FindEntry(System.Guid)+77 System.Collections.Generic.Dictionary`2[[System.Guid, mscorlib],[System.Nullable`1[[Microsoft.Crm.Sdk.PrivilegeDepth, Microsoft.Crm.Sdk]], mscorlib]].TryGetValue(System.Guid, System.Nullable`1 ByRef)+14 Microsoft.Crm.Security.User.GetPrivilegeDepth(System.Guid, Microsoft.Crm.IOrganizationContext)+f6
Did upgrading to 8.2.3 fixed the issue?
Regards,
Hi,
Further to this thread, we've been able to catch a DebugDiag trace of when this is happening for us, the stack trace is as follows:
Multiple threads enumerating through a collection is intrinsically not a thread-safe procedure. If the dictionary object accessed by these threads is declared as static then the threads can go in an infinite loop while trying to enumerate the dictionary if one of the threads writes to the dictionary while the other threads are reading\enumerating through the same dictionary. You may also experience High CPU during this stage.
We've had a look at the decompiled code CRM code where this code relates to and believe that the November 2018 CRM release (8.2.3) has changes in it which may have fixed this issue, although we are yet to test.
Regards,
Simon
Thanks Simon for confirming it.
Hi Ganesan,
No we haven't had any resolution from Microsoft, only the steps to hide the problem documented in the eralier message.
Regards,
Simon
Hi Simon,
Did you get any resolution steps from Microsoft? We are also facing the same issue in our CRM 2016 On-Premise.
Thanks,
Ganesan
I agree, this is a work around to what could be causing further performance impact. Please post any further responses as would be good to know.
Hi, Yes we've opened a ticket with Microsoft and are still working with them. There initial solution is to change the configuration of the IIS app pool to set it to recycle if the application pool runs at a certain percentage or above for so many minutes.
We currently have the CRM application pool configured as follows:
CPU – Limit (percent) = 90
CPU – Limit Action = KillW3wp
CPU – Limit Interval (minutes) = 2
This option hides the option rather than finding the cause of the issue, so we are still working with Microsoft to solve the issue.
Hi, We are also getting similar scenarios in our production environment and I believe they are similar in effect and cause. Did you log a ticket with Microsoft or get any further advice on this.
Stay up to date on forum activity by subscribing. You can also customize your in-app and email Notification settings across all subscriptions.
#1
André Arnaud de Cal...
291,240
Super User 2024 Season 2
#2
Martin Dráb
230,149
Most Valuable Professional
#3
nmaenpaa
101,156
Share
Report
All responses (
Answers (