Scalability of Microsoft CRM

(0) Share

Report

Posted on by Community Member

We have been working on testing the performance of our Microsoft CRM 2016 installation. To test, we are using a simple client written in .NET using the .NET SDK to query for a single entity. The entity in our case is a custom entity. There are only 2 rows in our custom entity table. the point of the test is not to retrieve the entity, but to test the response time and number of concurrent calls. Therefore, our query always fetches where entity.Id == Gud.Empty. Our test is pretty simple:

start with one thread, querying where entity.Id == Guid.Empty, in a loop.
every N units of time (ie 3-5 minutes), add another thread

We use explicit new threads on the client to avoid issues with the thread pool. As we add client threads, the response time stays flat and the requests per second increase until we hit about 10 clients. After that, the requests per second goes flat (is stays at the same rate) and the average query time increases linearly with the number of clients. I would think Microsoft CRM should be able to support more than 10 concurrent requests.

It really sounds like a request thread pool issue on CRM. Trial and error doesn't seem like a good option as our CRM developers have said that we are not that free to change defaults on the server without contacting Microsoft support before hand.

Microsoft CRM 2016
2 front end CRM servers behind a load balancer

*This post is locked for comments

I have the same question (0)

All responses (7)

Answers (1)

PranavShroti 4,510 on at

Like (0)

Report

Hi Phil,

Will it be possible to provide more details like

-Environment

-How many records you wana enter

-Total number of users you wana simulate

-Utility using bulk upload of records?

-When you say performance what exacotly you mean? Page load time or Integration 100 records in 5sec (just an example)

-When do you expect these 10 concurrent to be active, I am assuming during integration/migration only? what tool you are using for migration/integartion (if any)

Regards,

Pranav

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

- environment: I am not the system admin, but I have been told it is production scale with two front end web servers.

- number of records to enter: none. Query only for this test.

- number of users: more than the 10 I can currently call. The API that calls into CRM will be accessed by an internet based application. My goal is to understand at which point our API will stop scaling. Right now I can only report that if we have more than 10 concurrent queries executing, response time will degrade linearly with each additional client. Not good.

- utility to load: read only queries in this test.

- performance: how many simple Retrieve Multiple queries (that always return zero rows) execute per second. I have never seen that go over 125/sec. If I use more complex queries, then it adds to the variables.

- when active: our app is a new app. Our user profile is expected to be bursty at certain points thru the day, like at morning/evening rush hour.

I am a consultant. On this project my client has asked how well does this solution scale? Based on current results I can say the Microsoft CRM 2016 environment we have can support 10 concurrent queries against a table with 2 rows before latency per query starts to degrade. The performance will degrade linearly with each additional query.

I am sure their reply will be: "that is ok, I understand this is a free open source product with no support, oh wait, it's a Microsoft product that we are spending huge license costs for."

I was hoping this is a common question so I thought I would try this forum before going through all hassle of opening a support ticket.

Our testing app harness is pretty simple. I can look at sharing it on github for others try.

Was this reply helpful? Yes No
Suggested answer

PranavShroti 4,510 on at

Like (0)

Report

Hi Phil,

Since they are talking about two front end servers I believe this is an onPremise implementation. Cloud version of Dynamics crm only supports 2 concurrent threads to upload data.

Now coming to the performance testing I believe rather then relying on a .Net utility you should go for tools like

QTP: www.guru99.com/quick-test-professional-qtp-tutorial-1.html

Selenium: www.testhouse.net/.../automation-testing-with-dynamics-crm-using-selenium

Integration/Migration: Depends on what kind of data you are importing/uploading, since CRM is a relational system, there are lots of relationships if form of lookups, apparently resolving this will take more time, so how many records can you upload clearly depends what data you are handling. In your case you need to take the most complex entity to benchmark this.

Another aspect that comes into play is DB size, it may grow very fast, and specifically few tables and take must of the load like Activities, AsyncOperationBase (if sharing of records are happening) etc.

"- performance: how many simple Retrieve Multiple queries (that always return zero rows) execute per second. I have never seen that go over 125/sec. If I use more complex queries, then it adds to the variables."

I would suggest you ask right questions:

--how much maximum data you are considering

--What is the size of the database expected to grow in a year

--What is the configuration of DB box where CRM database is hosted?

--Where the log file and data files are hosted in SQL box? They should be on different drives for better performance.

All these will impact query performance.

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

Hi Pranav,

I do appreciate your attempts to suggestions to help me. We cannot use something like Selenium as we are not testing the CRM portal/web site. Our initial tests were using Visual Studio Web/Load tests, but ran into the performance issues described above. Our approach was to eliminate as many variables as possible to identify why our requests appear to be throttled. Our .NET client code is shown below for reference. As you can see the query is very basic. Our CRM instance is fairly empty right now and the Contact set has 36 rows.

When testing this code from a console app, when we have 1-10 instances, the queries/second increase linearly with each client. Once you hit 11+ clients, each request takes slightly longer time and the response time increases linearly with each new client. (ie the request rate stays the same).

It is clear that the CRM servers is the bottleneck and appears to limit how many concurrent queries can be run. From what I have read, I would guess it is related to the thread pool. In a standard .NET app, I would adjust the thread pool parameters either in the machine.config or through the TheadPool class in code. However, this is Microsoft CRM and any changes are typically require approval from Microsoft before we can make changes (that comes from our CRM team).

I fear this question is too specific and I should just submit a support ticket to Microsoft.

More information on the size of our CRM servers:

CRM front end/back end servers (2 each): Azure VMs: A4 v2 (4 cores, 8 GB RAM, 40 GB disk)

CRM database servers: Azure VMs: D3 v2 (4 cores, 14 GB RAM, 200 GB disk)

Uri uri = ...

ClientCredentials credentials = ...

IOrganizationService organizationService = new OrganizationServiceProxy(uri, null, credentials, null);

bool done = false; // done will be set to true externally by a timer

while (!done) {

FilterExpression filter = new FilterExpression();

filter.Conditions.Add(new ConditionExpression("contactid", ConditionOperator.Equal, Guid.NewGuid()));

QueryExpression query = new QueryExpression("contact");

query.ColumnSet.AddColumns("firstname", "lastname", "emailaddress1");

query.Criteria.AddFilter(filter);

// time and record how long this takes

EntityCollection entities = service.RetrieveMultiple(query);

// end of time

}

Was this reply helpful? Yes No
Verified answer

ashlega 34,477 on at

Like (0)

Report

Hi Phil,

are you running this test from the same "client" machine? I would try to put that utility of yours on different client machines and see what happens there because there is also a limit on the number of host-to-host connections in NET(the way I see it, you have maxed out that limit at 10 on the server side..):

msdn.microsoft.com/.../system.net.servicepointmanager.defaultconnectionlimit.aspx

Was this reply helpful? Yes No
Community Member on at

Like (0)

Report

You are correct. We had looked at that, but come to think of it, I am not sure we re-added it in our latest test code. I am going to double check that and retest again. I have a feeling the load level will look better. In all our APIs, we added code like this right in App Start (the GetSetting function reads the app setting or defaults to 1000 if not set)

ServicePointManager.DefaultConnectionLimit = GetSetting("Network:DefaultConnectionLimit", 1000);

Was this reply helpful? Yes No
ashlega 34,477 on at

Like (0)

Report

That setting will help on the client, but you may be hitting it on the crm server, too.. all those requests are from the same client, so look at it from the server perspective.. maybe still try more clients on different machines..

Was this reply helpful? Yes No