Email Router Demystified – Know the limitations

I have worked on a lot of cases in regards to the email router and I have come across some items that had not been documented or were documented in some very obscure locations. So I want to try and provide a compiled list of limitations with various services the Email Router connects to. Let’s first start off with limitations with Exchange Online

Exchange Online

We are starting to see a lot of our customers migrate their mailboxes to utilize Exchange Online. With this, we are finding a few items that customers are not aware of after signing up for Exchange Online. There are two big limitations of Exchange Online that applies to CRM.

Bulk E-Mail and Daily Recipient Rate Limits

There is a recipient rate limit, often called a sender limit or sending limit, for cloud-based e-mail accounts. This limit defines the maximum number of recipients that can receive e-mail messages sent from a single cloud-based account in a 24-hour period. The recipient rate limit for Exchange Online is 1,500 recipients per day.

This comes directly from the following Exchange Online article:
http://help.outlook.com/en-us/140/Ff381292.aspx

In certain situations, this limit can be raised. If you find yourself needing to send out more than 1500 emails in a day, then I would highly recommend contacting Exchange Online support and see if your organization qualifies to have the limit increased.

So what happens when you hit this limit?

When you hit this limit, you will see errors within the Event Viewer in regards to the Email Router. The errors you will see are something similar to this:

MM/DD/YYYY HH:MM:SS : #61042 - An error occurred while processing the outgoing e-mail message with subject "We haven't heard from you... CRM:0002503" for ExchangeOnline: http://CRMSERVER:PORT/UNIQUENAME for delivery through https://pod51018.outlook.com/ews/exchange.asmx. Microsoft.Crm.Tools.Email.Providers.EmailException: Error: An internal server error occurred. The operation failed.

at Microsoft.Crm.Tools.Email.Providers.ExchangeOnlinePollingSendEmailProvider.LogEwsResponseErrorWarning(String message, ResponseClassType responseClassType)

at Microsoft.Crm.Tools.Email.Providers.ExchangeOnlinePollingSendEmailProvider.ProcessMessageInternal(Entity emailMessage)

at Microsoft.Crm.Tools.Email.Providers.CrmPollingSendEmailProvider.Run()

Message and Recipient Limits

The following article discusses additional limits of Exchange Online.

http://help.outlook.com/en-us/140/Dd630704.aspx

There is one specific limit I want to specifically call out as we have already received calls from customers in regards to the Email Router’s performance. This limit is the Message rate limit:

This means that only 30 emails may be sent out in a minute per user. If your organization is configured to utilize an Administrator account to send emails out on behalf of other users, then this limit applies to the administrator account. If you have configured the Email Router on a per user basis, this applies to each user.

Let’s look at an example. Let’s say I sent out an e-mail blast to all of my Accounts. I currently have one thousand accounts in my system so this e-mail blast generates 1000 email messages. I have the Email Router configured to use a central administrator account to send these emails on behalf of my users. For this one blast, it would take approximately 34 minutes to send out all of these emails. However, if I have 10 users and each user owns 100 Accounts, that would mean that each user would send out 100 emails. The email router could process the 1000 emails in about 3.4 minutes.

Due to this limitation, you will noticed that the default configurations of the Outgoing profile is different between Exchange Online and an SMTP service. For Exchange Online, you will see the default maximum messages per cycle is set to 100 and the polling period is two minutes.

Figure 1: Email Router Outgoing Exchange Online profile default Advanced configuration

For an SMTP service, you will notice that the default configurations are radically different:

Figure 2: Email Router Outgoing SMTP profile default Advanced configuration

The maximum messages per cycle is 1000 and the polling period is one minute. So between these two profiles, you can see the Email Router will process ten times more emails while polling half the amount of time when using SMTP over Exchange Online.

CRM Limitations

Email Router Baseline Results

Over time, I have been asked about what type of performance one should expect with the Email Router. I did some “baseline” testing to give some sort of idea of what an organization can expect when utilizing the Email Router. Performance can drastically be different from environment to environment. With this understanding, please read the following disclaimer.

My tests were done on Hyper-V machines on an AMD 2.30 GHz Processor. Each machine was utilizing 4 GB of RAM and set with the architecture type of a 64 bit Windows Server 2008 R2 Enterprise OS. The virtual hard drives were on a backend SAN drive. My environment was configured to utilize five different machines, one for each technology (DC, CRM 2011 UR11, Email Router UR11/ADFS 2.0, Exchange 2010, SQL 2008 R2). These machines are solely utilized for this blog article so no other users/applications/customizations are being utilized. The authentication method I am using is ADFS external claims authentication to connect the Email Router to CRM. The Email Router configuration is using all defaults values within the Configuration Profiles. Your environment will be A LOT different than mine so your numbers may not match up IDENTICAL to my environment. However, the numbers provided below is a good “baseline” of what to expect.

Email Router Outgoing – SMTP

For this scenario, I utilized Exchange 2010 and used clear text for the authentication type. I configured it this way to try and compare apples to apples when we take a look at Exchange Online. See how I configured the profile below:

I created 1000 e-mails using a workflow that emailed 13 different users a very simple message from one user. Prior to creating these emails, I stopped the Email Router service to ensure that all 1000 emails had been created and ready for testing. The Email Router was ONLY processing outgoing messages. Once all 1000 emails were created within CRM and had a status of Pending Send, I started the Email Router service. I repeated this event 3 times and logged the data below:

Again, I want to repeat, your results will vary but I would safely say that the expected results are about 2 emails/sec. The one other note I want to add here. These numbers are when the Email Router submits the emails to Exchange and Exchange accepts the email. This does not include the delivery time it takes for Exchange to actually get these to the end recipient.

Email Router Outgoing – Exchange Online

Now let’s take a look at Exchange Online. I configured the profile to utilize the User account as shown below:

I repeated the same exercise for this test as I did in the one above. You will notice that the Email Router is going to take A LOT longer when processing bulk emails for Exchange Online. Why? Remember those limitations I had mentioned previously? The biggest culprit here is the message rate limit of 30 messages per minute. In addition, the default Configuration Profile for Exchange Online is different than with an SMTP service. This was illustrated above in the Exchange Online limitations.

The reason these profiles are so different is due to the limitations of Exchange Online. Back to the testing, I had to lower the total messages send due to the daily send limit of 1500 messages but you will be able to see the differences in the data captured.

As you can see, it is going to take over 3.5 times longer to send the same amount of emails when using Exchange Online than having an SMTP service available.

Email Router Incoming – Exchange 2010 OnPremise using Exchange Web Services:

In this test, I processed the inbox for just one user in CRM that have CRM emails from the workflows sent in the above tests to promote into CRM. I have configured Exchange to allow a service account to access the user’s mailboxes on their behalf. Please see the Configuration Profile below:

In this case, I allowed the Email Router to process 500 emails and then stopped the service. I then deleted the existing 500 emails in the mailbox and sent 500 more. I then started the service again. This additional stop and start time does cause the Email Router to take more time as the service “warms up”.

At this point, we can notice that the Email Router was able to process an average of 2.63 emails per second for a user’s mailbox.

Again, these numbers can vary with the number of users we have the Email Router processing as well as the number of emails in a mailbox. This test was just ONE user.

Forward Mailbox Incoming – Exchange 2010 OnPremise using Exchange Web Services

I repeated the same test above where I sent 500 emails to one user and that user was configured to use the Forward Mailbox. As per the test with the Incoming, I stopped the service after processing 500 and then started it again after taking my notes.

You can see in this test, the results not as good as using the Email Router to process incoming messages. In this scenario we processed an average of .975 emails per second.

I’m going to provide some more analysis about Email Router versus Forward Mailbox a little later. Let’s look at these same tests for Incoming with Exchange Online.

Email Router Incoming – Exchange Online

I used the identical process and servers but instead of connecting to an Exchange OnPremise environment, I was changing over to Exchange Online.

One can really see that when connecting to a mailbox within Exchange Online, it ends up taking about twice as long to process email messages rather than for an OnPremise environment. Again, I am going to provide more analysis on these details below but keep in mind the Service Providers when reviewing the data.

Forward Mailbox Incoming – Exchange Online

Again, the scenario is the same as OnPremise and we can see we are getting about half of the performance like we saw with just the Email Router. Our average processing is .43 messages per second.

Data Analysis

So now we have all of this data. What can we take away from it? The biggest thing is that these numbers are strictly for example purposes. EVERY environment is going to see some differences. I’ve provide an analysis between Outgoing messages above already so let’s jump into the analysis of Incoming.

You can see that there is a pretty big difference around the processing time for using Email Router for Incoming and Forward Mailbox. Why the differences? Well the biggest reason is how the message is processed. For the Email Router, we are able to read the email directly from the server. When we use the Forward Mailbox, we need to get to the attachment (remember there is a forwarding rule for user’s mailboxes that forward the email as an attachment to the forward mailbox) and read the properties of the attachment. This additional processing takes a little time to do.

At this point, some would say that, at a performance level, the Email Router is better than the Forward Mailbox. This is not always the case. There is a period of time where the Email Router will process emails and will eventually rescan those emails again. For instance, if you stop the Email Router service and start it again, the emails in the user’s mailbox will be gone through again, even the ones that were already processed. The Forward Mailbox offers a cleaner approach. Once the email is processed, it is deleted. The chances of rescanning emails is extremely unlikely.

Now let’s look at the differences between an Exchange OnPremise environment compared to that of Exchange Online. There are variables that are at play here that cause these differences. A big one is network latency. Obviously when I am connected to my OnPremise environment, my Email Router server, CRM server, SQL server, and Exchange server are all on the same LAN. Since it is on the same LAN, I am going to have very low latency and my bandwidth is utilizing our gigabit network. However, as soon as I use Exchange Online, these request must go out to the internet and the throughput can be based on my connection speed. Additionally, there are throttling limits in Exchange Online as we have previously discussed. This plays another big role in the difference between an OnPremise and Online Exchange server. Lastly, notice the Server Providers. We have to use 5 times the number of Service Providers with Exchange Online than with Exchange OnPremise. Each time a Service Provider is completed and a new one must be opened, it can take a little for that to be done.

Now let’s move from these baseline tests and look at specific limitations from CRM.

Email router will only process 5000 users

The PFE team wrote a really good blog about this which can be found here:

http://blogs.msdn.com/b/crminthefield/archive/2012/06/04/over-5000-users-in-the-crm-email-router.aspx

Email from CRM Users sent to Queues won’t promote into CRM

We have seen quite a few questions come in recently that talks about emails from CRM users sent to Queues now showing up in Queues.

This is being caused by the System Setting within CRM to Track E-mails between CRM Users as two activities. If you uncheck this setting, emails from CRM users would no longer promote into a Queue. This setting is a little misleading because Queues and Users are two different entities. Naturally, one would think that this setting has nothing to do with queues. Heck I know I did. However, this was covered in CRM 4.0 in the following KB article:

http://support.microsoft.com/kb/953340

As you can see in CRM 4.0, we included this in a hotfix and a registry key override was needed. Well in CRM 2011, we are moving away from registry entries and moving these changes into the MSCRM_CONFIG database. In order to make this change in CRM 2011, you must change DoNotIgnoreInternalEmailToQueues setting to True with the OrgDBOrgSettings tool. This tool and instructions can be found here:

http://support.microsoft.com/kb/2691237

And on it goes…

These are a few of the bigger limitations of the Email Router. There are other limitations but some of these will be handled in the troubleshooting and performance section. If there are other limitations that come up, I’ll be sure to update this article.

So now we know some of the larger limitations of the Email Router, let’s look at how we can identify and troubleshoot issues with the Email Router.

To Be Continued...

Back to the Intro