Server-Side Synchronization Series Part 2: How Server-Side Sync reaches out to Exchange

This second part of the Server-Side Sync blog series will be going over how we reach out to Exchange and what properties are used to determine how far we go back in our search for Exchange items. This part will just be going over Incoming Email, and we will cover ACT in more detail later in the series.

How does Dynamics schedule the interval timings to reach out to Exchange? Do all users reach out at the same time?

When test/enable is run on a mailbox for all three Sync methods (Incoming, Outgoing, and ACT), Dynamics 365 reaches out to Exchange through servers running the Asynchronous Processing Service. For online, this means that certain server clusters take turns sending requests to the Exchange environment specified in the Email Server Profile (in the above setup, this is Exchange Online). We send several different SOAP requests to Exchange, such as finditem, getitem, senditem, and syncfolderhierarchy. We send a test email to the user's own mailbox as part of the steps when performing the test. A Full list of EWS operations can be found here: 

https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/getitem-operation

After test/enable completes these operations, the "Configuration Test Results" screen above will change to "Success" and a bunch of values are modified on the associated MailboxBase table for the user. Please note the button labeled "Download Mailbox Details" in the header of the mailbox below:

This button makes a call to the MailboxBase and returns a ton of useful column data that will tell you the next time the user Mailbox record will reach out to Exchange:

Mailbox Async Processing State
mailboxid : <>
hostid: Null
processingstatecode : 0
processinglastattemptedon : 9/9/2022 12:39:57 PM

Mailbox Synchronization Methods (2 is server-side sync or email router, 1 is Outlook sync, 0 is none)
incomingemaildeliverymethod : EmailRouter
outgoingemaildeliverymethod : EmailRouter
actdeliverymethod : EmailRouter

Mailbox Enabled State
enabledforincomingemail : True
enabledforoutgoingemail : True
enabledforact : True

Mailbox Idle State (shows how many times the mailbox was processed with no email items or Appointments, Contacts and Tasks to synchronize)
noemailcount : 175
noactcount : 1422

Mailbox Backoff Parameters (shows when the mailbox is scheduled to process next time)
postponemailboxprocessinguntil : 2/23/2023 2:45:18 PM
postponesendinguntil : 1/30/2023 9:47:58 PM
receivingpostponeduntil : 2/23/2023 2:45:18 PM
receivingpostponeduntilforact : 2/23/2023 2:57:09 PM

A full list of each value and how to read them to determine when Server-Side Sync will fire next can be found on my blog post here:

How to read Download Mailbox Details

It's important to note that the "postponemailboxprocessinguntil" value will have 5 minutes added to the exact time the mailbox completes processing, so will always have at least 5 minutes between each Exchange check (time that it takes sync to complete + 5 minutes).

 So were reaching out to Exchange on a regular schedule now, but what does this actually do?

The value we want to focus on for this part is the Dynamics MailboxBase field "ProcessEmailReceivedAfter". This value is not on the default mailbox form, but can be added or queried through the mailbox entity API. This timedate value is where we store the received time of the last email we successfully promoted from Exchange. We use this timedate field value when we ask Exchange for Emails within the inbox:

It's important to understand this field to understand what we ask Exchange for when we reach out every 5 minutes. Here is the back and forth conversation we have with Exchange each time we reach out for incoming email:

Dynamics: Hey Exchange, can you provide me emails within the inbox of the mailbox "example@bacon.edu" since we last spoke at the (ProcessEmailReceivedAfter value)?
Exchange: Prove who you are; provide your authentication. (401)
Dynamics: Here are my credentials
Exchange: Here are all the emails within the inbox since that time
Dynamics: Awesome, please provide me the details of each email, such as subject, body, and party information
Exchange: Here are all the fields you requested

If we found emails to promote to Dynamics (such as a reply to an item in Dynamics), we will update the "ProcessEmailReceivedAfter" time to the exact received time of the most recent email we promoted. The next time we reach out to Exchange, we will only query for emails since that time, since we know nothing new has entered the Inbox before that value.

But wait, if we only check for emails back to a certain time, what if we move items from a subfolder to the inbox? Wouldn't they be missed since their received times are before what we have set?

That is correct. If you move emails from a subfolder and we have promoted an item that arrived after the received time of the email you're moving, we will never evaluate this moved item on subsequent cycles. This is where this field comes in; you can manually modify this time for the specific user context, causing our next sync cycle to go back farther in time to evaluate emails. This would allow the sync session to find the email moved from the subfolder in this scenario.

Wont we also reprocess all the other emails still in the Inbox that we already evaluated? Wont this make duplicates?

We would also evaluate all of the other emails with received times after this timedate value, but duplicate emails cannot occur due to the "MessageID" field on the emails in Exchange. Each email has a unique Message ID, which we pass as part of the email creation within Dynamics.

On the Dynamics SQL side, "MessageID" has a duplicate key constraint, meaning that the column can only have unique values (aside from one specific exception, which we will discuss in a later blog series). This prevents duplicate emails from being created in Dynamics even if they are reprocessed.

The next part of our blog series will go over correlation methods, which is how the emails in the above scenario were automatically promoted based on how the user is configured.

Part 1, Part 3, Part 4, Part 5, Part 6