Azure Functions cold start: impact on Dynamics 365 Business Central webhooks

I think that many of you know what Dynamics 365 Business Central webhooks are. In few words, webhooks is the way to get notified if an entity changes in Dynamics 365 Business Central, exactly like having a push notification mechanism. I talked about Dynamics 365 Business Central webhooks here if you want to know more.

Webhooks in Dynamics 365 Business Central are absolutely a cool feature in theory, but during real-world projects they have some limitations that you shouls be aware of. The main limitations I’ve found in real world projects using Dynamics 365 Business Central webhooks are the following:

• You cannot subscribe to entity modifcations for only some fields you’re interested on. For example, if you’re interesting to be notified only if the Description of the Item entity is changed, this is not possible (you are notified for all Item entity modifications).
• Changes made by users who are not able to schedule job queues will not be triggered until another user who is able to schedule job queues makes another change to the same table.
• The subscription flow (start or renewing a subscription) is not always reliable (can be affected by your notification url availability).

In this post I want to talk about this last point because it’s a topic where you can work in order to have a reliable subscription flow.

Why the webhooks subscription flow is not always reliable?

As explained in my previous post, to register a webhook subscription, you need to send a POST request to the following endpoint, with Content-Type = application/json:

POST https://api.businesscentral.dynamics.com/v2.0/TENANTID/production/api/v2.0/subscriptions

and by passing the following JSON object in the body:

{

  "notificationUrl": "YourWebhooksListenerURL",

  "resource": "companies(COMPANYID)/salesOrders",

  "clientState": "SomeSharedSecretForTheNotificationUrl"

}

Here:

• notificationUrl is the url of your service listening for webhooks notifications.
• resource is the API address of the Dynamics 365 Business Central resource that we want to subscribe (in this case I want to subscribe the salesOrders API). It’s important to remember that the subscription is per-company, so you need to provide the company id.
• clientState is optional and can be included as an “opaque key”, enabling the subscriber to verify notifications.

When the POST request is issued to the subscription API, Dynamics 365 Business Central sends a request to the notificationUrl endpoint by passing a validationToken parameter in the query string.

The subscriber service retrieves this token from the query string and then it sends back to the caller (Dynamics 365 Business Central) a response with a body that contains the validationToken and by setting http status code = 200 (success).

When Dynamics 365 Business Central receives the response, it checks for the validationToken and if all is ok the subscription is established.

The same flow is for the renewal process. Webhooks subscriptions expires in 3 days (default value). They can be renewed by sending a PATCH request to the subscriptions API (as in the following URL) with Content-Type = application/json:

PATCH https://api.businesscentral.dynamics.com/v2.0/TENANTID/production/api/v1.0/subscriptions(SUBSCRIPTIONID)

Where’s the problem here?

As said before, when you send a request for establishing a webhook subscription to Dynamics 365 Business Central, the ERP sends a request to your listener endpoint (your service that needs to receive notifications) with a Validation Token and your service needs to respond within 5 seconds with HTTP status 200 and including the value of the Validation Token parameter in the response body.

And if your service is down? If your service is not able to respect this timing? You loose notifications!

Ok… and your answer could be: I’m sure that my listening service is always active because I’m using Azure Functions for this. In my previous post about webhooks, I received some questions by partners regarding this topic and they asked me why their Azure Functions listening service were not reliable.

I’ve done a quick analysis with some of them and the cause of the notifications lost was one: Azure Functions cold start.

The majority of partners that uses Azure Functions in their projects are using the Consumption model. Azure Functions running in the Consumption plan scale automatically and you pay for what you use (functions execution). But Azure Functions running in the consumption plan have what is called a cold start, a small period of time that the Function App needs in order to warm up and start the execution of your code.

Why this?

Because in the Consumption model your Functions run in a shared environment and when triggered they need compute resources to be allocated. When a Function is executed, it will stay active (warm) for about the next 20 minutes for executing subsequent requests.

The cold start of a Function app in the Consumption plan typically ranges between 1 and 3 seconds but sometimes can be more than this (and honestly I see that often is much more than this).

What happens if you receive a notification from Dynamics 365 Business Central and your Azure Function app is not warm? You loose that notification.

How can I avoid Azure Functions cold start?

If you want a reliable webhooks handshake with Azure Functions as listener, you need to avoid the cold start problem. To avoid Azure Functions cold start you can:

1. As a developer, you can reduce the cold start time by keeping your package size as small as possible and keep your code execution as short as possible
2. Keep your functions warm.
3. Create Azure Functions in the Premium plan.

I don’t want to talk about point 1 here (this is only code-related) but I want to focus on point 2 and 3 that in my opinion are the best way to reduce cold start in Azure Functions and so solving lots of your problems with Dynamics 365 Business Central webhooks notifications.

If your listener Azure Function app need to be in the Consumption model, cold start is reality and you need to keep your function warm if you want to avoid it. The most cost-effective way to do that is using Azure Monitor Availability Tests.

With Azure Monitor Availability Tests you can “ping” an url with a timimg you want and this can help on maintaining your function app warmed. These tests are nos simple pings, they don’t use the Internet Control Message Protocol (ICMP) to check your service’s availability but instead they use more advanced HTTP request functionality to validate whether an endpoint is responding. These ping tests are absolutely free!

To create an availability test with Azure Monitor, select an Azure Application Insights instance, then click on Availability and then on Add Classic test:

Now create your test. Select URL ping for SKU, insert your Azure Function url and select the test frequency (how often the test is run from each test location):

When you create an availability test, it’s important to test from multiple locations. If you click on Test locations you can select the Azure Regions where web requests are sent to your URL. It’s recommended to have a minimum of 5 test locations to ensure that you can distinguish problems in your website from network issues. You can select up to 16 locations:

Click on Create and your availability test will be deployed and executed:

You can monitor the availability test results directly from the test itself:

In this way your Azure Function app is “pinged” every 5 minutes from multiple locations and it’s maintained always active also if it’s deployed in the Consumption plan and if Dynamics 365 Business Central is not sending requests from days. With a default frequency of 5 minutes and 5 test locations, your Azure Function app is tested every minute on average. And this is all free!

The other possible option to avoid cold start is using Azure Functions Premium plan. With the Premium plan, your Function apps are scaled automatically and you have always at least one instance of the Function app maintained as warm (active).

To deploy an Azure Function app in the Premium plan, create a new Function app and select Plan Type = Functions Premium:

The Sku and size option indicates the compute resources that you reserve for this Function App. You can leave the Sku and size with the default value because you can scale it later if needed.

When your Function app is deployed in the Premium plan, if you select the Scale out (App Service Plan) menu, you can see the following:

In the Premium plan, you have always at minimum 1 instance always ready and warm up for this function app (and you can increase this number if needed). You can also specify the number of instances to scale out (maximum burst). With this setting, the function app is always executed in a pre-warmed instance and it always starts immediately.

This is a chart that shows the measure of cold start (in seconds) when using one of my Azure Functions in Consumption and Premium tier:

As you can see from the chart, the Consumption model has cold start that often is above 5 seconds while the Premium plans always responds in milliseconds (an instance is always warm).

The Premium Plan is absolutely a powerful plan to run your Azure Functions in, expecially in production environments and when you need high performances and always-on instances. You can use it to eliminate cold starts and it still have the automatic scaling option of the Consumption plan.

What about Azure Functions Premium plan pricing?

Billing for the Premium plan is based on the number of core seconds and memory allocated across instances. There is no execution charge with the Premium plan. At least one instance must be allocated at all times per plan. When creating or scaling your plan, you can choose between three instance sizes. You will be billed for the total number of cores and memory provisioned, per second that each instance is allocated to you. Your app can automatically scale out to multiple instances as needed.

Acrtual prices for the Pay as You Go option in the Premium plan are the following:

• vCPU: $0.173 vCPU/hour
• Memory: $0.0123 GB/hour

You can also move a Function app from Consumption to Premium plan on busy periods and then move it to Consumption when needed to save costs. Here is a simple Azure CLI cmdlet that does that:

az functionapp update --plan Your_AF_Premium_Plan -n YourFunctionApp

Please remember these considerations because these are hidden things that can affect your experience with Dynamics 365 Business Central webhooks.

Saying that, I think that there are other (and sometimes more reliable) ways to send events from Dynamics 365 Business Central to external applications. If you will be next week in Hanburg in the session I have at Directions EMEA 2022 about real-world Dynamics 365 Business Central and Power Platfom architectures, you will see something about that too.