Exporting Application Insights logs with Continuous Export or Diagnostic Settings

I’m continuously emphasizing that for monitoring cloud applications (like Dynamics 365 Business Central) and cloud services, using Azure Application Insights is a must.

For analyzing telemetry data, I‘ve talked in the past about how to use Log Analytics and the features available in the Azure Portal, how to use Powershell, how to use Power BI and more. But what to do if you want to extract the Application Insights telemetry data and use these data in an external application for different kind of tasks?

This is where a not so well known feature like Continuous Export comes in help. With Continuous Export you can export your telemetry data on an Azure Storage in JSON format and from there you can download the data stream and then process it externally with the tools you want.

Continuous export is only supported for classic Application Insights resources. Workspace-based Application Insights resources must instead use diagnostic settings (recommended new way of handling telemetry export).

Using the old Continuos Export feature is quite easy. Select your Application Insights instance, then go to Configure and select the Continuous Export option:

Here click on Add and then select a storage account (where telemetry data export will be saved) and the telemetry data type that you want to export:

Now Continuous Export is configured. Telemetry data will be exported in JSON format into your storage account. You will find a folder in the following format:

<ApplicationInsights_ServiceName>_<ApplicationInsights_InstrumentationKey>

with subfolders for each exported data type. Under the data type folder, you will find subfolders for date and then subfolders for hours. Inside this last subfolder you will find a JSON file with the log data.

Continuous Export is a feature currently available but please remember that it has been deprecated. Now you should migrate to a workspace-based Application Insights resource and use diagnostic settings for exporting telemetry.

Workspace-based resources support full integration between Application Insights and Log Analytics. Workspace-based resources send Application Insights telemetry to a common Log Analytics workspace, which allows you to access the latest features of Azure Monitor while keeping application, infrastructure, and platform logs in a single consolidated location.

To use the new data export way, you first need to create a Log Analytics workspace in the Azure Portal and then (if your Application Insight resource was created using the Classic template) you need to migrate it to the new workspace-based resource type. Please remember that if you’ve previously activated the Continuous Export feature, you need to disable it in order to migrate to workspace-based configuration.

To migrate, select Configure –> Properties and then click on Migrate to Workspace-based:

Once your resource is migrated, you will see the corresponding workspace info in the Application Insights instance Overview pane. Clicking the blue link text will take you to the associated Log Analytics workspace where you can take advantage of the new unified workspace query environment.

To write queries against the new workspace-based table structure/schema, you must first navigate to your Log Analytics workspace.

Please remember that when you query directly from the Log Analytics UI within your workspace, you will only see the data that is ingested post migration. To see both your classic Application Insights data + new data ingested after migration in a unified query experience, use the Logs (Analytics) query view from within your migrated Application Insights resource.

Once you have successfully migrated your Application Insights resource to a workspace-based one, you can use Diagnostic settings to replace the functionality that continuous export used to provide. Under Monitoring, select Diagnostic settings –> add diagnostic setting from within your Log Analytics workspace instance:

A diagnostic setting specifies a list of categories of platform logs and/or metrics that you want to collect from a resource, and one or more destinations that you would stream them to. Here you can for example select to export your telemetry data to a Storage account with a desired data retention policy (from 1 to 365 days. If you do not want to apply any retention policy and retain data forever, set retention (days) to 0.):

With both ways, you have all your data exported externally as JSON format in a Storage Account. From here, you can do what you want for automating the telemetry data processing.

In a real world scenario, for example I’ve created a Blob Trigger Azure Function that, when a telemetry file is saved into a storage account, the file is automatically parsed as needed (something like in the following code):

This is also useful if you want to preserve your telemetry data logs indefinitely (bypassing the standard retention policy of the Application Insights service).