Large data integration with D365F&O leveraging Azure CosmosDB and Az Functions

Scenario: 

When should you use Cosmos Db:

• Azure Cosmos DB has SSD backed storage with low-latency order-of-millisecond response times.
• Azure Cosmos DB's support for consistency levels like eventual, consistent prefix, session, and bounded-staleness allows for full flexibility and low cost-to-performance ratio. No database service offers as much flexibility as Azure Cosmos DB in levels consistency.
• Azure Cosmos DB has a flexible data-friendly pricing model that meters storage and throughput independently.
• Azure Cosmos DB's reserved throughput model allows you to think in terms of number of reads/writes instead of CPU/memory/IOPs of the underlying hardware.
• Azure Cosmos DB's design lets you scale to massive request volumes in the order of trillions of requests per day.

• Low Response time
• Needs to handle massive data throughputs

Creating a CosmosDB:

Reach out to https://portal.azure.com and search for Cosmos DB:

In the screen that opens, choose Azure NOSQL >> fill out the form as follows:

Select Subscription, Resource group, Location. Optionally you could also choose to apply Free Tier Account.

Click Review + Create to create/deploy a Cosmos DB account.  

It will take a while to create and make your DB ready. As stated above here are the Todolist and Item, which got created:

You can create records, or upload a bulk file into the CosmosDB, in just no time:

I have prepared a Json file beforehand to feed it into the Cosmos DB:

Remember: there should be an id column, which should be unique, rest whatever fields you keep in your JSON file, it will accept that. After Upload, I am querying the CosmosDb:

It's showing the records in JSON format. Alternately you can directly see all the records, by clicking on items themselves:

Querying a CosmosDb

You can create StoredProcedure or UDFs also, if you want to make things more easy or reusable. 
 

Solution Design

Imagine we have a record structure that fills up a table: SalesPool, which is having the following fields:

Fan-in/Fan-out functions:

Start with Visual studio and create a Durable Function Orchestrator like this:

Step 1:

Create a class to generate token like this:

using Azure.Identity;

using Azure.Security.KeyVault.Secrets;

using System;

using System.Collections.Generic;

using System.Collections.Specialized;

using System.Linq;

using System.Net;

using System.Text;

using System.Threading.Tasks;

using Azure.Security.KeyVault.Secrets;

using Microsoft.Extensions.Logging;

using Newtonsoft.Json.Linq;

namespace DF_SalesPool_MultiThreadedDemo

{

    internal class GenerateToken

    {

        public string GetSecretValue(string secretName)

        {

            string kvUri = Environment.GetEnvironmentVariable("VAULTURI");

            var client = new SecretClient(new Uri(kvUri), new DefaultAzureCredential());

            var secret = client.GetSecretAsync(secretName).GetAwaiter().GetResult().Value;

            return secret.Value;

        }

        public string GetToken(ILogger log)

        {

            string responseInString = String.Empty;

            using (var wb = new WebClient())

            {

                GenerateToken kvManager = new GenerateToken();

                string tokenUrl = kvManager.GetSecretValue("tokenurluat");

                var data = new NameValueCollection();

                data["client_id"] = kvManager.GetSecretValue("clientiduat");

                data["client_secret"] = kvManager.GetSecretValue("secretvaluat");

                data["grant_type"] = "client_credentials";

                data["Scope"] = kvManager.GetSecretValue("scopeurl");

                var response = wb.UploadValues(tokenUrl, "POST", data);

                responseInString = this.getTokenValue(Encoding.UTF8.GetString(response));

            }

            return responseInString;

        }

        public string getTokenValue(string responsePayload)

        {

            dynamic json = JObject.Parse(responsePayload);

            string accessToken = json.access_token;

            return accessToken;

        }

    }

}