Manage data package generation in Logic Apps, for D365FO, using Azure Functions

Subhad365

7 User Group Leader

Like ( 0 )

Report

Background

Managing data operations using Data Packages is one of the very common execution strategies for any D365F&O implementation. A data package for a finance and operations app can consist of one or many data entities. A typical data package consists of a group of entities for a specific task, process, or function. For example, the data entities that are required for general ledger setup might be part of one data package. Importing data that are required for customer creation (e.g. customer group, payment terms, payment methods, reason codes, customers) could be another data package.
It’s a very common requirement to make the data packages be exchanged as a part of integration. We could have an external system that

a. drops files on a shared FTP site
b. drops files on a blob storage
c. sends mailers containing file content
d. queued content containing file information
e. HTTP request having the file content in its payload

We need to create a ‘listener’ that can obtain the file, transform them into necessary FinOps acceptable format and then need to push the same to FinOps, by packaging them as a data package (.ZIP file) for further execution. There are very good blogs that describe the execution of data package once the ZIP file is created, by using logic apps.
https://www.encorebusiness.com/blog/logic-apps-in-dynamics-365-fo-file-based-data-integrations/
https://azureintegrations.com/2020/05/24/azure-integration-dynamics-365-uo-dmf-using-logic-apps/
https://medium.com/hitachisolutions-braintrust/logic-app-integration-with-d365-f-o-524ac4909f0
However this article talks about generating the ZIP file from the incoming trigger that could readily be picked up by the logic app for further executions.

Challenge of using Logic App

We can use logic app to create the ZIP File, that contains the following:
a.       The actual data files (XML/CSV/Excel/any FinOps data management understandable format.
b.       Manifest file (an XML containing the datasources, queries, and fields’ level details).
c.       PackageHeader file (an XML containing the datasources used in the process).
We can design the system so as to accept the incoming files, to transform it to ZIP file, by calling the FinOps APIs, by hardcoding the content of the manifest and PackageHeader XMLs. However the problem is: if tomorrow the system changes to include/exclude an entity out of the existing sequence of entities from the definition group, then we need to redesign the logic app again, so as to incorporate the change. Eventually there comes a technical involvement and dependency.
Why don’t take a step further and keep the entire structure as configurable? The below section illustrates the idea.

Structure of the ZIP file generator

Here we are proposing to keep the Manifest file and PackageHeader file in a predefined container called ‘Definition’. We are using a Blob based trigger Azure function, which can sense the incoming files in a ‘Processing’ blob container and referring to the Definition container and then creating a Zip File out of them. The Zip File would be placed in the ‘Processing’ folder and remove/delete/move the incoming/transformed files to a ‘Processed Folder’.
The Logic App will pick up the ZIP file from the ‘Processing’ folder to perform the rest of the processes.
Validation needs to be included so as to ensure that the logic app doesn’t pick up the ZIP file from the processing folder and the Function app doesn’t take up the Zip file for processing when it gets created in the ‘Processing’ Folder.
Using this approach, if the user wants, then tomorrow he/she can change the content of the Manifest or the PackageHeader file, without much needing to change any underlying code.

Structure of the Azure function

The Azure function is a Blob-trigger, that is having the following parts of the structure

JSON Settings

The various JSON elements that are used, are mentioned as follows:

Note the entry under ‘DefinitionFileContainerName’. This contains the name of the container where Manifest and Package files are kept. The entries ‘ManifestFileName’ and ‘PackageHeaderName’ indicate the name of the manifest and PackageHeader file names.

As you’re aware that the JSON settings are not deployed in the Azure Function, hence we need to create the above mentioned JSON entries as ‘Environment variables’ of the Function configuration:

Creating the ZIP file

The following async method takes care of the ZIP file creation:
public async static Task CreateAndUploadZipFile(string fileName, string containerConnectionString)
        {
            string definitionContainer = Environment.GetEnvironmentVariable("DefinitionFileContainerName");
            string manifestFileName = Environment.GetEnvironmentVariable("ManifestFileName");
            string packageHeaderName = Environment.GetEnvironmentVariable("PackageHeaderName");
            string referenceManifestContainerConnectionString = Environment.GetEnvironmentVariable("DefinitionConnectionString");


            CloudBlobContainer container = GetContainerReference(referenceManifestContainerConnectionString, definitionContainer);
            SetPermissions(container);
            CloudBlockBlob blobManifest = container.GetBlockBlobReference(manifestFileName);
            CloudBlockBlob blobPackageHeader = container.GetBlockBlobReference(packageHeaderName);

            CloudBlobContainer triggerContainer = GetContainerReference(referenceManifestContainerConnectionString, "processing");
            CloudBlockBlob blobOrgFile = triggerContainer.GetBlockBlobReference(fileName);
            SetPermissions(triggerContainer);

            using (var memoryStream = new MemoryStream())
            {
                using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
                {
                    await AddToZipFile(archive, manifestFileName, blobManifest);
                    await AddToZipFile(archive, packageHeaderName, blobPackageHeader);
                    await AddToZipFile(archive, fileName, blobOrgFile);
                }

                Guid obj = Guid.NewGuid();
                string outputFileName = String.Format("{0}.zip", obj.ToString());

                CloudBlockBlob uploadfileRef = triggerContainer.GetBlockBlobReference(outputFileName);

                memoryStream.Seek(0, SeekOrigin.Begin);
                await uploadfileRef.UploadFromStreamAsync(memoryStream);

                await blobOrgFile.DeleteAsync();
            }


        }
The explanation for the above code is as follows:
a.       cloudBlobContainer is the reference for the container storing the manifest and the PackageHeader files.
b.       triggerContainer is the reference for the container where the incoming file/transformed file comes. This is where the function app is listening to.
c.       Note the code:
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
                {
                    await AddToZipFile(archive, manifestFileName, blobManifest);
                    await AddToZipFile(archive, packageHeaderName, blobPackageHeader);
                    await AddToZipFile(archive, fileName, blobOrgFile);
                }

Here we are using the ZipArchive class, which is a part of the System.IO.Compression interface. Here you are adding your necessary files to the ZipArchive file bucket, by calling the aysnc function ‘AddToZipFile’
Here, though for example purpose, I am showing only one file, but you can make the system loop through all the incoming files at a given moment of time, and add them to the Zip Archive. However this article will not talk about the logic of transposing the files- will talk about them on a future blog.
Here goes the structure of the AddToZipFile method:

public static async Task AddToZipFile(ZipArchive archive, string fileName, CloudBlockBlob blob)
        {
            var zipFile = archive.CreateEntry(fileName, CompressionLevel.Optimal);
            using (var entryStream = zipFile.Open())
            {
                ManageDownloadBlobs dnlnBlob = new ManageDownloadBlobs();

                var result = await dnlnBlob.DownloadAsync(blob);


                using (var fileToCompressStream = new MemoryStream(result.ToArray()))
                {
                    fileToCompressStream.CopyTo(entryStream);
                }

            }

        }
Look at the method: downloadAsync(blob). This is another async method which is awaited till the entire file stream is downloaded in the ‘var result’ variable. It’s structure is as follows:
public async Task<MemoryStream> DownloadAsync(CloudBlockBlob blob)
        {
            using (var stream = new MemoryStream())
            {
                await blob.DownloadToStreamAsync(stream);
                return stream;
            }
     }
Note the return type of the method: Task<MemoryStream>. It makes the original caller to await till the entire stream is returned. Hence we are using:
var result = await dnlnBlob.DownloadAsync(blob);

Once the ZIP file is made, it’s uploaded to the target container, by creating name of the file with GUID:
Guid obj = Guid.NewGuid();

In the example above, we are deleting the source file. But if needed, we can move it to some 'Completed' folder.

Validation

We can suppress the function app not to read the ZIP file, by additionally writing:
if (! name.EndsWith(".zip"))
            {
                CreateAndUploadZipFile(name, connectionString);
                log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
            }

Conclusion

Once the ZIP file is generated and dumped in the necessary container, we can await the Logic App to take it up and process from hereafter. We can ask the Logic App, not to consider anything else other than ZIP File:

Comments

*This post is locked for comments