Azure Data Factory – Manually Deploy to other Environments using ARM Templates

www.RahulSharma.in

508

Like ( 0 )

Report

When we are working with Azure Data Factory (ADF), best is to setup a development environment with DevOps (Git) for CI/CD but sometimes you might want to deploy it manually. Deploying in ADF means moving the ADF Pipeline from one environment to other environments (development, test, prod). ADF does this by using the ARM Templates to store various ADF entities like pipelines, data flows, datasets, linked services etc.

An ARM template is a JSON file that defines the infrastructure and configuration for your project.

We can move ADF to another environment using below methods:

Manually upload ARM Template using Azure Portal
ADF integration with Azure Pipelines

How to configure Git repository?
In case, you want to take a look into Git setup.

In this post, we will do it using the manual method.

Also read: How to Create Azure Data Factory for Azure SQL DB to Azure File Storage Pipeline

Manually Deploy Azure Data Factory using ARM Templates

Below are the brief steps to export and import the ADF.

On Azure Portal, go to your development ADF environment and export the ARM template using Export ARM Template option.

Go to your Test / Prod Azure subscription, and open Deploy a Custom Template service (direct link). This will help us import the exported template.

Select Build your own template in the editor to open the Resource Manager template editor.

Select Load File, and upload the exported ARM template file (arm_template.json) from step 1. You can see the code for your ADF is all about JSON. If you want, you can change any of the parameter values at this point or on the next screen so that it points to the TEST/PROD resources.

At this point, if you already have ADF in the target environment it will deploy ADF entities like pipelines, datasets etc., else it will fail with below error.

The Resource ‘Microsoft.DataFactory/factories/RSDataFactory’ under resource group ‘ResourceGroup’ was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix\
ERROR DETAILS

Create an Azure Data Factory using ARM template

The above error comes because arm_template.json doesn’t include the Data Factory creation configuration. But don’t worry, it is an easy fix. You can find a folder called “factory” in the ARM Template zip file we downloaded in step 1, which contains data factory ARM Template file. You can either use this template to create the factory first or modify the main arm_template.json file to import/create everything with one file.