Running RSAT within a DevOps pipeline
Like
Share
ReportNow that OneVersion is here and providing us with so many benefits, it’s also safe to say that the update cadence is presenting many customers and partners with a nice challenge to test every update in a timely fashion. Regression testing is now more than ever a crucial recurring activity with every update. Luckily, we have the Regression Suite Automation Tool (RSAT) to help us to do that. Using the Task Recorder, we can record user interactions and feed those recordings to the tool. From there we can automatically playback those recordings and perform the necessary validations.
At this point in time, the tool has been adopted by many customers and partners improving overall efficiency of regression testing. But can we take this a step further? The past few weeks I’ve received the following question a number of times: Can we make RSAT an integral part of our DevOps pipelines? The answer of course is yes! And that is exactly what we are going to do: Include RSAT in a build pipeline.
Before we dive in, let’s make things clear. The goal of this article is to show how it can be done. Whether we should do it as a part of the build pipeline is a different discussion. The scenario for this post is to have RSAT executed as the testing part of a build pipeline (Not necessarily a CI build, but think of a build pipeline used before initiating deployment to other environments). Another good scenario is after deployment of a build to a testing environment, run the RSAT as a test cycle on that environment.
Preparing the build machine
Currently we are using the build VM to host our build agent so we can leverage that VM to install and configure RSAT. We will also need some scripts to automate RSAT execution and pass the testing results back to the pipeline. These scripts can be found at the bottom of this article. The installation of RSAT itself is described very well in the documentation that comes with the tool so at this point it is assumed you have RSAT configured and can run the tool on the build machine.
Perform the following steps on the build machine to prepare it for running RSAT: (Any needed scripts will be in the links later in this post)
- Create an environmental variable that points to the RSAT installation folder.
- Create a folder containing the following files:
- BuildSettings.settings
This file can be created by saving your active RSAT settings from the tool to this folder.
This will be passed to the console app during build. - RSATBuildAutomationCommon.psm1
Contains some parsing logic to parse the results from the log files RSAT is going to output. - RunRSATExecution.ps1
Instruments the RSAT console application to run the specified test suites.
- BuildSettings.settings
Implementing RSAT into the Azure DevOps Pipeline
For our scenario here, we are using a new Pipeline with only one step to run RSAT.
To put it in the Dynamics 365 For Finance And Operations build pipeline it is exactly the same step.
Now first go to the Options tab page and add a Demand to make sure only agents are selected that have the DynamicsRSATFolder environment variable. Also add this demand to the existing build pipeline if you are running RSAT in there.
Then go back to the tasks page and add a PowerShell task with the following parameters:
- Script path
We are using the environment variable to get the location where RSAT was installed and point to the BuildAutomation folder containing our scripts. - Arguments
The following options are available for use:- TestSuitesToRun
Only mandatory parameter containing a comma separated list of test suite names to run. - RSATInstallationPath
Use when you want to deviate from the environment variable. - SettingsFilePath
Use to specify a specific settings file path. - LogFolder
Use to change the default logging folder.
- TestSuitesToRun
So let’s take a look at how this looks like during pipeline execution.
Marked in red is the actual output you would get from the ConsoleApp.exe executable.
The area marked in blue is where we parse the actual results from the log files. In case of errors, the build will fail.
That’s it, all done!
Comments
-
-
Hi Kenny Saelen, Thank you for Implementing RSAT on build pipeline Article. In my current, Im working implementing similar solution. As a part of this, I have follow the same steps that you mentioned. when ran the pipeline, Im getting the below error. "No agent found in pool Default which satisfies the following demand: $(DynamicsRSATFolder). All demands: DotNetFramework, msbuild, $(DynamicsRSATFolder), Agent.Version -gtVersion 2.144.0 " The following steps that i have perform. 1. On the Build server(VM), I have created a environment variable for RSAT application file path and name it as "DynamicsRSATFolder" 2, I have created a folder(BuildAutomations) in RSAT Application folder and copy the .ps1 files(RSATBuildAutomationCommon &RunRSATExecution,) that you provided in the end of the Article and RSAT configuration settings. file 2. I have created new demand "DynamicsRSATFolder" under options tab in build pipeline, 3. In the PowerShell task, Script path -->$(DynamicsRSATFolder)\BuildAutomations\RunRSATExecution.ps1 Arguments --> -TestSuitesToRun "WISTRA RSAT TEST" -SetttingsFilePath $(DynamicsRSATFolder)\BuildAutomations\BuildSettings.settings -Verbose But I'm getting error . Can you help me on this plz
-
-
I tried to run the pipeline similar way but getting below error, if any help on this is much appreciated. 2021-05-20T14:27:52.0316453Z 2:27:52 PM: Working directory: (C:\Program Files (x86)\Regression Suite Automation Tool) 2021-05-20T14:27:52.0325644Z 2:27:52 PM: ------RSAT Console App output START------- 2021-05-20T14:28:07.1775747Z Test run results: 2021-05-20T14:28:07.1834707Z Test 57109: Not executed. Error message: 2021-05-20T14:28:07.1850669Z Excel file C:\RSAT\57109\attachments\Create_BOM_Assembly_57109_Base.xlsx is invalid 2021-05-20T14:28:07.1851517Z 2021-05-20T14:28:07.1855995Z System.Runtime.InteropServices.COMException (0x80070520): Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80070520 A specified logon session does not exist. It may already have been terminated. (Exception from HRESULT: 0x80070520).
-
@Denis trunin Depends on what the scenario is where you use it. In case of using it during the build process, the step will fail in the build pipeline when errors were seen during testing. At that point, it is up to the dev team to look into the DevOps test results to see why it fails as the results will be visible in the test run.
-
@Jinmin Zhou This post was to show how it could be done is a build when devs would like to fail build if RSAT fails. The ideal setup would be to use RSAT in a release pipeline where you deploy a package to a TST environment and where you have a pipeline agent installed on the machine and RSAT configured. Then after the deployment of the latest build to TST, you can run RSAT on that TST environment. As for the database backup, restoring a DB is indeed recommended to start with a solid baseline. Although I have also seen cases where teams make the test self sufficient in terms of data and the first sets of test actually create the data the subsequent tests will need. This off course is entirely up to you to decide how you want to do it.
-
-
60
*This post is locked for comments