Skip to main content

Notifications

Announcements

Announcing Category Subscriptions!
News and Announcements icon
Dynamics 365 Community / Blogs / Discovering D365 for Finance and Operations tec... / Build process failing in Pr...

Build process failing in Prepare For Build step

nspiteri Profile Picture nspiteri 541

The Problem


I like to run the automated build pipelines in D365 as these make releases smoother when you have multiple persons/teams working together.

There is a constant frustration that I have been having which is of failures in the Prepare for build step which cause the whole process to fail.

If you have a build process with multiple models that might take over 2 hours to do, you're generally going to run it out of office hours and the last thing you want is to show up in the morning with a broken build.

There are 2 areas where the Prepare for Build generally fails:

a) when its stopping the Batch service (either due to a timeout or due to the service not accepting messages)

b) when its taking a backup of the models to your DynamicsBackup folder (normally J:\DynamicsBackup\Packages)

After a while I decided it was time to make this step a bit more robust.

The Proposed Solution

The following is a solution that I present to this problem.  It involves some modification to the powershell files in your build machine (normally under c:\DynamicsSDK).

Let's start!

The two issues stem from the way the processes are being shut down and what processes are being shut down.  In script DynamicsSDKCommon.psm1 there is a function called Stop-AX7Deployment.

This first stops IIS through a call to another function Stop-IIS, next it loops and closes 3 other services: DynamicsAxBatch, SSISHelper (DIXF) and MR2012Process (Management Reporter).

There are 2 flaws with this process:

a) it is stopping IIS but not stopping IISExpress (the error on the backup is caused because IISExpress is locking files)

b) it has no error handling if any of the services fail to stop, or fail to stop on time (it only gives them 30 seconds to stop and then it crashes out)

The fix:

1) To get error handling to work correctly we need to change the ErrorActionPreference to stop within the function, we can then change it back in the end.

In the beginning of the function add:

    #we need to temporary change the action preference, so we save the original value
    $old_ErrorActionPreference = $ErrorActionPreference
    $ErrorActionPreference = 'Stop'

In the end of the function add:

    #Resetting the action preference to the original value
    $ErrorActionPreference = $old_ErrorActionPreference

2) Change the timeouts to be longer (I suggest 120 seconds) by editing the StopWait variables:

    Param([int]$ServiceStopWaitSec = 120, [int]$ProcessStopWaitSec = 120)

3) If iisexpress isn't closed, close it.  Do this by writing the following after Stop-IIS

    # Need to stop iisexpress as well.
    $process = Get-Process -Name 'iisexpress' -ErrorAction SilentlyContinue
    if($process -ne $null)
    {
        Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue
        Start-Sleep -Seconds 1 # Wait 1 second for good measure
    }

4) Create a try catch right after the condition that checks if the service is stopped:

            if ($Service.Status -ne [System.ServiceProcess.ServiceControllerStatus]::Stopped)
            {
                try
                {

When catching the error do the following, basically if the service fails to stop, just kill off the process itself.

                catch [Exception]
                {
                    $ErrorMessage = $_.Exception.Message
                    Write-Message "- Exception catched on $($DynamicsServiceName) " -Diag

                    $processName = switch ( $DynamicsServiceName )
                    {
                        'DynamicsAxBatch' { 'Batch' }
                        'Microsoft.Dynamics.AX.Framework.Tools.DMF.SSISHelperService.exe' { 'Microsoft.Dynamics.AX.Framework.Tools.DMF.SSISHelperService' }
                        'MR2012ProcessService' { 'MRServiceHost' }
                    }
                    #After the stop timeout, by the time it reaches here, the process is already dead.
                    #So we check if still exists and if it does we do a dirty kill.
                    $process = Get-Process -Name $processName -ErrorAction SilentlyContinue
                    if($process)
                    {
                        Stop-Process -Name $processName -Force -ErrorAction Stop
                    }
                }

The full function

You can copy the full function for pasting later over here:

function Stop-AX7Deployment
{
    [Cmdletbinding()]
    Param([int]$ServiceStopWaitSec = 120, [int]$ProcessStopWaitSec = 120)

    # Get verbose preference from caller.
    $VerbosePreference = $PSCmdlet.GetVariableValue("VerbosePreference")

    #we need to temporary change the action preference, so we save the original value
    $old_ErrorActionPreference = $ErrorActionPreference
    $ErrorActionPreference = 'Stop'

    # There are a number of Dynamics web sites. Safer to stop IIS completely.
    Stop-IIS

    # Need to stop iisexpress as well.
    $process = Get-Process -Name 'iisexpress' -ErrorAction SilentlyContinue
    if($process -ne $null)
    {
        Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue
        Start-Sleep -Seconds 1 # Wait 1 second for good measure
    } 

    $DynamicsServiceNames = @("DynamicsAxBatch", "Microsoft.Dynamics.AX.Framework.Tools.DMF.SSISHelperService.exe", "MR2012ProcessService")
    foreach ($DynamicsServiceName in $DynamicsServiceNames)
    {
        $Service = Get-Service -Name $DynamicsServiceName -ErrorAction SilentlyContinue
        if ($Service)
        {
            if ($Service.Status -ne [System.ServiceProcess.ServiceControllerStatus]::Stopped)
            {
                try
                {
                    # Get the service process ID to track if it has exited when the service has stopped.
                    [UInt32]$ServiceProcessId = 0
                    $WmiService = Get-WmiObject -Class Win32_Service -Filter "Name = '$($Service.ServiceName)'" -ErrorAction SilentlyContinue
                    if ($WmiService)
                    {
                        if ($WmiService.ProcessId -gt $ServiceProcessId)
                        {
                            $ServiceProcessId = $WmiService.ProcessId
                            Write-Message "- The $($DynamicsServiceName) service has process ID: $ServiceProcessId" -Diag
                        }
                        else
                        {
                            Write-Message "- The $($DynamicsServiceName) service does not have a process ID." -Diag
                        }
                    }
                    else
                    {
                        Write-Message "- No $($Service.ServiceName) service found through WMI. Cannot get process ID of the service." -Warning
                    }

                    # Signal the service to stop.
                    Write-Message "- Stopping the $($DynamicsServiceName) service (Status: $($Service.Status))..." -Diag

                    Stop-Service -Name $DynamicsServiceName -ErrorAction Stop

                    # Wait for the service to stop.
                    if ($ServiceStopWaitSec -gt 0)
                    {
                        Write-Message "- Waiting up to $($ServiceStopWaitSec) seconds for the $($DynamicsServiceName) service to stop (Status: $($Service.Status))..." -Diag
                        # This will throw a System.ServiceProcess.TimeoutException if the stopped state is not reached within the timeout.
                        
                        $Service.WaitForStatus([System.ServiceProcess.ServiceControllerStatus]::Stopped, [TimeSpan]::FromSeconds($ServiceStopWaitSec)) 
                        Write-Message "- The $($DynamicsServiceName) service has been stopped (Status: $($Service.Status))." -Diag
                    }

                    # Wait for the process, if any was found, to exit.
                    if ($ProcessStopWaitSec -gt 0 -and $ServiceProcessId -gt 0)
                    {
                        # If the process is found, wait for it to exit.
                        $ServiceProcess = Get-Process -Id $ServiceProcessId -ErrorAction SilentlyContinue
                        if ($ServiceProcess)
                        {
                            Write-Message "- Waiting up to $($ProcessStopWaitSec) seconds for the $($DynamicsServiceName) service process ID $($ServiceProcessId) to exit..." -Diag
                            # This will throw a System.TimeoutException if the process does not exit within the timeout.
                            Wait-Process -Id $ServiceProcessId -Timeout $ProcessStopWaitSec
                        }
                        Write-Message "- The $($DynamicsServiceName) service process ID $($ServiceProcessId) has exited." -Diag
                    }
                }
                catch [Exception]
                {
                    $ErrorMessage = $_.Exception.Message
                    Write-Message "- Exception catched on $($DynamicsServiceName) " -Diag

                    $processName = switch ( $DynamicsServiceName )
                    {
                        'DynamicsAxBatch' { 'Batch' }
                        'Microsoft.Dynamics.AX.Framework.Tools.DMF.SSISHelperService.exe' { 'Microsoft.Dynamics.AX.Framework.Tools.DMF.SSISHelperService' }
                        'MR2012ProcessService' { 'MRServiceHost' }
                    }
                    #After the stop timeout, by the time it reaches here, the process is already dead.
                    #So we check if still exists and if it does we do a dirty kill.
                    $process = Get-Process -Name $processName -ErrorAction SilentlyContinue
                    if($process)
                    {
                        Stop-Process -Name $processName -Force -ErrorAction Stop
                    }
                }
            }
            else
            {
                Write-Message "- The $($DynamicsServiceName) service is already stopped." -Diag
            }
        }
        else
        {
            Write-Message "- No $($DynamicsServiceName) service found." -Diag
        }

    }

    #Resetting the action preference to the original value
    $ErrorActionPreference = $old_ErrorActionPreference
}


Conclusion

Please pass me any tips for improvement.  This code has drastically reduced my failed builds but there are probably other things that can be done to improve it.

If one doesn't want to go to the effort of changing things in Powershell, you can also kill off the processes using additional actions on the build definition, however to keep it clean, I prefer to adapt the existing Powershell.

Comments

*This post is locked for comments