Making Field Service Smarter using AI Builder

Quality audit is part of a work management life cycle in most Field Service organizations. To ensure customers are satisfied, equipment is up and running, and there are no SLA breaches, a quality audit is conducted in all or statistically sampled work orders. This often means creating another booking, or a task, or creating a follow-up work order for quality check and of course additional resource requirements – but is there a smarter solution possible in Dynamics 365 Field Service? This post explores the possibility of using AI Builder.

The Challenge:

In our Contoso field service organization, this is how the work happens:

1. Field Technicians are booked to conduct the work
2. Field Technicians complete the work, take a picture and uploads in the app
3. Supervisors take a look at the picture, make a conscious call to either approve or visit a site to confirm work is properly completed
4. Work Order is completed after the quality check has passed

For the point (3) above, we’ll use AI Builder’s object detection model to train it on pictures of quality vs faulty work, and then once a model is trained, hook it up with Field Service to check the quality of work and then change the status of Work Order automatically. This is how the process will look like:

The Setup:

We’ll need the following Power Platform products:

1. Dynamics 365 Field Service
2. AI Builder
3. Power Automate Flow

All of the above can be setup as a trial for this post.

As for the images of quality vs faulty work, I’ve tapped into vast datasets available on Kaggle and found this one with the images of Lego bricks. For our experiment today, we’ll use 20 images each of 4×4 vs 1×2 bricks to differentiate quality vs fault work:

Faulty Work:

Quality Work:

The Play:

Following are three steps to run this play:

1. Dynamics 365 Setup
2. Build a model in AI Builder
3. Create a flow to call AI Builder model with Dynamics 365

1. Dynamics 365 Setup:

Create a new field on Work Order entity:

Name: Completed Work

Type: File

….and drop it on a form.

Field Technicians upload the image of completed work in this field. We’ll use Power Automate to call AI Builder model on this field’s content.

2. Build a model in AI Builder

Create a new AI Builder model of type Object Detection and name it as ‘Quality Check’. Click Next..

Select ‘Common Object’ and click Next..

Add two ‘Object names’ as Quality and Faulty work. Click Next.

Click on Add images and upload all 40 images..

Click on Add images and upload all 40 images..

Click on Tag images and start tagging images one by one as Quality…

and Faulty work:

Let’s have a look at the Summary and then click on Train. It will take few minutes.

Once trained, run a few Quick Test. Seems to be working for me .

Quick Test 1:

Quick Test 2:

Create a flow to call AI Builder model with Dynamics 365

Let’s go to Power Automate flow and click New. We will use ‘Common Data Service (current environment)’ trigger and have the following actions:

Trigger: When a record is created, updated or deleted on Work Order

Steps:

1. Get file from Work Order’s file field
2. Predict action to call our AI model and pass on the image
3. Based on the prediction, either mark Work Order as Closed or send an email to Field Technician

Here is our Power Automate flow:

..the second step:

the third and following steps…

All done! Start the play by uploading the image of completed work (in this case 1×2 or 4×4 lego brick images). The flow will run, send the image to AI model, and update the status of Work Order or send the email. Successful output:

Viola! That was fun and what we just experimented with is, in fact, a game changer in terms of making enterprise applications (like Dynamics 365 Field Service) really smarter – coupled with the fact that you can get this up and running in a very short timeframe and without expensive solution development/deployment.

Hope this was useful. If you have any suggestions/questions, please do reach out – would love to hear from you.