16 Jun 2015 9:30 AM Introduction Should we create test automation? This is a question that countless development teams have discussed. Below are a few of the questions that are inevitably discussed. How much time do we have? How many team members do we have devoted to verify the quality? Do the assigned team members have the skills to create test automation, or would training be required? What other options are there besides creating automation? (Manual verification, outsourcing, spot-checking, etc.) Could the time invested in creating the automation be better spent testing the product? How many test iterations are planned this release? (A high number could be a good candidate for test automation whereas a low number may not.) Could the test automation be used in future releases? When I started at Microsoft years ago, my team’s assumption was that we would attempt to automate everything. We devoted resources to create thousands of automated test cases. Automation was created for our Dynamics GP desktop client, windows applications, installs, and browser applications. Fortunately, our Desktop client test automation is based on a macro tool, which is built into and ships with Dynamics GP. This makes the test automation incredibly stable and effective. Much of our other test automation has not fared as well as the desktop client test automation. It was based on frameworks that were not always reliable. The applications under test were difficult to automate and included external application dependencies that increased the complexity. A strategic investment in some test automation would have been good. Many in our software industry have come to realize that automating 100% of an application is difficult and may not be a good investment of time. This in no way is to suggest that test automation is a bad thing. It is a great tool when used properly in the right scenarios. If we shouldn’t assume that test automation is the answer, we should consider whether creating test automation for our new Dynamics GP service based architecture (SBA) is worthwhile. I would propose that it is for the following reasons. SBA does not have a UI. This would make any manual testing difficult and very tedious. Since there is no UI, creating test automation for SBA is easier to create and maintain. UI automation can be difficult to write and unstable when executing. Since SBA doesn’t involve UI, it shouldn’t include these challenges. While there are tools like Postman that would enable the creation of service requests, it would be very time consuming to execute a large number of tests. There are multiple items that need to be verified. The most obvious is the service response. A tool like Postman would provide this. The other is SQL. While you could manually verify any changes in a tool like SQL Management Studio, it would be painful. Any other means of testing outside of creating test automation would not be repeatable. If the service procedures go through multiple iterations, the tests should also. Hopefully you’ve decided to invest in creating test automation for SBA. The test framework that was developed and used internally has been made available externally. This code enables Dynamics GP partners and customers to quickly and easily write test automation for their service endpoints. It provides the means for communicating with the endpoints and verifying the service response and SQL results. The rest of the content is devoted to providing instructions and examples to aid in this effort. Machine Requirements The following are the requirements for a machine or machines that will be used in the creation and execution of test automation. Working Dynamics GP 2015 environment with GP services installed and configured. (If desired, this can exist on a machine other than the machine executing the test automation.) Visual Studio 2013 install that supports unit testing (Visual Studio Ultimate is an example.) Fabrikam data (This is not an absolute requirement. It is simply required to run the sample test cases without modifying any code.) Building The Sample Extract the Visual Studio solution to the root of your C drive. (You can place it elsewhere, but keep in mind that any changes will affect the paths for the directions below and also some test automation paths within the code and config.) Use Visual Studio to launch C:\GPSvcTest\Microsoft.Dynamics.GP.Svc.Tests.sln. Download zip file from http://json.codeplex.com/. Extract to a folder and copy Bin\Net45\Newtonsoft.Json.dll to C:\GPSvcTest\Lib. Add reference to C:\GPSvcTest\Lib\NewtonSoft.Json.dll to the Microsoft.Dynamics.TestUtilities.GPService project. Build the solution. After successfully building the solution, navigate to Test Explorer (Test > Windows > Test Explorer). This is where the current test cases are displayed. Configuration Open C:\GPSvcTest\Bin\Debug\Test\Microsoft.Dynamics.TestUtilities.config in a text editor. Modify the following as needed. Config/Homepath will need to be modified if the solution above was extracted somewhere other than the root of the C drive. It should point to the directory where the zip file was extracted. Config/SqlServers/SqlServer(0)/ServerName needs to be the name of your SqlServer. Change "reimaginetest" to your SqlServer name. Config/Users/User(0)/Domain, UserName, and Password need to be valid credentials for an active directory user with access to call the service endpoints. (Verify that this user can make a GET request to an endpoint through a browser successfully.) Config/GPServices/GPService(0)/ServerName needs to be the name of the server hosting the GP service. Make sure it's valid for the SSL certificate that was created. (In a one box scenario, it's simply that machine's name. In a multi-box scenario, it's the front-end server.) Save the file. (Optional) For successful execution of the all sample Customer tests, the following is needed. Launch Dynamics GP and log into Fabrikam, Inc. Create customer with number = GPSVCTEST. To enable execution with a company other than Fabrikam, do the following. Modify the Config/SqlServers/SqlServer(0)/Databases(0) and (1) values to correspond with your desired company and database names. If you choose to use a company other than what is set within Databases(1) above, you may need to modify the CompanyName property. This can be set broadly within the GPWebRequestTestBase/Initialize method, in individual class files, or finally on individual requests. See the sample code for more details. The automation supports executing with both json and xml. It's currently set to use json. Running with xml simply requires a change to the config file. Modify Config/GPServices/ContentTypes to set the desired content type. Running Test Automation After building the solution, the test cases should be available for viewing in the Test Explorer (Test > Windows > Test Explorer). Right-click one or more tests and choose to run or debug the test case(s). Breakpoints will only be hit in debug mode. You can additionally right-click within the test case code and click to run the automation. Below is an example of a test case. Creating New Test Automation The following are the steps required to create automation for new service endpoints. Open Microsoft.Dynamics.TestUtilities.GPService > Common > Entity.cs. Add a unique name for your endpoint. For example, if you were adding an Employees endpoint, add that enum value. 2. Open Microsoft.Dynamics.TestUtilities.GPService > Common > UriEntitiesMetadata.cs. Within the AddEntities() method, add metadata describing the endpoint. The following would be added for the Employees endpoint. This information is used when building up the complete url that will be used to communicate with the service endpoint. uriEntityDetails.Add(Entity.Employees.ToString(), new UriEntityDetail(Entity.Employees, "Payroll/", "Employees", false)); 3. Within the Microsoft.Dynamics.GP.Svc.Tests project, add a new “module” folder under the Tests folder. I would add a “Payroll” folder for the Employees endpoint. Add another folder below the newly created named for the endpoint. Employee would be created for the Employees endpoint. 4. Create a new class in the above folder for the entity. Employee.cs would be created for the Employees endpoint. 5. Copy the contents of Template.cs into the new class file and modify for the endpoint. Name the class following this pattern: endpoint name + test (EmployeesTest). If you don’t add Test to the end, it could conflict with any new classes created for communicating with the endpoints. Modify the constructor to match. The CurrentTestCategory should be the module name (i.e., Payroll). The TestSpecificSystemTablesToSaveState and TestSpecificCompanyTablesToSaveState should include any system or company tables that could change from test case to test case (due to create, delete, or update requests). Any tables added to this list will be saved and restored between test cases. This enables the test cases to execute with the same data every time. Modify the baseline path to be a valid path based on the above folders. Example below. TestEnvironment.BaselinePath = @"\GPSvc\Microsoft.Dynamics.GP.Svc.Tests\Tests\" + CurrentTestCategory + @"\Employee\Baselines\"; 6. Create a test case based on the template provided. The example below will communicate with the Employees endpoint for the given employee ID. The following are some other helpful tips. 1. If the test automation doesn’t seem to be communicating with the desired endpoint, set a breakpoint on the above line: this.ExecuteAndVerify(request). Debug the test case, and create a watch on request.CompleteUrl to verify that the url is correct. If a portion of it is not, modify the appropriate metadata to correct. For any Get (read) requests, copying the url into the browser and executing is an easy, quick test. 2. Data is very important when building up the automation environment. Much care should be given to what data is added and how it is maintained for the future. If a test case requires data that is not available in the current dataset, you can use the following to modify: this.SqlServer0.ExecuteSQLQuery(“Insert query here”, databaseID); (databaseID is from the config mentioned above.) 3. Any test cases that are expected to modify the database should define test case details. The test case details define any tables that should be dumped and where clauses that should be applied when dumping the tables. Below is an example that would dump three tables with the defined where clauses. Not defining a where clause results in the whole table being dumped. Accepting Baselines When new test automation is initially run, it is guaranteed to fail. This is due to the framework’s built in functionality to compare a given test execution test outputs with baselines. Since the baselines don’t yet exist, the test cases will fail. The test automation will save all test outputs to a TestOutputs folder (relative to given tests, i.e. Payroll/Employee/TestOutputs). Once these test outputs have been verified as correct, they should be moved to the Baselines folder (next to TestOutputs) where they will serve as baselines. Future test executions will compare test outputs to the files that were promoted to serve as baselines. More Information The code in the above mentioned zip file enables the following: Service Creating service requests (setting urls, verb, headers, attaching objects, defining credentials, etc.) Serializing and de-serializing objects using json or xml (Required for most create or update requests) Executing service requests Capturing service responses SQL Saving and restoring sql table states Executing sql queries Capturing the contents of sql tables Test results Verifying service responses Verifying sql table contents Filtering service responses or sql table contents of data that diffs every time. Sample Test template code for all test types Code reads from config, making code machine agnostic Test case examplesCode to verify newly defined tests Code to verify newly defined tests Conclusion Hopefully you find the SBA test framework easy to use and valuable in improving the quality of your service endpoints. If you run into questions or issues, feel free to add a comment. This will allow any discussion to be shared among all those creating test automation. Thanks and happy testing!