Like
Share
ReportIn the last previous post we saw how X++ source code can be transformed into a structured XML representation based on the abstract syntax trees (ASTs) that the X++ parser generates from source code. As the AST passes through the compiler it is enriched with more information (like the types that are calculated by the type resolver). Just before the ASTs reach the IL code generation, the compiler can write out the XML representing the AST. That XML can then be stored in a special XML database that allows efficient querying over the code using an elegant query language called XQuery, which is a superset of XPath that you may already know. The previous post contained information about how to install the database and how to get your feet wet with running some queries over the code.
The next step is on github. Microsoft hosts an open source project on https://github.com/microsoft/Dynamics365FO-AppChecker that contains several things that are of interest to this effort. Let us go through what is there.
The Socratex Explorer:
The github site contains a set of tools that you will find useful as you work with writing and consuming queries against your source code. In particular, it hosts the Socratex Explorer that allows you to actually see the source code that you are searching over. You can download a version that is completely self-contained from the github site: It does not rely on having any particular version of .NET installed on your computer, for instance.
Go to the github site and download the executable Socratex Explorer. Go to the Actions part of the site and select the Build.Explorer.netcore workflow. You will see a list of builds that are hopefully all green:
Click on the latest workflow run and locate the Artifacts part.
Clicking on the box icon will cause a download of a zip file to happen. The zip file contains the SocrateX.exe and the PDB that goes with it. Just take the executable file and extract it somewhere, perhaps the desktop. The executable file is selfcontained: All the assemblies needed (including the .NET version) are packaged within the executable. For this reason it will take a little while when you start it the first time.
When you start the app it will open a dialog that allows you to provide the credentials to the database that you want to work on:
As you can see you can provide a URL and a port number for for the BaseX server. Running locally (i.e. specifying localhost as the server name) works, but we need to start the server to connect to be able to connect to it. When you installed BaseX, a series of programs were added to your system; One of them will start the server so it listens for incoming connections on the default 1984 port. On my system it is called “BaseX server (Start)”. Once that is done, you can provide the default password (which is the same as the default username) and get connected to the database.
When you are successfully connected to the database you will see something like this:
Let’s just check whether we are connected correctly and see how things work by issuing a simple XQuery query: the list of numbers from 1 to 10:
Enter the query (1 to 10) in the query window and press the green Go button (or press F5) in the queries pane and after a brief delay caused by opening the database (this only happens once) you will see the results in the Results window as shown above. Now let’s get a little more adventurous: Let’s find all classes with methods that have while statements in them:
It is that simple! You now have the ASTs for the while statements listed in the Results window. It is still not very useful though, since it is impossible to relate this information to any particular class or method, and perhaps you were hoping for a better experience than seeing an AST! Let’s see what we can do about that. Let us create a resulting document that will contain all the while statements. We will start with a simple query:
The name of the element (WhileStatements in this case) is not important here. When you execute that query you will get the empty document, which is not very surprising (or very interesting for that matter). We will develop a more useful query in a few steps; this is how I often go about writing them. First we need to traverse the classes:
When we run this we will get a document with a Class node for each class in the database as you can see in the Results window. Still not useful, but we have to crawl before we can walk and ultimately run. Let’s refine the query by at least including the artifact that contains while statements:
So we traverse the classes, storing each one in the $c variable. Then we look for WhileStatement elements nested in any depth (hence the double slash, //) within the class; each one is stored in the $w variable. In the output element (named While, but again the name is not important) we include the Artifact attribute (i.e. @Artifact) of each of the classes.
Why are there more than one identical While elements in the result? Because many classes contain more than one while statements of course.
There is an Artifact attribute on every toplevel artifact (Class, Form, Query, Table, View etc.). It contains the name and the kind, as you can see above. Let us include the whereabouts of the while statements by including the (StartLine, StartCol, EndLine, EndCol) attributes. As it happens almost all elements in the AST have these four attributes because this is the information that allows the X++ compiler to tell the user where a compilation error happened:
We are getting the position values from the $w variable that holds the while statements. Finally we are getting something we can relate to. Now, finally comes the magic. Click on one of the While elements in the result window - It does not matter where on the line and the source code is opened with the matched coordinates indicated as the selection. All this happened because the While element contained the Artifact, StartLine, EndLine, StartCol and EndCol attributes. This is a convention that you should follow in your queries to get the ability to navigate the source code.
This way of creating useful queries reflects what I typically do when I want to investigate aspects of our codebase (consisting of some 57 million lines of code). I typically start out with the template, and then work to specialize it more and more to get what I am looking for. When you get stuck refer to the sample queries or refer to the Priscilla Walmsley's excellent book, appropriately titled: XQuery (ISBN-13: 978-0596006341, ISBN-10: 0596006349) or by refering to the XQuery definition (Query - W3C - You will also find this link in the Help menu in the Socratex Explorer). You will find that XQuery is a small but very well thought out language.
As you develop queries you will need to know what the ASTs look like for the particular construct that you want to focus on. For instance, how would you happen to know that while statements happen to be represented as elements with the name WhileStatement, in which the expression and the statements are nested? The easiest way to know this is by simply using the Socratex explorer. Start with a place where you know you have the construct you are looking for. Then write the query that returns the AST for that artifact, like
/Class[@Name='MyClass']
and inspect the XML to understand how your code is represented.
The github site described contains much more than we have covered here. One of the things that you absolutely should check out is the list of predefined queries that we have assembled for you. You will find them in Dynamics365FO-AppChecker/Sample rules/X++/. These rules serve at least two purposes: One is to provide value when weeding out trouble with the queries. and the other is to allow you to learn more about how to write XQuery queries. In future blogs we will sometime illustrate things by using the sample queries.
To end this blog and to make sure you have internalized all this information, I urge you to try to develop a query that shows all the classes in your database such that clicking in the resulting lines will open the source code. Then refer to the BasicStatistics sample query to get an overview of what is in your database. Finally use the classMetrics query to dig into the characteristics of all your classes, including the cyclomatic complexity.
*This post is locked for comments