Introduction to Socratex, Part 1

I would like to start a sequence of blog items about the development of application code in X++ based on the developer tools that we provide for that purpose. I will be covering diverse topics as they become important to me and to you – This will cover traditional X++ developer issues but will change as the winds of change sweep over our industry. A lot of the material deal with issues that I see causing issues for our internal developers, as well as what I hear through the grapevine that you are struggling with.

If I can inspire and explain, and thereby make you better Dynamics F&O ambassadors for your customers, then these blogs will have served their purpose.

Introducing Socratex

In this first installment we are going to look at Socratex that allows you to easily diagnose problems with X++ code. As I hope you will discover, you can learn about your X++ code and extract information from it in ways that that are just not possible without it. I am sure that it will be well worth your time to give it a try.

Socratex is an acronym for SOurce Core Reasoning And Trait Extraction.  It is pronounced Socratech: Since Socrates (the philosopher) was Greek, the X is the Greek letter X (Chi) that is pronounced ch, like in the word technology: in Greek: τεχνολογ?α. The analogy with Socrates goes further: The Socratic method is about asking and answering critical questions stimulating critical thinking to generate ideas and identify presuppositions. This is what I want to achieve also, but the questions and answers are about your code, not your life.

I like to think of it this way I want to be able to query over my code as my application developer colleagues query data. I want to understand code, not as flat text, but as structured concepts that carry meaning. These concepts should be queryable just as well as any other set of data. This is what we have accomplished: Code is Data!

How does it work?

To understand exactly what this data is all about you need to understand a little about how compilers work. As it happens, most modern compilers work in a similar way: There is a front end that parses the source code. The objective of the parsing is to decide whether the source code is correct with respect to the grammar of the language and, if it is, to build a representation that the rest of the compiler can easily use to accomplish its work. This representation is known as Abstract Syntax Trees (ASTs). The compiler has a pipeline containing code that refines and reasons over these abstract syntax trees; typical examples are type resolution and ultimately code generation. This is not a compiler writing tutorial – The important thing is that the source code is transformed into a tree structure. Now here is the thing: The X++ compiler has a mechanism for serializing the ASTs into a string representation. That string representation is XML. This is a logical choice, when you think of it: It allows you to represent structured information like trees trivially, and (equally important) it has a very strong query language that works on it.

Consider this snippet:

class MyClass
{

    private void SampleMethod()
    {

        int i = 1;
        i = i + 1;
    }
}

The code looks like this when it is serialized into XML (with most of the gory details left out for enhanced legibility):

As you can see, we can get an XML representation of the source code as XML documents. How does that do anyone any good? As it happens there is a very efficient database that is built to store and query XML documents. This database is called BaseX (https://basex.org, BaseX - Wikipedia) and is open source with a permissive license, running on Java. This database allows you to query the database with an elegant and expressive query language called XQuery that is a superset of XPath that you may already know.

What do you have to do?

You must install BaseX on your box, and that means that you have to install Java first. One option is to go to Oracle’s site (Java Downloads for All Operating Systems) and download the 64-bit version of the Java runtime. Oracle has recently added restrictions on what you can do with Java without paying for it, so you may want to use an open source version, like OpenJDK (java.net) instead. Note: It is important that you install the 64-bit version, NOT the 32-bit version. Once you have installed that, you can install BaseX on your box. It is simple to do, and it works right out of the box.

So far so good! We are getting closer to the value we are looking for. I still have not talked about how the X++ code ends in the database, and how to query it.

As it happens, there are two different ways to get the X++ code ASTs into the database. One is by using an existing flag on the Visual Studio (henceforth: VS) tooling, and the other is to use the command line X++ compiler (xppc.exe) to do the work. Each has its own pros and cons as described below:

Using Create Deployable Package

Let me explain the simplest solution first: Use the Create Deployable package menu item from the Deploy menu in the D365FO tooling inside VS. This will open the dialog shown below:

 As you can see, there is a checkbox marked: Run Appchecker under the list of packages. If you check this, two things will happen:

1. The package will be injected into the BaseX database on the box where VS is running.
2. A set of rules (that we will look at below) will be run over the code and diagnostic messages generated.

Try it! After your deployable package is generated, you can open the BaseX GUI application; you will see that there is a database with the same name as the package that you just created a deployable package for. Open the database (with the Database / Open & Manage menu). Then enter:

/Class/@Name

to see the names of all the classes in the package that was just compiled.

Take one of these classes, say MyClass (use one of the classes in your package) and use the following command.

 /Class[@Name=’MyClass’]

 This will show you the full AST of the class in all its glory:

This is a good start, but we will show how to use other tools that are better suited for the purposes we have in one of the next installments.

Using the xppc command line compiler.

The second option is a little more complicated, but it is preferable because of the way the X++ source code is handled as we will see in a later installment. The X++ compiler has a set of command line switches that enable the compiler to save the XML files in each directory. These flags are:

You need to add all three of these flags to the X++ compiler command line. Now, these command lines are very long and extremely cumbersome because so much information is needed for  the compiler to do its work. It is not practical to write these command lines by hand. As it happens, it is very easy to get the system to generate the command line for you: It can be provided by the VS tooling. Go to the Tools / Options… dialog, expand the Projects and Solutions item in the left-hand pane and select the Build and Run option. On the right you will see two options dealing with verbosity of the build output. Set them both to Normal if they are not already. With this in place VS will show you the command line that it uses to invoke the compiler in the output window. You can just copy it from there, and add the flags mentioned above and store it in a command file.

If you use option 1 as explained above, you will not get the source code serialized into the database. Having this source is often very useful, as you will see later. We will work to have the option to include the source when a deployable package is generated too. Once you have done the compilation with these flags, you will see a directory structure with all the classes, tables, forms etc. as XML files. This structure is ready to be inserted into the database, either by using the Add command from a BaseX client window or by using the BaseX Gui window dialog from Open & Manage menu item.

Please note carefully that the database contains your source code intellectual property. This is true irrespective of whether the source code is included (as indicated by setting the includeSourceInAsts option) or not. After all, an AST is just a different representation of the source code. Please make sure that you do not inadvertently install the database on a box that you do not control and where the database can be accessed by third parties. If you use the astOutputPath flag, remember to remove that directory once the data has been inserted into the database.

This blog is becoming too long. Let’s end here and pick it up in part 2 where we will talk about some more tools and the github site that supports Socratex.