AX 2012 (R2) Compilation Performance and Best Practice

This question is answered

I would like to hear others share their experience with possible best practice and recommendations around getting AX 2012 compilation time down to a bare minimum.

We know a X++ compilation involves writing to disk on the AOS server and writing transactions to the model database.

We used to have a Best Practice Whitepaper and Checklist for Dynamics AX 2009, and many of the points are still valid for the next generation Dynamics AX. I still would like to hear from the community what hints and tips they would like to share regarding this specific process; compiling AX 2012.

AX2012 may take 2-3 hours. AX2012 R2 over 5 hours. Can anything be done to reduce the compilation time without "triple" the hardware?

Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

Verified Answer
  • It's a big issue indeed. As shown there's some ways to shave off some time, but in the grand scheme of things it's not a whole lot.

    Problem is really the X++ compiler, it simply doesn't scale. Notice how it uses just 1 core, and not even at 100%. Maybe with superfast SQL and disk it may use 100%, of one core only. Remember they started developing AX originally in the early 80s I believe, so 8086 type stuff with the memory restrictions involved, and the first release of AX came out when the first pentiums were brand new. The architecture of the compiler hasn't really changed since then.

    Bottom line, it doesn't scale so doesn't matter a whole lot how much hardware you throw at it. Problem is the legacy history of the X++ compiler versus the amount of new technology and code they've added to 2012 and R2.

  • The current X++ compiler does not scale very well with increased load of source code.

    This is due basically to its old architecture, and the fact that it interfaces with a metadata provider that does not perform very well either. You will not get a lot of benefits from increasing the speed of your CPU, since the CPU is not maxed out anyway: The compiler spends some time waiting for metadata to be fetched from the server.

    Even though the hardware the client is running on may be 64 bits, the client application (where the compilations happen) are still 32 bits. This sets a few restrictions on how much memory can be used in the compiler as well as the speed.

    The best thing you can do is to make sure the metadata does not have to travel too far: In other words, try running the client on the server box (if possible: There may be many reasons you may not be allowed to do this).

All Replies
  • Thanks for stepping in, Peter and Joris. I'm glad you are able to share your insights even given your NDAs. :-)

    The notion "keep your metadata close" is quite interesting. I would like to see an overview of the "moving parts" or components involved here. I know some people claim having a SQL Server instance on the same machine as the AOS improves compilation performance, but that assumes the same machine is capable of handling all components involved (AOS and SQL Server in one box). If the "metadata" are best collected from in-memory on the AOS, having the SQL Server on the same box wouldn't help necessarily.  

    I would like to mark Peter's and Joris' answers as "answer", but I can't help feeling there is more to be said. :-)

    (fingers crossed).

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • This is a fascinating discussion because our AX consultants and Microsoft support keep telling me that running the AOS and SQL Server on the same box are ill advised.  

    I guess in the "right situation" it is ideal.  

    Several members of our IT department (we are not consultants but rather a company that develops its own software to avoid consulting fees) will be attending Convergence 2013 in New Orleans.  We will be bringing a printed form of this discussion with us to see if anyone from Microsoft has an opinion.

  • Well, I haven't yet concluded that having the SQL Server on the same box is advisable. SQL Server will have it's own memory space, independently of the AOS and the AX Client. Default settings when installing an SQL Server instance is that it will allocate its memory space as best it can, but it will not release any memory it grabs unless it is restarted. I believe the AX2012 AOS does the same unless you change the GC settings (someone correct me if I'm wrong). Sure, if you have enough RAM, this won't be a problem. Having SQL Server on the same box eradicates any "network latency" if that would prove to be an issue.

    So far, it seems to me the bottleneck most likely never is the network latency, but rather a combination of CPU and RAM, allocates primarily to the AOS and the Client.

    In many scenarios and perhaps in general, having SQL Server on the same box as the AOS is bad, but we are looking for an optimal build scenario, shaving off them precious minutes or perhaps hours of the total compilation time.

    Quick CPU, enough RAM, quick access to Metadata and removing any unnecessary chit-chat (warnings and errors) seems to be the highlights.

    I find it a very fascinating discussion, indeed. ;-)

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • I have marked both the answers from Joris and Peter as "answers", but feel free to contribute if anyone else has additional insights. :-)

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • but it will not release any memory it grabs unless it is restarted. I believe the AX2012 AOS does the same unless you change the GC settings (someone correct me if I'm wrong).

    Maybe the hotfix mentioned in the link below fixes that problem?

    kaya-systems.com/ax2012-memory-leak

  • isn't this hotfix included in r2?

  • It is. But memory leaks to me is a different issue. They are memory consumption that is due to a flaw/bug and not by-design.

    The AOS will by-design not release allocated memory if it can avoid it by default. This is expected behavior and something we need to take into consideration. Obviously, on a environment with little system memory, the OS will need to shift things around and use the swap-file. I once saw SQL Server and AX 2012 running on same box with 4GB allocated RAM. It just didn't work at all having SQL Server and AX fight for the RAM. ;-)

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • You can limit the max. amount of memory usage of the AOS by applying the MaxMemLoad command, as specified here technet.microsoft.com/.../aa569637

    The same can be done for SQL by specifying the max. amount of memory.

  • Although that particular command (MaxMemLoad) doesn't seem to work very well....

    Back on topic about how to get AX to compile faster:

    Would it be possible and, if so, faster to split the full compile into smaller compiles of tables, classes, forms, etc and run these each in their own client session?

  • Well, the order of how the elements are compiled is important, but more often it will need several runs before all the dependent elements are compiled correctly. So the short and safe answer is no. Besides, you also need to be sure CIL compiles, and this step is AX2012 specific and crucial for success. I've seen various examples of X++ compile but CIL not compiling due to inconsistency between X++ and managed code. The only way to catch those are if the AOT is fully compiled first.

    My best bet is to have a build server with scripts and setup predefined for the most optimal build performance. It would  be cool to run a poll here and see what compile times people have. I reckon most people will have no less than 4 hours compile on AX2012 R2 and maybe 2,5 hours on AX2012.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Based on ONE test performed on a pre production high end Hyper-V platform and SQL Server 2012 SP1 CU2 Enterprise (32 Gb reserved vRAM) for AX 2012 R2 GA/RTM, I was able to cut down compilation time with approxemately one hour when running the compilation from the SQL Server with locally installed AOS instance and client, compared to running from a dedicated AOS server. So this single test supports the suggestions from mr.Villadsen.

    BTW - A very interesting thread that I would like to contribute to despite beeing marked as answered :-)

    Best regards

    AX2009Tech aka Back2AX, Norway

  • Thanks, AX2009Tech.

    A couple of questions.

    Are you saying moving the AOS to SQL made a difference, or using the client on the same "box" as the AOS?

    Or simply all on the same box made it go faster?

    One our less compilation time. What was the total time, if you don't mind me asking?

    What is your Best Practice Compilation Level set to?

    :-)

    I will do some tests on my own soon. Just need to prepare a server in-between everything.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Skaue,

    AOS and Client installed on the SQL Server (all on the same box), and compilation ran from the SQL Server. Basically I only use this AOS instance when compiling and synchronizing to avoid network communication. I don't have the exact figures available on my private PC, but it went down from around 4,5 hours (run on of the AOS servers normally used in this environment) to 3,5 hours. BP = default.

    As mentioned in the first reply, this verifies what Mr.Villadsen has explained (a very trustworthy source in this context).

    Best regards

    AX2009Tech aka Back2AX, Norway

  • Hi there,

    Funny that you ask this question right now, as we've just decided to try to parallellise the compiler. Basically what we do, is start-up 8 Ax-clients and let them all compile part of the AOT. Currently we still have some issues with the Visual Studio projects (they give strange compile errors), so we placed them out-of-scope for now. I'm not even sure whether this is really necessary.

    So the normal full compile also takes us 4 hours, The first tests with the parallellised compiler (without the VS-projects) takes us 30 min, on a 3-tier configuration (3 different boxes), virtualized 2-core AOS-server.

    So these results are hopefull, but we haven't tested it out yet in production environments, so I'm not really sure whether my compiler does exactly the same as the normal one. I'll keep you posted

  • That's an interesting approach, but not very reliable. You obviously compile each object once, but there is a reason why the full compile does 4 passes over all objects. You need to be able to resolve dependencies correctly. Maybe you can run your process 4 times to simulate the 4 pass compile across all objects, which I guess would put you at 4x 30 mins which is still only 2 hours.

    I don't know. It's an interesting idea, but I'm a little hard pressed to trust it 100%