AX 2012 (R2) Compilation Performance and Best Practice

This question is answered

I would like to hear others share their experience with possible best practice and recommendations around getting AX 2012 compilation time down to a bare minimum.

We know a X++ compilation involves writing to disk on the AOS server and writing transactions to the model database.

We used to have a Best Practice Whitepaper and Checklist for Dynamics AX 2009, and many of the points are still valid for the next generation Dynamics AX. I still would like to hear from the community what hints and tips they would like to share regarding this specific process; compiling AX 2012.

AX2012 may take 2-3 hours. AX2012 R2 over 5 hours. Can anything be done to reduce the compilation time without "triple" the hardware?

Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

Verified Answer
  • It's a big issue indeed. As shown there's some ways to shave off some time, but in the grand scheme of things it's not a whole lot.

    Problem is really the X++ compiler, it simply doesn't scale. Notice how it uses just 1 core, and not even at 100%. Maybe with superfast SQL and disk it may use 100%, of one core only. Remember they started developing AX originally in the early 80s I believe, so 8086 type stuff with the memory restrictions involved, and the first release of AX came out when the first pentiums were brand new. The architecture of the compiler hasn't really changed since then.

    Bottom line, it doesn't scale so doesn't matter a whole lot how much hardware you throw at it. Problem is the legacy history of the X++ compiler versus the amount of new technology and code they've added to 2012 and R2.

  • The current X++ compiler does not scale very well with increased load of source code.

    This is due basically to its old architecture, and the fact that it interfaces with a metadata provider that does not perform very well either. You will not get a lot of benefits from increasing the speed of your CPU, since the CPU is not maxed out anyway: The compiler spends some time waiting for metadata to be fetched from the server.

    Even though the hardware the client is running on may be 64 bits, the client application (where the compilations happen) are still 32 bits. This sets a few restrictions on how much memory can be used in the compiler as well as the speed.

    The best thing you can do is to make sure the metadata does not have to travel too far: In other words, try running the client on the server box (if possible: There may be many reasons you may not be allowed to do this).

All Replies
  • We have been using an ASUS KGPE-D16 motherboard with dual AMD Opteron 6128 processors, 64Gb DDR3 and 6 Intel 520 series SSD drives all connected as various logical drives.  SQL Server 2008 and AX 2012 R2 are both installed on the same box running Windows Server 2008 x64... compilation times still take over 4 hours to complete.  

    The same box can compile an AX 4.0 code file filled with customizations in less than 30 minutes.

    The problem we are seeing is that the AOS is still single threaded, even though it is now a 64bit application.

  • Thanks for sharing, Reuben! Your setup looks pretty good. I don't have 6 SSD disks, but the rest looks similar.

    Is this AX2012 or AX2012 R2 compiling?

    Since you are running this on the same box, and you have plenty of RAM, I would assume any bottlenecks will either be CPU or disk. Are you running RAID 5, 6 or 10? I've heard RAID 6 might be safer but RAID 5 is quicker. RAID 10 would be even more quicker, but steals even more diskspace.

    Did you set Recovery Mode to Simple or Bulk in order to improve SQL Server performance?

    What is your Max Degree of Parallelism (MAXDOP) while compiling? Recommended setting for AX is 1, but maybe it could be 0 while compiling? Maybe someone knows if compiling spawns multiple query batches which would mean setting MAXDOP to 0 which would spawn as many as possible.

    Have you looked at how threads are being spawned over the CPUs while compiling, and also how IO is performing? Since the AOS compiles on 1 thread, and this thread seemingly sticks to 1 processor, can we be sure UI or SQL Server do their business on other processors? :-)

    It would be nice to find an optimal set of settings, so even if the hardware isn't state of the art, it will still compile as quick as possible.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Tommy, you seem just like me... eager to get answers about AX 2012 R2 compiling times.  I used RAID 0 and put the code, database mdf and ldf files in their own SSD drives which is why I had to use 6 SSD SATA drives.  Recovery Mode to Simple is the option I choose.

    No, the AX 2012 R2 compiler will not spread out its cpu load across all 16 cpus on these AMD Opterons.  I use a meager amount of 3% of available cpu power when compiling.  

    Now, my previous post was made in a rush.  I meant to conclude my statement by saying that I am very eager to find a faster cpu for this server.  We are dealing with a single-threaded compiler (from my understanding so far) and that is going to demand a high clock frequency processor with loads of Level 2 and Level 3 cache to improve performance during compilation.  

    For example, I used to manage a database in Microsoft Access that stored casino reward points.  Everyday, the system would synchronize all card swipes to give the player his new point total.  The system took hours to tabulate all these points on an eight core 2.5 Ghz Intel Xeon processor.  The solution?  I built a custom server using an Intel E8600 cpu which offered 3.3 Ghz of number crunching power.  The result was a synchronize time of just 1 hour because the processor ran at a higher clock speed and had loads of on-chip cache.

    In conclusion, I am very tempted to just buy an Intel Core i7-3770K 3.5Ghz Quad-Core Processor and motherboard to dedicate as my development server.  True, I will not be running Windows Server 2008 on an officially recommended box, but it will cost me thousands less than buying a comparable Intel Xeon.

    Anyone else out there have success with high clock rate cpus when compiling AX 2012 R2?

  • Hmm. That is interesting! So you want to focus on the actual CPU and it's ability to crunch the operations fast enough. Well, I have a box here with 3.6Ghz and 4 cores (screencast.com/.../ltJTxuCM4Gn). I will try see if it can chew better than the other box I have. :-)

    What about SQL Server itself? We also know every element being compiled cause several inserts into the database. Are we sure the database is performing at it's best? Could we improve performance by making improvements on the indexes or statistics prior to the compilation?

    I am planning to setup SQL Server 2012 (Ent), but I don't expect huge performance improvements over SQL 2008 R2 in regards of compilation time.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • No, in SQL Server 2008 and using other disk monitor utilities, there is very little I/O demand while compiling.  SQL Server 2012 is mostly meaningless in the face of this fact.  I encourage you to check this out for yourself!

    However, as with all AX database systems, they love low access time.  SSD's offer 0 ms, rotating 10K drives still only can get at best 5 ms access time.  If you get the chance to try out that 3.6Ghz box, why not use some SSD's at the same time to build a solid dev box.  FYI, we use Intel 520 Series SSD's in our HP Proliant servers and have had no problems.

  • Well, thank you for sharing your insights and experience.

    It would be interesting to see who could come up with the decent spec for a build server with an affordable price. ;-)

    So basically getting a CPU that can crunch operations and having your SQL Server on quick disks is your key points.

    Does your RAID Controller support TRIM as well? How big are your disks?

    I found this interesting post around RAID 0 with SSD:

    www.anandtech.com/.../trim-raid0-ssd-arrays-work-with-intel-6series-motherboards-too

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • It's a big issue indeed. As shown there's some ways to shave off some time, but in the grand scheme of things it's not a whole lot.

    Problem is really the X++ compiler, it simply doesn't scale. Notice how it uses just 1 core, and not even at 100%. Maybe with superfast SQL and disk it may use 100%, of one core only. Remember they started developing AX originally in the early 80s I believe, so 8086 type stuff with the memory restrictions involved, and the first release of AX came out when the first pentiums were brand new. The architecture of the compiler hasn't really changed since then.

    Bottom line, it doesn't scale so doesn't matter a whole lot how much hardware you throw at it. Problem is the legacy history of the X++ compiler versus the amount of new technology and code they've added to 2012 and R2.

  • Well, that might be, but has the AX 2012 code really become 4-6 times more than AX 2009?

    What else is added in Ax2012 that makes X++ compilation take 4-5 hours, compared to 30 minutes in the past?

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Turning off Best Practise checks before compiling helps  a bit.

  • Tommy,

    So far I have discovered that AX 2012 has increased the complexity of Inventory Dimensions.  The added tables require many more lines of code to support the new relationships along with providing the appearance of seamless data integration to our user base.

    IMO, someone at Microsoft should examine the X++ compiler at an assembly level and build in optimizations for at least the SSE, SSE2 instruction sets.  As you notice, the AOS is now 64bit which means that the machines running this newer architecture also very likely support the modern instructions.

  • Interesting. Turning off unnecessary "chit-chat" (aka Best Practices) should shave off some minutes hopefully, unless you need it for some reason. I also noticed that disabling certain licenses generates extra synchronization time, but I haven't noticed any additional compilation time.

    As for the compiler instructions being outdated, I would be surprised if they do anything about that in AX 2012. Perhaps if we cry loud enough, they will fix/improve it in AX 2015.

    I guess the best thing we can hope for is to configure the environment as best we can given the current compiler engine.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • The current X++ compiler does not scale very well with increased load of source code.

    This is due basically to its old architecture, and the fact that it interfaces with a metadata provider that does not perform very well either. You will not get a lot of benefits from increasing the speed of your CPU, since the CPU is not maxed out anyway: The compiler spends some time waiting for metadata to be fetched from the server.

    Even though the hardware the client is running on may be 64 bits, the client application (where the compilations happen) are still 32 bits. This sets a few restrictions on how much memory can be used in the compiler as well as the speed.

    The best thing you can do is to make sure the metadata does not have to travel too far: In other words, try running the client on the server box (if possible: There may be many reasons you may not be allowed to do this).

  • The current X++ compiler does not scale very well with increased load of source code.

    This is due basically to its old architecture, and the fact that it interfaces with a metadata provider that does not perform very well either. You will not get a lot of benefits from increasing the speed of your CPU, since the CPU is not maxed out anyway: The compiler spends some time waiting for metadata to be fetched from the server.

    Even though the hardware the client is running on may be 64 bits, the client application (where the compilations happen) are still 32 bits. This sets a few restrictions on how much memory can be used in the compiler as well as the speed.

    The best thing you can do is to make sure the metadata does not have to travel too far: In other words, try running the client on the server box (if possible: There may be many reasons you may not be allowed to do this).

  • The current X++ compiler does not scale very well with increased load of source code.

    This is due basically to its old architecture, and the fact that it interfaces with a metadata provider that does not perform very well either. You will not get a lot of benefits from increasing the speed of your CPU, since the CPU is not maxed out anyway: The compiler spends some time waiting for metadata to be fetched from the server.

    Even though the hardware the client is running on may be 64 bits, the client application (where the compilations happen) are still 32 bits. This sets a few restrictions on how much memory can be used in the compiler as well as the speed.

    The best thing you can do is to make sure the metadata does not have to travel too far: In other words, try running the client on the server box (if possible: There may be many reasons you may not be allowed to do this).

  • The current X++ compiler does not scale very well with increased load of source code.

    This is due basically to its old architecture, and the fact that it interfaces with a metadata provider that does not perform very well either. You will not get a lot of benefits from increasing the speed of your CPU, since the CPU is not maxed out anyway: The compiler spends some time waiting for metadata to be fetched from the server.

    Even though the hardware the client is running on may be 64 bits, the client application (where the compilations happen) are still 32 bits. This sets a few restrictions on how much memory can be used in the compiler as well as the speed.

    The best thing you can do is to make sure the metadata does not have to travel too far: In other words, try running the client on the server box (if possible: There may be many reasons you may not be allowed to do this).