I would like to hear others share their experience with possible best practice and recommendations around getting AX 2012 compilation time down to a bare minimum.
We know a X++ compilation involves writing to disk on the AOS server and writing transactions to the model database.
We used to have a Best Practice Whitepaper and Checklist for Dynamics AX 2009, and many of the points are still valid for the next generation Dynamics AX. I still would like to hear from the community what hints and tips they would like to share regarding this specific process; compiling AX 2012.
AX2012 may take 2-3 hours. AX2012 R2 over 5 hours. Can anything be done to reduce the compilation time without "triple" the hardware?
Thanks Dan, I was going to come over to your desk and ask for this :-)
an update on my previous post. Still running the following server:
VMWare with 2 vCPU (Intel X5670 @ 2,93 GHz), 8 GB RAM
However installed AOS, client and SQL Server 2012 locally (disabled TCP/IP in the SQL configuration, just used Shared Memory).
Compilation with binary 6.2.1000.413 (not containing the latest compile optimizations) now took 2 hours, 50 minutes.
Based on this, it seems the kernel enhancements are primarily related to the TCP communication between the AOS and database.
Adding 2 more vCPUs, dedicating 2 CPUs to AOS, 2 CPUs to SQL Server and adding some more memory probably will push the compilation time further - at least we are moving in the right direction :)
I don't think this was mentioned in here before:
The thing which had the biggest (positive) impact on compile times in our installations here was to change the setting in Control Panel / Hardware and Sound / Power options to "High performance" instead of "Balanced" which seems to be default, even in Windows Server... Chosing this option sounds kind of obvious but it definitely is something you want to check.
How do I contribute to the Benchmark on Github? I added the code by Dan and did some compilations during this weekend. I have a box running two HyperV images, one SQL and one AOS. The AOS I tested against had R2 (184.108.40.206) and Data Import Export Framework (220.127.116.11).
First run on AOS without hotfix KB2844240. It took 1h and 45m. I then installed the hotfixed and it shaved off 1-2 minutes.
I then configured an AOS on the SQL server and ran without hotfix. Shaved off just another minute. I then installed the hotfix and shaved off another minute. I then gave the SQL Server more juice and final result was 1h and 39m on last run.
I could try to hype the image even further, but I don't want to spend too much time hammering on this if it will only squeeze out just a few minutes.
The core is 3.6Ghz. I want to avoid installing the entire thing on the host machine, if I can. I know I can improve the performance of my images if I put more effort into ut, but less than 2 hours compile time isn't all that bad.
It would be cool to see a collection of various setups and how they perform. If someone had a similar hardware as me but compiled on half my time, I would surely like to know what I am doing wrong. :-)
Hi Tommy, if the KB only gave you a few extra minutes, it's more likely due to SQL caching than anything else :-)
Be sure that you have run axutil schema after installing the hotfix.
I had only a slight improvement with just the hotfix (2:50 down to 2:46 for R2), but updating the schema provided provided further gains (down to 1:55).
Of course, Daniel. The description of the HotFix tells us to update the schema for the modelstorel. Rookie mistake by me.
I used powershell:
Went from 1h 39m to 0h and 57m. That is quite an improvement!
Notice that I am using "localhost" and I have explicitly disabled anything but shared memory.I discovered that if I didn't use "localhost" it would generate network traffic regardless of the AOS and the SQL Server resided on the same machine. I'm thinking about moving "away" from the metadata now; enable tcp/ip and also try use an AOS on my other virtual server. Just to see how compile time will increase.
In the interest of sharing, I let another compile run this night, but this time on a different machine (virtual) and not on the SQL Server. Again, both servers are hosted on the same host. This time compile time took 1h and 9m. In other words around 10 minutes longer.
I am more or less concluding that having the AOS and SQL on the same machine, attempting to solve less network chit-chat, this will help somewhat. The biggest difference seems to be the HotFix. I'm now curious to see what the HotFix actually changed. Evidently the most effective change lied in the schema change and not the kernel change (or perhaps they are dependent).
Sharings my result applying the hotfix.
[Setup - 2 VMs (Machine A: SQL, Machine B: AOS) hosted on my Win8 desktop with Hyper-V]
CPU - A: 2 cores, B: 2 cores (i5-2500 @ 3.3GHz)
RAM - A: 4GB, B: 6GB (Fixed RAM allocation)
Storage - both on Crucial M4 SSD (Host's OS is on a seperate HDD)
The AOS is R2 with Data Import Export framework. (and a few other objects that are negligible)
[Before applying hotfix]
Compile time - around 2hr20mins (Did multiple runs to confirm)
[After applying hotfix]
Compile time - around 1hr30mins (Did multiple runs to confirm)
Around 35% gain. And I am quite satifisfied with a 1.5hr compile time knowing that I haven't done other optimizations (e.g. run everything on 1 machine, Shared memory, etc)
On the other hand, would be interest to hear about the compile time running on HDD.
(Btw, I can confirm with Tommy's finding that ..... without running axutil schema the hotfix doesn't do much. LOL)
Great stuff, Dolee!
I used SQL Server Data Tools (msdn.microsoft.com/.../hh297027) to run a compare of two modeldatabases with and without the hotfix, and the only important change I found was in dbo.ModelSecurityPermission (just as the MSDN article explains).
Basically, the current improvement drops this constraint:
CONSTRAINT [PK_ModelSecurityPermission_RECID] PRIMARY KEY NONCLUSTERED ([RECID] ASC)
And adds this:
CREATE CLUSTERED INDEX [I_ModelSecurityPermission_OWNERROLE] ON [dbo].[ModelSecurityPermission]([OWNERHANDLE] ASC, [OWNERLAYER] ASC, [OWNERCHILDNAME] ASC, [OWNERGROUP] ASC);GO
CONSTRAINT [PK_ModelSecurityPermission_OWNERROLERECID] PRIMARY KEY CLUSTERED ([OWNERHANDLE] ASC, [OWNERLAYER] ASC, [OWNERCHILDNAME] ASC, [OWNERGROUP] ASC, [RECID] ASC)
The table did not have a clustered index. Now it does! I hope to see more improvements like this. :-)
in my customer deployment the table you mention has a clustered key, but with other colums...
CREATE CLUSTERED INDEX [I_ModelSecurityPermission_OWNERROLE] ON [dbo].[ModelSecurityPermission]
how could I verify if this fix apply to this customer or not?
As far as I can tell, the index looks exactly the same. ;-)
Which could indicate the HotFix is applied. But if you want to make sure the HotFix is not only just been applied to the schema, you could check the build number on the kernel. It should be equal or higher than 6.2.1000.1013
kernel versión: 6.0.947.862
App versión: 6.0.947.280
Solution versión: 18.104.22.168
we'll apply CU5 + the Fix you mention.
however, there must be a typo error in your message: one table cannot have more tan 1 clustered index.
in the table I've checked the RECID column is not indexed.
I think that we're suffering that issue.
on the other hand, we're symplifying the roles assigned to the users; do you have links to permissions configuration best practices?
thank you again,
Actually, after applying the fix I have the following index
/****** Object: Index [PK_ModelSecurityPermission_OWNERROLERECID]
Script Date: 6/1/2013 2:51:33 AM ******/ ALTER TABLE [dbo].[ModelSecurityPermission] ADD CONSTRAINT [PK_ModelSecurityPermission_OWNERROLERECID] PRIMARY KEY CLUSTERED (
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] GO
You will want the RECID included as the commonly used query
declare @p1 int
declare @p3 int
declare @p4 int
declare @p5 int
exec sp_cursoropen @p1 output,
N'SELECT OWNERGROUP, OWNERCHILDNAME, RELATIONNAME, EFFECTIVEACCESS, SYSMANAGED, MANAGEDBY, OBJECTGROUP, OBJECTNAME, OBJECTCHILDNAME, KERNELTYPE
WHERE OWNERHANDLE = @P1 AND OWNERLAYER = @P2 AND OWNERGROUP = @P3 AND OWNERCHILDNAME = @P4
ORDER BY RECID'
,@p3 output,@p4 output,@p5 output,N'@P1 int,@P2 int,@P3 int,@P4 nvarchar(max)',504754,0,4,N''
select @p1, @p3, @p4, @p5
orders by RECID (unnecessarily so, in my opinion).
However as of build 6.2.1000.1013 you will still not get an optimal query plan as the value for the OWNERCHILDNAME parameter-value is erroneously typed as nvarchar(max) instead of nvarchar(255), the actual data type definition of the OWNERCHILDNAME column.
This will result in the slightly more convoluted query plan:
Compute Scalar(DEFINE:([Expr1006]=NULL, [Expr1007]=NULL, [Expr1008]=NULL, [Expr1009]=NULL, [Expr1010]=NULL, [Expr1011]=NULL, [Expr1012]=NULL, [Expr1013]=NULL,
|--Clustered Index Insert(OBJECT:(CWT), SET:([CWT].[COLUMN0] = [terberg_ax2012_acc2].[dbo].[ModelSecurityPermission].[OWNERHANDLE],[CWT].[COLUMN1] = [terber
|--Sort(ORDER BY:([terberg_ax2012_acc2].[dbo].[ModelSecurityPermission].[RECID] ASC))
|--Nested Loops(Inner Join, OUTER REFERENCES:([Expr1019], [Expr1020], [Expr1018]))
| |--Constant Scan
|--Clustered Index Seek(OBJECT:([terberg_ax2012_acc2].[dbo].[ModelSecurityPermission].[I_ModelSecurityPermission_OWNERROLE]),
In the tests I've done have had acceptable performance with compile time 1 hour and 40 minutes.
basically I have an environment of 24 cores with 70 Gb Ram SAN unit with capacity of 300 GB where we store the databases.
For this unit perform an alignment SAN 1024k with formatting to 64K
We apply the latest fix from Microsoft Dynamics AX R2 KB2855375 [blogs.msdn.com/.../overview-of-ax-build-numbers.aspx]
Compilation with binary (6.2.1000.1322) this work Fine
It's a big issue indeed. As shown there's some ways to shave off some time, but in the grand scheme of things it's not a whole lot.
Problem is really the X++ compiler, it simply doesn't scale. Notice how it uses just 1 core, and not even at 100%. Maybe with superfast SQL and disk it may use 100%, of one core only. Remember they started developing AX originally in the early 80s I believe, so 8086 type stuff with the memory restrictions involved, and the first release of AX came out when the first pentiums were brand new. The architecture of the compiler hasn't really changed since then.
Bottom line, it doesn't scale so doesn't matter a whole lot how much hardware you throw at it. Problem is the legacy history of the X++ compiler versus the amount of new technology and code they've added to 2012 and R2.
The current X++ compiler does not scale very well with increased load of source code.
This is due basically to its old architecture, and the fact that it interfaces with a metadata provider that does not perform very well either. You will not get a lot of benefits from increasing the speed of your CPU, since the CPU is not maxed out anyway: The compiler spends some time waiting for metadata to be fetched from the server.
Even though the hardware the client is running on may be 64 bits, the client application (where the compilations happen) are still 32 bits. This sets a few restrictions on how much memory can be used in the compiler as well as the speed.
The best thing you can do is to make sure the metadata does not have to travel too far: In other words, try running the client on the server box (if possible: There may be many reasons you may not be allowed to do this).
We tested several different scenarios, and we found out that the following is best practice:
- Install AOS en SQL on the same box (splitting over two boxes will double compile time)
- Use latest CPU architecture with 2x vCPU (1x vCPU will double time, the same to older CPU)
We timed a full compile of AX2012R2 on a VMware system with 2x vCPU Intel XEON E5-2690 2.9Ghz in 2 hour and 05 minutes. The same compile on the same Virtual Machine on an older XEON 5650 used up to 4 hour and 35 minutes. This means that using the latest CPU architecture will speed things up quite good.
Although... I personally find it really strange Microsoft is not putting more effort to this problem. Not all customers are running "the latest and greatest" hardware, and placing a hotfix in the production environment (we of course hope we never have to do this but..) is getting a more and more daunting task.
If the customer does not want to buy more recent hardware, let him buy a latest Core i7 workstation. Use this workstation (install AOS and SQL on this machine) only for full compile and full CIL actions.
Alright, I just posted it. Read all about it here: daxmusings.codecrib.com/.../dynamics-ax-2012-compile-times.html
40 minutes. That's right :-)
Inheritance is definitely a big deal, but not the only thing.
For inheritance, try to inherit from RunBase. Compile. Then change the inheritance to inherit from RunBaseBatch instead. Your runbasebatch methods will not show up on your class (in intellisense when you try to use the class from code). The only way to get that class fixed is to go to RunBaseBatch and compile forward on that base class. The compiler won't tell you this though.
As for single object compile... What if you change the name of a table field for example? That table compiles fine and your X++ classes also think it's fine (it may actually run fine if the object IDs didn't change). But as soon as you try to generate CIL, or move the code to another environment and new IDs are assigned, things don't compile in classes referring to the old field names.
CIL generation has complicated matters since it's more strict (which is a good thing) than what we had before. CIL technically does a full generation every time (don't trust the incremental! there's easy ways to show the incremental is flawed).
These are just a few easy examples (I'm sure there's even more interesting examples with changing properties on tables/fields/services/etc). They show the need for full compiles. Whether you can do the full compile in bits and pieces (back on topic) is a good question. But I wouldn't trust it entirely :-)
If you're implementing a full algorithm for dependencies etc, you should look at some of the startup classes that deal with XPO imports. There's some logic there about recompiling etc. I think it's purely based on compiler output an deciding to keep recompiling certain pieces until there are no more errors or until some threshold is reached.
Sorry I could not resist :) Probably I repeat some previous comments:
Ax compilation watching (like birds watching) revealed the Ax compile is performed in sequential manner - little bit SQL, some more AOS, and little bit Ax again in the loop.
It begs for test:
1) take best CPU for single thread operation. Here is some info about it:
I would take inexpensive Core i5-3570S from that list
2) take very fast ram ~16-32 GB (I think 32 GB is max for core i5 cpu) and board which allows this ram to operate (overclockers sites can help)
3) do not use virtualisation - undeniably it cause extra overheads
4) run all Ax components on this machine (all components of standard Ax required for compilation feels rather ok starting from 8GB RAM).
A and would be nice to overclock this thing to 4+ Ghz
A and CIL build (after compilation) is different kind of animal - it uses parallel threads and can employ many cores.
Six days, that is crazy. You need to review your setup. I'm about to post a blog entry about this. I got the compile down to 40 minutes =)
Hotfix for R2 compiler speed released: blogs.msdn.com/.../ax2012-r2-hotfix-available-improves-compile-speed.aspx
I just tested the released compile-optimization support.microsoft.com/.../KBHotfix.aspx, and it was not an amazing improvement.
My test server, VMWare with 2 vCPU (Intel X5670 @ 2,93 GHz), 8 GB RAM with AOS and client (database on a dedicated SQL Server), a full compile was reduced from approx 5 hours down to 3 hours, 40 mins.
An improvement, but still not satisfying.
Yes, if you read the documentation on MSDN ( msdn.microsoft.com/.../d6da631b-6a9d-42c0-9ffe-26c5bfb488e3.aspx ) it talkes about issues with COM objects. In fact, prior to CU7 there is a form called SysInetHtmlEditor that has this issue. You need to fix that to get rid of two compile errors, but otherwise it works great.
I blogged about this here: daxmusings.codecrib.com/.../what-you-need-to-know-about-15-minute.html
On top of the COM issues mentioned by Joris, you can get locked into the newer kernel version by connecting with a CU7 AOS. I received this failure on a CU6 environment after having run an AxBuild compile from a CU7 AOS:
Object Server 01: Fatal SQL condition during login. Error message: "The internal time zone version number stored in the database is higher than the version supported by the kernel (5/4). Use a newer Microsoft Dynamics AX kernel."
I pushed past this by setting the timezone version back (in the SqlSystemVariables table), but ONLY because it was on a test/development environment. I would not do that for an important system.
I have tested trying to compile a CU6 modelstore just by pointing the axbuild to a CU6 modelstore, but the tool will ignore this and instead compile the modelstore for the aos configuration setup on the system.
Daniel has tested compiling a CU6 modelstore by changing the AOS configuration, but then the modelstore could not be used using a CU6 kernel without doing a SQL hack.
I have tested the crazy thing as trying to run axbuild from a CU6 AOS bin folder, but they've added additional code to the binaries supporting this new build routine (obviously). ;-)
I am sort of concluding that the best solution is updating all binary components to CU7, all except "database". You will need to go through a proper code upgrade process first, making sure your code and any ISV or VAR code is successfully upgraded to CU7.