AX32Serv.exe utilizing up to 5 GB of RAM - is this normal or should I log a job with Microsoft?

Question Status

Verified
Zoran asked a question on 27 Jun 2012 10:26 PM

Hi guys,

I'm an infrastructure support guy and I was asked to take a look into a possible memory leak in Dynamics AX 2012 application servers. We have 4 environments and 4 app servers. The developers reported that sometimes the servers become so frozen that they had to restart either the Dynamics service or the server itself. They reported cases where AX32Serv.exe used up all RAM on the server. I haven't seen that, but from what I can see now, the process peaks at 5 GB on servers with 8 GB RAM and at 3 GB on servers with 4 GB RAM. However, I cannot tell now how it came down, that is, did the developers intervene or it came down by itself. I would open a support case with Microsoft to investigate this, but first I would like to confirm that there is a memory leak and that AX32Serv.exe has been behaving abnormally.

These servers are running on Windows 2008 R2 SP1 fully patched with McAfee file level AV client. 

Another interesting thing I found on one of these servers was LSASS.exe utilizing 3 GB of RAM (out of 8 GB total) and not giving it back at all. However, I'm not sure if this is in anyway related to Dynamics AX.

Any advice would be highly appreciated.

Thanks

Zoran

 

Reply
Verified Answer
Tommy Skaue responded on 14 Aug 2012 1:15 AM

I am seeing this myself. My AX32Serv.exe tends to jump up to around 3GB after running a full compile and CIL-generation. I normally restart the service after doing a full compile, but I can imagine the AOS will eventually grow back to 3GB unless frequently restarted.

There are some additional configurations for the service if you want to control memory consumption, though. You may want to read this article on MSDN:

technet.microsoft.com/.../aa569637.aspx

Reply
Zoran responded on 14 Aug 2012 2:59 PM

Thanks a lot Skaue, I will pass the doco to the developers. We restart the affected app servers overnight in order to release memory, but it would be even better if we could limit AOS to 75% of RAM so it doesn't starve the OS.

Reply
Tommy Skaue responded on 14 Aug 2012 5:42 PM

Well, if the SQL-server is also residing on the same server, it will by default hog as much as it can and leave 10%. You can define max memory consumption on the SQL-server instance itself. You defined it in MB. Default value is 2147483647 MB (Yaikes!).

I see typical developer machines having AOS, SQL, SSRS, SSAS and SharePoint sitting on the same server, and it just doesn't perform. As far as I know, both IIS (hosting SharePoint) and SSRS will slow down when available memory is low. You can change the thresholds on SSRS, and there are also possible tweakings for IIS, but needless to say, you can only tweak as much.

Learn how to configure all of these services for best performance on the developer box, or do as I do; scale out.

Good luck!

Reply
Zoran responded on 14 Aug 2012 6:17 PM

Hi Skaue,

Each of our 4 Dynamics environments has its own dedicated SQL server and SQL on these machines is limited to 75% of RAM, with 25% reserved for the OS. These are VMs, so we can easily add RAM if needed.

Our SPS 2010 environments sit on their own dedicated app, web and db servers.

We've recently separated dev/test/stg and prod environments at the physical level, so now prod runs on its own blades, switches and storage. This should also boost performances as we found that our non-prod systems consumed far more resources than prod ones.

However, since this is a virtualised environment, we cannot expect it to run as if it was a physical one.

One interesting thing I found is that companies should be more careful when selecting the appropriate Dynamics product (in this case AX vs NAV), as it seems that AX cannot perform well on a shared infrastructure.

Thanks

Zoran

Reply
Tommy Skaue responded on 15 Aug 2012 12:03 AM

Good, Zoran!

Configuring the dependent services for optimal performance is a topic on it's own. So any Dynamics AX partner needs to get ahead and learn how to make the system perform. As you've probably noticed, it doesn't perform so good out-of-the-box (yet).

As for memory consumption on the AOS-side, you may be interested in reading this blogpost:

blogs.msdn.com/.../il-compiles-explained.aspx

It explains a little more about what is happening when you're compiling AX 2012, and also a little why memory is consumed as it is.

Tommy

Reply
Zoran responded on 15 Aug 2012 4:25 PM

Thanks Tommy, will read the doco.

Reply
Zoran responded on 10 Sep 2012 6:15 PM

Just a quick update. I used the server config tool to set up a RAM limit. Created a new config instance by copying the running one, saved the config file, added 'MaxMemLoad,Text,75' (limit the available RAM for AOS to 75% of the total RAM on the server) at the end of the file, and then imported the modified config file.

The process of creating and loading a new config instance restarts the AOS service and disconnects all user sessions so plan a short outage. Upload of a new config file completes without any disruptions.

The servers are now okay, as far as RAM utilisation is concerned.

Thanks again Tommy

Reply
Pedro Rodriguez responded on 12 Nov 2012 1:04 AM

Yeah it seems that it also work for us :-) We had the same problem!

Reply
Tommy Skaue responded on 12 Nov 2012 1:09 AM

Just a quick update on this. The AOS in AX2012 will not release memory allocated. It might not be using all of the allocated memory at a certain point of time, but the allocated memory will not be released back to the operating system unless the services itself is restarted. This is by design and is related to how the .Net garbage collector works and hence how AX utilizes it.

Reply
Pedro Rodriguez responded on 12 Nov 2012 1:17 AM

And what could be the cause of this grow of memory? I have checked and it seems that when data is cached (when an user open a form, for example) memory is consumed and then is not released back, so the memory keeps growing.

Reply
Pedro Rodriguez responded on 12 Nov 2012 1:31 AM

And yes this parameter is not a solution. When we reach the top memory established by the parameter, then no more clients are allowed.

Reply
Tommy Skaue responded on 12 Nov 2012 1:33 AM

If code allocates memory server side, it will not be released. I have experienced once an AOS that consumed over 10GB RAM just because someone tried to add a company logo with size of 7MB and then run a report that cached that logo once pr line in a Reporting Services report. Since the dataset was built and cached server side, the AOS memory grew until the report was complete before it was sent to SSRS. In other words, you can easily create code that consumes a lot of memory server side.

As far as I know, there are no good built-in tools to identify a memory heavy process.

However, using the Trace Parser on code you have built is highly recommended:

mybhat.blogspot.no/.../performance-tuning-using-trace-parser.html

Reply
Pedro Rodriguez responded on 11 Feb 2013 11:57 PM

This solved our problem:

AX2012 memory leak – hotfix available

kaya-systems.com/ax2012-memory-leak

Reply
Verified Answer
Tommy Skaue responded on 14 Aug 2012 1:15 AM

I am seeing this myself. My AX32Serv.exe tends to jump up to around 3GB after running a full compile and CIL-generation. I normally restart the service after doing a full compile, but I can imagine the AOS will eventually grow back to 3GB unless frequently restarted.

There are some additional configurations for the service if you want to control memory consumption, though. You may want to read this article on MSDN:

technet.microsoft.com/.../aa569637.aspx

Reply