web
You’re offline. This is a read only version of the page.
close
Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Microsoft Dynamics AX (Archived) / Slow performance on vi...
Microsoft Dynamics AX (Archived)

Slow performance on virtual AX 2009 deployment

(0) ShareShare
ReportReport
Posted on by Jake 165

We have AX 2009 running on a VMware/Cisco UCS virtual solution.  Most end users connect to the app via RDP to a terminal server.  All servers - DB, AOS, Terminal Servers - are virtualized.  We have 3 AOS's and 4 Terminal Servers.  The Terminal Servers are load balanced through Windows, and the AOS's are load balanced internally by AX.  We have a chronic performance problem where the end users' sessions will slow to a crawl so badly that the only way we can fix it is to restart the AOS service on the AOS servers.  The DB and DB server always look fine.  No table locks, CPU, RAM, all look good.  So, we've ruled out the DB and DB server.  The AOS servers never get above 50% CPU utilization, but they'll hover around that point for a while.  RAM on the AOS servers never runs out or comes close either.  The Terminal Servers don't max out CPU either.  The Terminal Servers ran low on RAM a couple of times, but never ran out.  We added more RAM to the Terminal Servers and still get the problem.  Our AOS servers have 8GB of RAM.  Our Terminal Servers have 8GB of RAM.

We can't figure out why performance slows to a crawl even though the AOS Servers are only sitting at 50% utilization and everything looks fine.  Any ideas?

*This post is locked for comments

I have the same question (0)
  • Suggested answer
    Denis Patrakov Profile Picture
    Denis Patrakov on at

    Do all VMs have dedicated physical CPU cores or some of them share cores (say, 2 VMs 4 cores each on a 6-core server)? How many users connect to each AOS? 50% of CPU utilization is a way too high, try disabling trace options (except for SQL tracing) on you AOS' configurations if they are enabled. Enabled tracing makes an AOS to waste a lot of CPU time even when you don't run a trace.

  • Jake Profile Picture
    Jake 165 on at

    Hey thanks for the reply.  Our typical concurrent user population is ~220-240.  We have ~75 users on each AOS.  We didn't have tracing enabled and we were still seeing this CPU usage.  Now, in trying to troubleshoot it, MS had us turn tracing on to try to get an answer as to what's causing it.  Shouldn't servers be able to handle sustained 50% CPU usage?  I understand if they're pegged at close to 100%, but I thought they should be able to handle sustained usage above 50%?  Or the AOS for AX can't run at 50% of CPU?

  • Denis Patrakov Profile Picture
    Denis Patrakov on at

    It's not about an AOS' capability to load CPU but rather about user perception of a system usability. If you run a benchmark where you create order lines like crazy then 50% CPU load is OK, but real users are slow and lazy, they can't create 50 order lines per minute, they can't normally load an AOS' CPU cores up to 50% unless they post 2000+ lines SO/PO invoices without using a batch server. Note also that a VMware host should give you more accurate CPU load measures then a guest OS' Task Manager.

  • Schadi Profile Picture
    Schadi 25 on at

    Hi Jake, could you solve your performance problem? I have a similar problem with performance of AX  running on VMWare.

  • Jake Profile Picture
    Jake 165 on at

    I'm not sure yet if we've solved the problem.  We had some of Microsoft's O/S engineers, AX engineers reading dumps, going through extensive perfmon logs, etc.  There was a lot of dispute among them as to what the problem could be.  So, we ended up creating additional RDPs and AOSs.  So, now we have 6 AOSs and 6 RDPs.  We have all of our batch processes running on an AOS that is specific to them and no end users use.  We also are starting to install all of our printers directly on our RDP servers.  This is in conjunction with security groups where the users for a particular branch will only see the printers for their branch.  This has definitely sped up printing.  We've made these changes just in the last week, so we're waiting to see if all of these changes will solve our problems and improve performance.  This is month end close for us, so if we make it through this week and we didn't have to do any AOS restarts, I'll post to let you know.  

  • Suggested answer
    Community Member Profile Picture
    Community Member on at

    Hi Jake, what was the result of your changes?

    We are looking to implement a 600 user AX solution and for hardware costs alone have been looking at VMware to virtualise it. I know the SQL Server can be heavy on disk I/O, so we have made sure the IOPS will be there to handle it, but we are getting a lot of push back from Microsoft with regard to virtualising the AOS instances. They point to CPU and network as the primary bottlenecks, which they say is made worse by virtualisation.

    There are a number of things I would look at with regard to your problem before spending more money on extra AOS instances:

    Firstly what version of VMware are you running on? I've seen complaints about VMware performance only to discover that they were on ESX3.5 - ESXi5.0 is an order of magnitude faster than that platform!

    Secondly I would never use Windows tools to disgnose a performance issue in a VM, especially with CPU. It would be helpful if you looked at the %Ready stats of the CPU on the hosts, because although the VM CPU is showing 50% or less utilisation it may be the physical cores are waiting due to contention. This is impossible to diagnose from within the VM.

    Thirdly with regard to the network. Assuming you have ESX4 and therefore hardware level 7 in your VMs I would assume you are using VMXNET3 paravirtualised NICs for maximum throughput? One thing that is important is that in VMware Receive Side Scaling MIGHT NOT be enabled by default on the VM using VMXNET3 NICs. This means your network connection can only use cpu0 in the VM, and because AOS is very network bound with lots of small I/Os to the DB server you could be getting network latency; this would make the system look very slow, A reboot of AOS would clear this, masking the real issue.

    You need to enable it on the hardware NIC setting within the VM (device manager, advanced properties of the NIC) - Windows has it enabled by default.

    A major issue with virtualised environments is that perfornance problems are often disgnosed from within the VM by tools suited to physical servers. These may be a good start (to determine if RAM usage may be causing swapping for instance), but as long as the performance issue isn't code (surely not ;) ?) then it is often the physical hardware - so you need to do your disgnosis at the VMware host level.

    I'll be interested to find out how you get on.

    John

  • Bart Couvreur Profile Picture
    Bart Couvreur on at

    Hi Millennia,

    I just read this post from you from 3 years ago and the comment that Microsoft pushed back on virtualization of the AOS instances. We recently moved our AX 2009 onto  infrastructure off-site with a degree of virtualization as well and are encountering performance problems.

    Which way did you end up going and how did it work out for you?

    Thanks,

    Bart Couvreur

    ERP Manager at NHP Electrical Engineering

    Melbourne, Australia

    e-mail: bcouvreur@nhp.com.au

  • Suggested answer
    Umesh Pandit Profile Picture
    Umesh Pandit 9,315 User Group Leader on at

    old but worth checking this post, blogs.msdn.com/.../welcome-database-configuration-checklist-part-1.aspx

    blogs.msdn.com/.../ax-database-configuration-checklist-part-2.aspx

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Responsible AI policies

As AI tools become more common, we’re introducing a Responsible AI Use…

Neeraj Kumar – Community Spotlight

We are honored to recognize Neeraj Kumar as our Community Spotlight honoree for…

Congratulations to the November Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > 🔒一 Microsoft Dynamics AX (Archived)

#1
Martin Dráb Profile Picture

Martin Dráb 4 Most Valuable Professional

#1
Priya_K Profile Picture

Priya_K 4

#3
MyDynamicsNAV Profile Picture

MyDynamicsNAV 2

Last 30 days Overall leaderboard

Featured topics

Product updates

Dynamics 365 release plans