AX2009 & VMware performance issues

This question has suggested answer(s)

At our company we run VMware en have virtualized the AOS and SQL server of AX2009. And experience serious performance issues. If we install the same situation on an inferior, but physical, piece of hardware. The performance jumps up by more than 100%.


Benchmarking the SQL server indicates VMware machine outperforms de physical SQL server with ease. It seems like a communication problem(s) between virtual AOS and virtual DB server.


Vsphere 4.1.0. We setup VMware so the hardware resources may not be shared with other machines (not that any other are running on our ESX machine). Does anyone have an idea what could be the culprit?

All Replies
  • I think you need to narrow this down a bit further. Test virtual AOS with the physical SQL server and then you have a direct comparison between how your virtual AOS works compared to the physical AOS. If no major differences here then you have a SQL issue and it could be lots of issues in how the SQL is setup.... maintenace jobs etc.

  • This is the by default issue with Vmware  and Ax.

    So you can try The Microsoft Virtual PC it will work fine as compare to Vmware.

    One thing should be noted that Virtual machine system is always slow as compare to physical system.

  • Good morning,

    We have done testing with virtual AOS and virtual SQL machines in the mix. And our benchmarks (a SQL query heavy project recalculation batch) show the follow:

    ------- AOS ---SQL---benchmark

    AX50 Virtual Virtual - 215 seconds

    AX50 Fysical Virtual - 165 seconds

    AX50 Virtual Fysical - 151 seconds

    AX50 Fysical Fysical - 110 seconds

    We are going to try Hyper-V, because that is what Microsoft recommends. I'll keep you posted.

  • Hi Jeroen,

    Thanks for the numbers, highly interesting!  We seem to get fairly decent performance on most lighter AX operations using vmware - WIn2008R2, but as soon as you run heavier operations then virtual AOS servers are significantly slower in our case. I believe you need to make sure that you have enough disk performance as this is very, very crucial for AX.  

  • This is in my opinion one of the more interesting threads in a long time.

    Jeroen:

    Your findings and measurements under Hyper-V will be very interesting especially if the underlying hardware and infrastructure will be the same as for your previous test under VMWare (comparable). I haven't had the oportunity to do such tests, but I also have noticed unpredictable performance when running heavy load on an AOS instance hosted on VMWare. Since virtualization brings a lot of added compexity on the infrastructure level, the discussion should also reflect this and details regarding the underlying infrastructure should be decribed. And the physical host is at risk of having bottlenecks on the network level. In addition the priority for the AOS is also important and the same is seeing the total load on the host.

    David:

    Based on my experience, an AOS instance is mostly CPU, memory and network intensive, and normally less disk intensive. What could influence a virtualized AOS instance dramatically (bad) is memory pressure where the virtual machine starts paging "inside" the virtual disk file. In general the underlying disk system where the virtual disk files are stored, is important for performance in virtualized environments.

    My general advise is to avoid virtualization of AOS if it fits into the strategy of the customer (allows physical servers) and predictable performance is important.

    One theory of mine is that the AOS service utilizes memory mapped files which in turn impacts paging (pages/sec performance counter) and paging is bad when running under any hypervisor. I really think Microsoft should advise more in this matter and lifte the discussion above the product level (Hyper-V versus VMWare and others).

    Awaiting further updates to this thread.

    Best regards

    AX2009Tech aka Back2AX, Norway

  • Personally I didnt expect much from testing with Hyper-V. But my system administrator was so kind to provide me with means to test this virtualisation technology anyway. The results left me flabbergasted(i like that word) checking the benchmark a couple of times. Hyper-V outperformed VMware... by alot.

    Together with someone from cost control and a few of my IT collegues we identified 8 'slow' processes(heavy reports and batch processing). And benchmarked them into 1 average performance index.

    AX40 (Fysical): 100%

    AX50 (Fysical): 109% (9% slower in average)

    AX50 (Hyper-V): 131% (31% slower)

    AX50 (VMware): 176% (76% slower)

    One would expect that you could lose 10 up to 20% performance when virtualising a server. And Hyper-V gives us this expected performance loss (when compared to AX50 fysical). My system administrator thinks it might have to do with VMware being a OS independable technology, and Hyper-V being specifically designed for W2008 R2. Which all of our virtual servers are equipped with. I must admit I don't have much knowledge on this matter though.

    What I do know, is that we're switching soonish to an all Hyper-V rollout.

  • Interesting...

    I think the theory of your system administrator can be a good one and (but?) my next general question based on this is What about all the customers running AOS on WMWare if this is true for all environments? MS states that VMWare is supported (Virtual server support) and I guess most customers are running VMWare (historical reasons).

    I think the most important missing fact right now is the differences in the underlying infrastructure between your VMWare and Hyper-V implementations.

    The 9% decreased performance for a physical implementation of AX 2009 is reasonable (AX 2009 has a heavier footprint compared to AX 4.0).

    Best regards

    AX2009Tech aka Back2AX, Norway

  • I think the most important missing fact right now is the differences in the underlying infrastructure between your VMWare and Hyper-V implementations.

    VMware was shut down, Hyper-V got assigned the same resources on the same machine. The ISCSI discs where switched. Afaik. There is no differerence in underlying infrastructure. Not a fysical difference anyway.

    The other question you pose is an interesting one. Is this an isolated case? As you can guess, I do not know. I can only report my findings, and maybe another user can fill us in on this one :-)

  • Thanks Jeroen! Very nice!  I haven't had a chance to test Hyper-V and I think this is a real eye opener.

    @AX2009Tech - Perhaps I need to clarify, you are correct in that the AOS instance when running on a physical server is dependent on CPU, memory, network - and disk, well that isn't very important. However that is running on physical hardware. If you are running virtual images do not underestimate the need for fast disks on the physical host. It is quite crucial and quite often not dimensioned properly.

    Some numbers from our environment (running ESX version 4 and using a SAN) - : When I compile I get an average of 15000 context switching/sec on vmware, and on a three year old physical server I get about 65000.... This is only one measure point though. In english this means compiling the entire application takes 2 hours and 10 minutes on a virtual box, and about 55 minutes on the physical server (and both running db on the same physical SQL 2008 cluster).

    We are also running a fully virtual Citrix farm using Xenapp 6. This works fine with the AX client apart from excel imports and exports which are very slow.

  • Just to update this thread. If you are in the position to choose from virtual vs. physical I think you do have enough information on this currently.

    If you are still going virtual and can choose between Hyper-V and VMware then I think that Joeren's numbers certainly is enough of an answer (or as good as it gets right now).

    For us who still are not happy about this (likely due to existing investments) -  I am currently testing running one AOS only vmware running WIN2008R2 on a new ESX4.1 server running late model hardware. I will only compare one thing - and that is to compile the same application. I will post  my results here.

  • hi, guys:

    I was invited by someone to read this forum. First of all, we have AX 50 running on physical boxes not virtual box although we are intending to switch platform over since it will be great for our DR plan.

    Second, I'm more close to Vmware side from knowledge wise and I would like to know how you guys setup Vmware ESX box since there are so much layers and things you can tune.

    I would suggest to ask your Vmware admin to do VM layer performance report, ESX(i) layer performance report.

    If your datastore is sitting on iSCSI, also run vscsiStats to see virtual disk performance on ur AOS or SQL.

    Double check whether your SQL (on VMWARE) followed the best practice and also would like to know which SCSIcontroller is installed on AOS and network card as well.

    Tks.

  • has anyone heard the news from David ? it is almost three months from his last post ?

    I read all posts here, but I do not see what specification of physical server used and what specification VMserver and storage used to run AX2009.. specially on VMWARE, was it running on 4 cores or 6 cores or 8 cores ? how big memory configured to run the test ?

    can anyone here relate the test posted on this forum with this test geeksilver.wordpress.com/.../vmware-vsphere-4-cores-vs-8-cores-real-case-study-dynamics-ax-2009-mrp ? and explain it  here ?

    I am currently evaluating ERP products for the company I work for (SAP,Oracle, Lawson, AX2009). The thing is, we are just starting a VMWARE project (using Cisco UCS an NetApp).

    The master plan is, that we would like to run the ERP on VMWARE. If AX2009 has a performance issue running on it, well, it will become an important factor to be considered. I dunno if I can convince my directors to buy a new physical server just to run AX2009.

    I also realize this issue appears because Microsoft has Hyper-V, closest competitor right now with VMWARE, while other ERP vendors like SAP don't have this interest, so they [as far as I know] can run on VMWARE or Hyper-V.

  • Sorry for the delay as too much other stuff got in the way, but after many turns we managed to get our compile times down considerably by moving to a newer less loaded ESX server which is only running 4 other vmware images. It helped quite a bit but we are still not really where we want to be.

    In general there is a big difference between ESX 3.5 and 4.0, where the latter is considerably faster.

    Ironically our fastest server though is still a 3,5 year old HP proliant 460 something with two AMD dual core CPU's running Win2003 64 bit OS. Of course this will not last but I think this is still pretty impressive!

    I don't think it is a question if you can run AX on vmware, it will work. If there is enough of a budget and will (and competent people), you can solve anything but the road may be quite bumpy! The question should rather be - is it worth it? How much time and effort are you going to spend if you want to run AX virtually?

    You simply need to go for gold in terms of hardware - especially if you are running VMware - while AX 2009 will run really well on older hardware (just make sure your CPU's have lots of GHz.).

    If you are selecting AX and still want to go virtual, I think it would be a crime not to consider HyperV. In our case we already had vmware licensees and a number of ESX servers, but would we choose VMware knowing what we know today? :)

  • Thanks David... I keep in mind what you mention here..

    but still, I would like to run ERP on virtualization..

    We'll do best to build ERP on virtualization in a reasonable effort and time.

    But that is not all we consider to have it.

    we consider the future effort and time we need to maintain the system or when something happens with it.

    we also consider the resource consumption for the system in the future (electrical, storage, memory, space, etc). And That is not only for ERP server but the entire data center.

  • To be frank I don't see this as an AX issue and it never really was one. This is IMO nothing else but a virtualization issue where too many virtual farms are not dimensioned properly, not setup correctly or simply overloaded.

    AX 2009 is supported to run on platforms that follow the Server Virtualization Validation Program. So that should pretty much end that discussion. I have seen many development, test and production environments running on virtual servers for many years now (way before AX2009) and there has been a constant struggle between AX people trying to explain to VMware admins that performance is simply not good enough, only to get answers back that it "appears" as if nothing is wrong on the VMware hosts, or "the other vm's runs fine". AX is not the only one who had issues, just about every heavy application have had issues going virtual. This is however the past and the future is looking good!  

    With newer versions of applications, newer hardware, newer CPU's, newer OS with a lot more support for virtualization, newer virtualization software,  things are improving very rapidly. The future is virtualization, there can be no doubt about that - but the question is rather, can you maintain and support a virtual environment? Should you get performance issues, can these be solved quickly? This is rather about managing risks and knowing what resources you have available. If you already have 200 or so virtual servers running in a data center then you most likely already have resources available.

    Our AX environment is a mixture of  physical and virtual servers totaling about 40 servers and we will go fully virtual later this year. I am just happy I have a physcial server on my desk so I can check that performance!