AX Performance

(0) Share

Report

Posted on by Sasha

815

Dear All,

When analyzing the performance issues in our AX 2012 PROD environment we came across the situations that connection thought some specific servers is slower and sometimes this slowness becomes worst. In other words the system is noticeably slower when accessed through specific AOS, or client. When analyzing the latency we discovered, that the latency between those “slow” servers is always 2-3 ms, while the latency between the “fast” servers is always less than 1 ms.

E.g. it takes 6 sec to open the client through the “fast” servers and not less than 15 sec. for the “slow” ones. The performance difference inside ax is also noticeable when browsing the forms for example.

We suspect that sometimes we have the peak periods when the latency grows even more. Therefore the question is, can it be that such small difference in the latency 1 ms VS 3 ms can affect the performance of AX this much? Could you please share your thoughts and experience?

Thanks.

Sasha.

*This post is locked for comments

I have the same question (0)

All responses (7)

Answers (0)

Martin Dráb 237,967 Most Valuable Professional on at

Like (0)

Report

If the difference in latency is 2 ms, it will result in a difference of 9 seconds if you make 4500 calls. So, it's possible, but it would mean that you have a problem in your system that causes a huge amount of calls between client and server. The latency would make the problem bigger, it wouldn't be the root cause.

There may be manu other responsible - I suggest you measure where the time is consumed before starting to focus on any particular area.

Was this reply helpful? Yes No
Suggested answer

guk1964 10,888 on at

Like (0)

Report

Are you using a WAN or a LAN?

If a WAN then do you have thin client connectivity (usually essential to minimise latency issues).

If a LAN then consider what is the difference in what the users are doing to see if that gives any clues.

Similarly are there any differences is server spec e.g faster disks, or whether some servers are virtualised? (not a good idea for production).

Was this reply helpful? Yes No
Suggested answer

Vilmos Kintera 46,149 on at

Like (0)

Report

The formula for network architecture around AX is simple: communication between server components must be dedicated. That means Terminal server<->AX<->SQL Server network paths should be undisturbed by any other network traffic. Also if you use a SAN under your SQL Server, the SAN should be dedicated to your SQL only, and the networking should not have any other traffic as well. If you experience delays, check if there is anything else on the network, how many hops do you have, how is your switch configured, do you have the latest firmware installed, etc.

Badly designed networking and hardware infrastructure does impact negatively any ERP system indeed. But before you dig deeper, you should get an expert to check your environment from software perspective, as typically the problem is not with the hardware if it was working fine at any stage.

First things first, validate if you are following Microsoft best practices for your SQL setup (TempDB # of files, Trace Flags, dedicated disks for data/logs, index/statistics maintenance, etc.). I've linked some of the MIcrosoft PFE's recommendations and my own 2-cents on the topic here:

https://www.daxrunbase.com/2016/05/11/ax-database-tuning-and-maintenance/

Once you verify everything is fine there, you may move on with the investigation to see what is happening, i.e. you get bad query plans, or the query plans are rotating out fast since your SQL Server has internal memory pressure, and similar things.

Was this reply helpful? Yes No
Suggested answer

guk1964 10,888 on at

Like (0)

Report

All good advice. Other simple things that have a big effect in most cases.

Set the Power option to high on both servers and laptops (and retail store databases) - you are more worried about performance than saving energy. Balanced is the default setting - and it in some cases that also causes instability.

Check your MAXDOP setting.

Don't allow Microsoft to auto update/windows/or office products on your server. Manage that yourself out of hours.

Was this reply helpful? Yes No
Sasha 815 on at

Like (0)

Report

Dear All,

In our case it's not the problem of SQL server,

SQL server is optimized, though we have some expensive queries from time to time, we see related performance degradation and can explain it.

Our issue is that when we connect to the system through AOS1 - it is really fast and working fine. When we open the same system through AOS2 - it's still acceptable but noticeably slower. So it takes some seconds more to open AX client, it takes 0.5 seconds or 1 sec. more to open the PO form, etc... So here the system is slower in general. When we do the Tracert from AOS 1 to the SQL Server, it shows 2 hops (AOS to Switch/Router) and then switch to the SQL server, on both hops the latency is <1 ms always which is good. When we do tracert from AOS 2 to SQL Server - still 2 hops with the same router/switch, however the hop1 latency is always 2-3 ms, while the latency of hop 2 is still < 1ms. For me that is a clear indication that the is some network related problem with AOS2 machine. As this is 1Gbit LAN - as per my knowledge the latency must allays be <1ms, at least 99.99% of the cases.

Problem is, that when we discuss with the network team, it's hard to convince that such difference <1 ms vs 2-3 ms can have an impact on AX perf. From my POV it can, especially if the network gets overloaded and latency grows few times, in case of AOS 1 - latency can still be within 1ms, for the case 2 it might be already 6ms, and I think users will feel this.

Please let me know what you think.

Thanks.

Was this reply helpful? Yes No
Suggested answer

Vilmos Kintera 46,149 on at

Like (0)

Report

If it is provably working from one AOS it is clearly an issue outside of AX/SQL configuration.

Like I suggested, check the network for: firmware updates (should be the most recent stable version) and configuration (jumbo frames/throttling/quality of service), and other network traffic routed through the same switch

(should be dedicated to your servers only) on the switch. Check cables, faulty ports, monitor the packets.

Also make sure you are using the recommended driver in Windows for the network card, and if the settings are correctly configured. Check for any differences between the 2 AOS instances, if their hardware are identical including the NIC.

As a last resort if infrastructure claims that it is all OK, you might try to rebuild the AOS starting with a clean Windows install and see if the issue goes away.

Was this reply helpful? Yes No
Suggested answer

guk1964 10,888 on at

Like (0)

Report

This may be too obvious to mention but if the issue is AOS related then have checked the Dynamics AX AOS Event Log? When you have many Dynamics AX AOS Servers, you can use DynamicsPerf to collect the events. The VBS script AOSANalysis.CMD will automatically retrieve all Dynamics log from all AOS where there is an active Dynamics AX AOS configuration, for the last 14 days. Execute the following query from DynamicsPerf to find the latest events: Select top 25 * from AOS_EVENTLOG order by Time_Written Desc

If no joy with that then try to capture the Dynamics AX Event Trace on the AOS server where the issue occurs. Start the Event Tracing for Windows (ETW) within Windows Performance Monitoring or use PowerShell script. (Only run the trace for few minutes because it collects all the sessions connected to the AOS and its size can easily reach several GB). Import the trace into Trace Parser and look for top Call stacks and SQL Statements order by duration and count.

Windows Performance Monitoring, may not be the answer look but it can help to verify e.g. latency of the Disks especially for Data and Log files of Dynamics AX and TempDB databases, and availability of the Memory, utilization of the processors and networking bandwidth. Also check the usage of all resources for all critical server roles such as SQL Server, AOS Server, Reporting Server and Terminal Server.

To set up the Windows Performance Monitoring see: . https://blogs.msdn.microsoft.com/axinthefield/setting-up-windows-performance-monitor-templates/

I mentioned power setting. Unless you have two servers with different settings this may not solve your problem but may still help with general performance. Something I read in a previous post and made a note refers to HP servers which needs a bit more awareness. On the ILO boards is a setting called "Power Regulator" which by default is set to "Dynamic". With this Setting the HP Server uses the Power Settings configured on OS Site (which is set to "High Performance"). If you change this Setting to "Static High Performance", then it directly has an effect on the average power consumption of the Server (maybe a 30% increase - however if you try a major process such as mrp or inventory recalculation before and after changing and measure the average run time this can show almost a 50% increase in performance.

(Are your AOS on separate physical servers or on the same server? What version of SQL. (Standard or Enterprise? which release?)

Was this reply helpful? Yes No

Community site session details

AX Performance

Helpful resources

Quick Links

Responsible AI policies

Neeraj Kumar – Community Spotlight

Congratulations to the November Top 10 Community Leaders!

Leaderboard > 🔒一 Microsoft Dynamics AX (Archived)

Featured topics

Product updates

Community site session details

AX Performance

Helpful resources

Quick Links

Responsible AI policies

Neeraj Kumar – Community Spotlight

Congratulations to the November Top 10 Community Leaders!

Subscribe to this forum!

Select categories

Leaderboard > 🔒一 Microsoft Dynamics AX (Archived)

Featured topics

Product updates