Skip to main content

Notifications

Microsoft Dynamics AX (Archived)

AX Performance

(0) ShareShare
ReportReport
Posted on by 815

Dear All,

When analyzing the performance issues in our AX 2012 PROD environment we came across the situations that connection thought some specific servers is slower and sometimes this slowness becomes worst. In other words the system is noticeably slower when accessed through specific AOS, or client. When analyzing the latency we discovered, that the latency between those “slow” servers is always 2-3 ms, while the latency between the “fast” servers is always less than 1 ms.

E.g. it takes 6 sec to open the client through the “fast” servers and not less than 15 sec. for the “slow” ones. The performance difference inside ax is also noticeable when browsing the forms for example.

We suspect that sometimes we have the peak periods when the latency grows even more. Therefore the question is, can it be that such small difference in the latency 1 ms VS 3 ms can affect the performance of AX this much?  Could you please share your thoughts and experience?

Thanks.

Sasha. 

*This post is locked for comments

  • Suggested answer
    guk1964 Profile Picture
    guk1964 10,877 on at
    RE: AX Performance

    This may be too obvious to mention but if the issue is AOS related then have checked the  Dynamics AX AOS Event Log? When you have many Dynamics AX AOS Servers, you can use DynamicsPerf to collect the events. The VBS script AOSANalysis.CMD will automatically retrieve all Dynamics log from all AOS where there is an active Dynamics AX AOS configuration, for the last 14 days. Execute the following query from DynamicsPerf to find the latest events: Select top 25 * from AOS_EVENTLOG  order by Time_Written Desc

    If no joy with that then try to capture the Dynamics AX Event Trace on the AOS server where the issue occurs. Start the Event Tracing for Windows (ETW) within Windows Performance Monitoring or use PowerShell script. (Only run the trace for few minutes because it collects all the sessions connected to the AOS and its size can easily reach several  GB). Import the trace into Trace Parser and look for top Call stacks and SQL Statements order by duration and count.

    Windows Performance Monitoring, may not be the answer look but it can help to verify e.g.  latency of the Disks especially for Data and Log files of Dynamics AX and TempDB databases, and availability of the Memory, utilization of the processors and networking bandwidth. Also check the usage of all resources for all critical server roles such as SQL Server, AOS Server, Reporting Server and Terminal Server. 

    To set up the Windows Performance Monitoring see: . https://blogs.msdn.microsoft.com/axinthefield/setting-up-windows-performance-monitor-templates/

    I mentioned power setting.  Unless you have two servers with different settings this may not solve your problem but may still help with general performance.  Something I read in a previous post  and made a note refers to  HP servers which needs a bit more awareness. On the ILO boards is a setting called "Power Regulator" which by default is set to "Dynamic". With this Setting the HP Server uses the Power Settings configured on OS Site (which is set to "High Performance"). If you change this Setting to "Static High Performance", then it directly has an effect on the average power consumption of the Server (maybe a 30% increase - however if you try a major process such as mrp or inventory recalculation before and after changing and measure the average run time this can show almost a 50% increase in performance.

    (Are your AOS on separate physical servers or on the same server?  What version of SQL. (Standard or Enterprise? which release?)

  • Suggested answer
    Vilmos Kintera Profile Picture
    Vilmos Kintera 46,149 on at
    RE: AX Performance

    If it is provably working from one AOS it is clearly an issue outside of AX/SQL configuration.

    Like I suggested, check the network for: firmware updates (should be the most recent stable version) and configuration (jumbo frames/throttling/quality of service), and other network traffic routed through the same switch

    (should be dedicated to your servers only) on the switch. Check cables, faulty ports, monitor the packets.

    Also make sure you are using the recommended driver in Windows for the network card, and if the settings are correctly configured. Check for any differences between the 2 AOS instances, if their hardware are identical including the NIC.

    As a last resort if infrastructure claims that it is all OK, you might try to rebuild the AOS starting with a clean Windows install and see if the issue goes away.

  • Sasha Profile Picture
    Sasha 815 on at
    RE: AX Performance

    Dear All,

    In our case it's not the problem of SQL server,

    SQL server is optimized, though we have some expensive queries from time to time, we see related performance degradation and can explain it.

    Our issue is that when we connect to the system through AOS1 - it is really fast and working fine. When we open the same system through AOS2 - it's still acceptable but noticeably slower. So it takes some seconds more to open AX client, it takes 0.5 seconds or 1 sec. more to open the PO form, etc... So here the system is slower in general. When we do the Tracert from AOS 1 to the SQL Server, it shows  2 hops (AOS to Switch/Router) and then switch to the SQL server, on both hops the latency is <1 ms always which is good. When we do tracert from AOS 2 to SQL Server - still 2 hops with the same router/switch, however the hop1 latency is always 2-3 ms, while the latency of hop 2 is still < 1ms. For me that is a clear indication that the is some network related problem with AOS2 machine. As this is 1Gbit LAN - as per my knowledge the latency must allays be <1ms, at least 99.99% of the cases.

    Problem is, that when we discuss with the network team, it's hard to convince that such difference <1 ms vs 2-3 ms can have an impact on AX perf. From my POV it can, especially if the network gets overloaded and latency grows few times, in case of AOS 1 - latency can still be within 1ms, for the case 2 it might be already 6ms, and I think users will feel this.  

    Please let me know what you think.

    Thanks.

  • Suggested answer
    guk1964 Profile Picture
    guk1964 10,877 on at
    RE: AX Performance

    All good advice. Other simple things that have a big effect in most cases.

    Set the Power option to high  on both servers and laptops (and retail store databases)  - you are more worried about performance than saving energy. Balanced is the default setting - and it in some cases  that also causes instability.

    Check your MAXDOP setting.

    Don't allow Microsoft to auto update/windows/or office products on your server.  Manage that yourself out of hours.

  • Suggested answer
    Vilmos Kintera Profile Picture
    Vilmos Kintera 46,149 on at
    RE: AX Performance

    The formula for network architecture around AX is simple: communication between server components must be dedicated. That means Terminal server<->AX<->SQL Server network paths should be undisturbed by any other network traffic. Also if you use a SAN under your SQL Server, the SAN should be dedicated to your SQL only, and the networking should not have any other traffic as well. If you experience delays, check if there is anything else on the network, how many hops do you have, how is your switch configured, do you have the latest firmware installed, etc.

    Badly designed networking and hardware infrastructure does impact negatively any ERP system indeed. But before you dig deeper, you should get an expert to check your environment from software perspective, as typically the problem is not with the hardware if it was working fine at any stage.

    First things first, validate if you are following Microsoft best practices for your SQL setup (TempDB # of files, Trace Flags, dedicated disks for data/logs, index/statistics maintenance, etc.). I've linked some of the MIcrosoft PFE's recommendations and my own 2-cents on the topic here:

    https://www.daxrunbase.com/2016/05/11/ax-database-tuning-and-maintenance/

    Once you verify everything is fine there, you may move on with the investigation to see what is happening, i.e. you get bad query plans, or the query plans are rotating out fast since your SQL Server has internal memory pressure, and similar things.

  • Suggested answer
    guk1964 Profile Picture
    guk1964 10,877 on at
    RE: AX Performance

    Are you using a WAN or a LAN?

    If a WAN then do you have thin client connectivity (usually essential to minimise latency issues).

    If a LAN then  consider what is the difference in what the users are doing to see if that gives any clues.

    Similarly are there any differences is  server spec e.g faster disks, or whether some servers are virtualised? (not a good idea for production).

  • Martin Dráb Profile Picture
    Martin Dráb 230,934 Most Valuable Professional on at
    RE: AX Performance

    If the difference in latency is 2 ms, it will result in a difference of 9 seconds if you make 4500 calls. So, it's possible, but it would mean that you have a problem in your system that causes a huge amount of calls between client and server. The latency would make the problem bigger, it wouldn't be the root cause.

    There may be manu other responsible - I suggest you measure where the time is consumed before starting to focus on any particular area.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Announcing Our 2025 Season 1 Super Users!

A new season of Super Users has arrived, and we are so grateful for the daily…

Vahid Ghafarpour – Community Spotlight

We are excited to recognize Vahid Ghafarpour as our February 2025 Community…

Congratulations to the January Top 10 leaders!

Check out the January community rock stars...

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 292,111 Super User 2025 Season 1

#2
Martin Dráb Profile Picture

Martin Dráb 230,934 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans