Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Dynamics 365 Community / Forums / Finance forum / DMF Data import perfor...
Finance forum

DMF Data import performance - running out of memory

(0) ShareShare
ReportReport
Posted on by Frank Bruemmer 343

Hello,

I'm importing customers, vendors, etc into a tier 2 environment with 32GB RAM on the AOSes. Larger imports freeze after a while, after 3800 records the last time. According to the even viewer on AOS1, I'm having memory issues : 

Object Server DynamicsAXBatchManagement:
Unhandled exception
More Information: Out of memory.

We're running 10.0.04/PU28.

I find it hard to believe that in 2019 this causes an unhandled exception that leads to a stop of the import. There is nothing else going on in this pretty much vanilla environment.

Does anyone have some experience or advice other than buying more RAM or breaking the data into digestable chunks ?

Best regards

Categories:
Data Integration
  • André Arnaud de Calavon Profile Picture
    André Arnaud de Cal... 291,280 Super User 2024 Season 2 on at
    RE: DMF Data import performance - running out of memory

    Hi Frank,

    Please keep this thread updated once you did your new test. If it fails again and no customizations are involved, you can also create a ticket for Microsoft Support.

  • Frank Bruemmer Profile Picture
    Frank Bruemmer 343 on at
    RE: DMF Data import performance - running out of memory

    This occurred on multiple instances and my primary goal was to get the data loaded so other team members could work with it.  This has been loaded now and is used, I can't run any tests at this time in that environment. I will try the same file in a few days when I get another environment.

  • Frank Bruemmer Profile Picture
    Frank Bruemmer 343 on at
    RE: DMF Data import performance - running out of memory

    Andre,

    I used the Customers V3 data entity using a 'standard acceptance test' tier 2 environment, no modifications. I have too imported many more rows in the past and considering that DMF is supposed to be the 'high-volume' interface I'm disappointed that 32GB don't get you anywhere. We tried this a few times and it failed a few times. Yesterday we fired that import off before going home. There were no other things going on in that environment.

  • André Arnaud de Calavon Profile Picture
    André Arnaud de Cal... 291,280 Super User 2024 Season 2 on at
    RE: DMF Data import performance - running out of memory

    Hi Frank,

    Can you indicate which exact entity was used when you experienced this issue? Is it one out of the box or a custom one? I have imported over 10000 records on both standard and custom entities with ease. I do agree with Martin to retry the same import and possibly do some monitoring.

  • Martin Dráb Profile Picture
    Martin Dráb 230,235 Most Valuable Professional on at
    RE: DMF Data import performance - running out of memory

    Does it mean that you didn't try to reproduce it by running the same import again?

  • Frank Bruemmer Profile Picture
    Frank Bruemmer 343 on at
    RE: DMF Data import performance - running out of memory

    Thanks for your response. I loaded data for other entities such as vendors w/o problems, but none had as many rows as the customers. I only noticed the exception in the event viewer. This happened around 8PM EST last night. Like I said, there is nothing else going on in that environment. I'm the only one accessing it and the AOS pretty much idles now with memory utilization at around 20%. I'm pretty sure that import is responsible for the extraordinary memory consumption. The easiest way for me is probably to break up that file into three or four chunks and for the future, to use an environment with more RAM. I don't recall having these issues in previous projects where I think I had more memory.

    It also appears that there is no virtual memory enabled on these AOSes. There is about 8GB of currently allocation virtual memory. That is still short of the total of 56GB I normally have at my disposal. I wonder if adjusting this setting will make a difference. I know, paging is terrible for performance. But it is still better than provoking an exception. These import jobs may run slowly but steady during night time. Any opinion on this anyone ?

  • Martin Dráb Profile Picture
    Martin Dráb 230,235 Most Valuable Professional on at
    RE: DMF Data import performance - running out of memory

    Does it mean that it happens to you with any of the entities you mentioned?

    Have you collected more information about memory consumption? For instance, how much memory was available before you started the import and which process consumed the most memory during the import?

    You could even utilize a memory profiler tp analyze which exact objects are consuming the memory, but I would leave it at a later phase of your analysis.

    Unfortunately memory still has finite capacity even in 2019 and if there isn't enough memory to continue, the .NET Frameworks throws System.OutOfMemoryException. While I agree that we shouldn't get into such a situation in this case, when it happens, the behavior you see looks correct to me.

  • Suggested answer
    nmaenpaa Profile Picture
    nmaenpaa 101,156 on at
    RE: DMF Data import performance - running out of memory

    You can't buy more RAM on the Microsoft managed systems. I suggest you try to change your import files - split to smaller pieces.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

November Spotlight Star - Khushbu Rajvi

Congratulations to a top community star!

Forum Structure Changes Coming on 11/8!

In our never-ending quest to help the Dynamics 365 Community members get answers faster …

Dynamics 365 Community Platform update – Oct 28

Welcome to the next edition of the Community Platform Update. This is a status …

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,280 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,235 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans