web
You’re offline. This is a read only version of the page.
close
Skip to main content
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Supply chain | Supply Chain Management, Commerce / Huge data uploading Fr...
Supply chain | Supply Chain Management, Commerce
Suggested answer

Huge data uploading From legacy system taking very long time

(0) ShareShare
ReportReport
Posted on by Usama Tur Razzam 115

Hi Experts,

We have a very huge data from our legacy system which needs to upload to FinOps on our go live. But the problem is that data is too big i.e. we will have 50k customers and vendors, 50k Projects, 0.5 million Sales Orders, 0.5 million Purhcase Orders, 0.1 million General Journals etc.

The approach to upload the data is below:

We upload the data from Legacy system to our staging tables in SQL (since we are on prem) and from there we run our business logic in runnable class and do the processing on the data. This approach is taking very very long time like per document almost 2-3 seconds. Can someone share opinion on how we can reduce the time and what should be the strategy given such huge volume of data?

Thanks in Advance.

I have the same question (0)
  • WillWU Profile Picture
    WillWU 22,361 on at
    RE: Huge data uploading From legacy system taking very long time

    Hi Usama,

    Batch data APIs are designed to handle large-volume data imports and exports.

    Please check:

    docs.microsoft.com/.../integration-overview

  • Suggested answer
    Andreas Raithel Profile Picture
    Andreas Raithel 4,963 Moderator on at
    RE: Huge data uploading From legacy system taking very long time

    Hy Usama,

    you can set entity specific values for max record count and Import Task count on the framework paramaters/Entity settings /configure entity execution parameters form. This assigns dedicated hardware/cpu resources for the threads on the batch servers and can speed up significant DMF operations. Using DMF we could achieve Uploads of bulk data migration depending on the entity up to 100-300 ms per record, which means that the run time of several hours was acceptable.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

News and Announcements

Join experts in person to explore low code and AI agents at the Power Platform Community Conference sponsored by Microsoft

Quick Links

Responsible AI policies

As AI tools become more common, we’re introducing a Responsible AI Use…

Abhilash Warrier – Community Spotlight

We are honored to recognize Abhilash Warrier as our Community Spotlight honoree for…

Congratulations to the September Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > Supply chain | Supply Chain Management, Commerce

#1
CA Neeraj Kumar Profile Picture

CA Neeraj Kumar 947

#2
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 248 Super User 2025 Season 2

#3
BillurSamdancioglu Profile Picture

BillurSamdancioglu 176 Most Valuable Professional

Last 30 days Overall leaderboard

Featured topics

Product updates

Dynamics 365 release plans