Finance | Project Operations, Human Resources, ...

Import Million records in D365 FO

(0) Share

Report

Posted on by vicky1234

hi,

I have a requirement to upload 5 million customers in D365 FO and need to explore diffferent option which are faster and better.

I also want to understand how can we explore the option of "ODBC" and "azure sql db " while importing the records via DMF in a way that it can help me import the records faster.

I think we can use csv instead of excel and it would shrink the file size.

Kindly share your experience on this.

Thanks,

I have the same question (0)

All responses (9)

Answers (0)

Suggested answer

SouravDam 11,730 on at

Like (0)

Report

RE: Import Million records in D365 FO

Hi vicky1234,

You can use Data management for "Customers V3" data entity.

Now talking about huge data.. you can create multiple load one after another. Every load may contain say around 25k new records. Once one batch ends, you can load another batch with new 25k records.

Breaking data in multiple loads will give you much faster master data upload in system.

Regards,
Sourav

Was this reply helpful? Yes No
vicky1234 66 on at

Like (0)

Report

RE: Import Million records in D365 FO

hi Saurav,

Thanks for the reply Is there any std multi threading approach as well which we can apply over here.

Was this reply helpful? Yes No
Suggested answer

SouravDam 11,730 on at

Like (0)

Report

RE: Import Million records in D365 FO

Hi vicky1234,

As of now.. data management is the best possible way to load huge data at a time.

Another option is Excel add-ins but it has limitation and cannot push much faster than Data management.

Also you can check with developer if they create any staging table and from there they pick the data to push in main table. It will also help you to track what are the data that are pushed and what is still pending.

Example, we created a custom form called "Customer entry #" interface where data is getting loaded first and from there, through batch process, new record (and updated record) are pushed in the main Table.

But unfortunately I cannot share the code detail as part of privacy. You can show your developer and discuss.

Regards,
Sourav

Was this reply helpful? Yes No
Suggested answer

Bharani Preetham Pe... 3,634 Moderator on at

Like (1)

Report

RE: Import Million records in D365 FO

We have a standard function for this in the DMF. Use the following link.

stoneridgesoftware.com/.../

Was this reply helpful? Yes No
Suggested answer

SouravDam 11,730 on at

Like (0)

Report

RE: Import Million records in D365 FO

Nice thread Bharani Preetham Peraka.

One question for my own understanding.. as I see all these batches are running at the same time which is pre-defined as per the framework parameters and no need any manual intervention eveytime to load data.

But will it impact anyway to the number sequence in case if customer account number is set to 'Continuous' ?

And how such batches usually work.. means is it like one batch will start after the first one ends? Because I see atleast 3-4 batches are executing at the same time!

Regards,
Sourav

Was this reply helpful? Yes No
Suggested answer

Bharani Preetham Pe... 3,634 Moderator on at

Like (1)

Report

RE: Import Million records in D365 FO

Manual intervention is not required. It is something like a normal batch job but here the set batch jobs will run simultaneously. Earlier I did the same for my custom table.

Are you telling if set to continuous, the number sequences will hung? I guess there wont be any issue on that. But if the number sequence exceed then it will cause an issue. Also while setting this execution parameters, you should be very careful about setting the tasks because if it is more than the capacity of system, it will cause the d365 to hang. So be very careful while setting this.

Was this reply helpful? Yes No
Martin Dráb 236,570 Most Valuable Professional on at

Like (1)

Report

RE: Import Million records in D365 FO

Sometimes is benefitial to create a custom data entity with just those things that you need for the particular performance-critical import. Removing unnecessary logic or taking some shortcuts (that you can affort because you know the imported data) may have significant impact on performance.

My strong recommendation is to avoid continuous number sequence whenever possible. Why exactly do you want to use it for customers?

Was this reply helpful? Yes No
vicky1234 66 on at

Like (0)

Report

RE: Import Million records in D365 FO

Hi Martin

thanks for the reply.

I have a requirement to import around 5 million customers in the D365 FO environment during the go live phase and hence I am planning out different options that I should consider for importing it.

Also since the client is having a third party software however it uses sql as backend, I wanted to check if I can leverage the option of "Azure Sql DB" and "ODBC" connection .

I haven't used this option before but would like to understand more on this.

Was this reply helpful? Yes No
Martin Dráb 236,570 Most Valuable Professional on at

Like (0)

Report

RE: Import Million records in D365 FO

As far as I know, ODBC isn't and never was supported by F&O.

And if think that "Azure SQL DB" is used only by BYOD export.

Was this reply helpful? Yes No