Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Dynamics 365 Community / Forums / Small and medium business | Business Central, N... / Importing more than 20...
Small and medium business | Business Central, N...
Suggested answer

Importing more than 200k records takes more than 2 hours

(1) ShareShare
ReportReport
Posted on by Anxhela C 6
Hi all, 

Every day the system will receive 5 files of different types. These files have more than 200k records/each. I want to import these files into BC and then explode them with another table to put the data that these files have.

The files are in .csv with 10 columns each.
What I have worked on so far have been two ways of importing the file into a Stagging file and then with a job queue to read the records, and then insert the lines into a Price Stagging.
1. Reading the file with xmlport and inserting records in price staging, and
2. Since the columns of the file are different, I used Data exchange definition for fields mapping and insertion in file staging.
 
But both methods are not efficient as the import takes more than 2 hours of time.
Is there a more efficient way to import?
 
 
P.s.: Not all lines imported into the staging table will be inserted into the final table.
  • Jun Wang Profile Picture
    Jun Wang 7,440 Super User 2024 Season 2 on at
    Importing more than 200k records takes more than 2 hours
    you could search a 3rd party app to import data into BC.
  • Matteo Fiocco Profile Picture
    Matteo Fiocco 96 on at
    Importing more than 200k records takes more than 2 hours
    Hi,
     
     

    Optimize the XMLport or Data Exchange Definition

    • Batch Processing: Instead of processing all records in one go, consider batching the records. This can help manage memory usage better.
    • Buffer Table Usage: Use a buffer table to temporarily hold data during the import process, reducing the overhead during insertion into the final staging tables.
    • Minimize Data Transformation: Ensure that any data transformation is done after the initial import to the staging table, rather than during the file import.

    Use .NET Interoperability for File Reading

    • System.IO for Fast File Reading: Instead of relying solely on XMLports or Data Exchange Definitions, you can use .NET interoperability to read the file directly into memory. The System.IO.StreamReader class can read large files more efficiently.
    • Bulk Insert Techniques: After reading the file into memory, you can process the data and insert it in bulk into the staging table using AL code. This can significantly reduce the time taken to insert records.

    Parallel Processing

    • Split Files: If possible, split the large files into smaller chunks and process them in parallel using separate job queue entries or background sessions. This distributes the workload and speeds up the overall process.
    • Multi-Threading: Though Business Central runs on a single-threaded AL runtime, you can simulate parallelism by queuing multiple imports simultaneously if your infrastructure supports it.

    SQL Server Integration

    • Direct SQL Integration: For very large datasets, consider using SQL Server Integration Services (SSIS) or directly writing to the Business Central database tables via SQL (with caution). This method requires a deep understanding of Business Central’s database schema and must be used carefully to maintain data integrity.
     
     
     
  • Suggested answer
    YUN ZHU Profile Picture
    YUN ZHU 75,595 Super User 2024 Season 2 on at
    Importing more than 200k records takes more than 2 hours
    Hi, As far as I know, there are three ways to import CSV: XMLPort, CSV Buffer, and Power Automate. You can try other methods.
    More details: 
     
    But this should be caused by the quantity. If the number of imports can be reduced, I think the efficiency will be greatly improved, or try to split the files.
     
    Hope this can give you some hints.
    Thanks.
    ZHU

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Congratulations 2024 Spotlight Honorees!

Kudos to all of our 2024 community stars! 🎉

Meet the Top 10 leaders for December!

Congratulations to our December super stars! 🥳

Get Started Blogging in the Community

Hosted or syndicated blogging is available! ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,642 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,371 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans