Finance | Project Operations, Human Resources, ...

CSV import in batch job (using sysOperation Framework)

(0) Share

Report

Posted on by Tushar G

Requirement :

I am importing forecast records (using CSV) in AX and then generating forecast lines during import.

I want this process to run in batch job.

Current Solution :

There is a dialog (Form) that opens up with parameters and then below code is triggered on click of OK button to upload file.

The import is working fine but for large chunk of data its taking too much time and is blocking the UI till the import completes.

1. Is it possible to run the upload progress in batch job?

Part of my customisation :

fileUpload = File::GetFileFromUser() as FileUploadTemporaryStorageResult;
if(fileUpload && fileUpload.getUploadStatus())

{

file = AsciiStreamIo::constructForRead(fileUpload.openResult());

}

After this, I am reading content of the file and processing forecast.

Another Solution : Browse control

I wanted to use a simple browse control 'FileNameOpen' but i think it's obsolete as there is no FileUploadStategy created for it.

2. Can anyone tell me if there is still a 'Browse' control that we can use to read the file from the path browsed?

I have the same question (0)

All responses (19)

Answers (0)

Martin Dráb 237,880 Most Valuable Professional on at

Like (0)

Report

1. Are you sure you can't use Data Management? It already does what you're asking for (i.e. executing imports without blocking UI and optionally running in batch) and you wouldn't have to develop it again.

2. You can utilize the File Upload control in your forms, but the old way that worked when AX client was a Windows application doesn't work in web browsers.

Was this reply helpful? Yes No
Tushar G 35 on at

Like (0)

Report

Thanks Martin for a quick response.

I can't use Data Management because I have to apply Forecast logic using the parameters provided by user.

There is no browse control as its browser based client.

Is it possible to create a batch job to implement the import of CSV after user provides input from dialog?

Was this reply helpful? Yes No
Suggested answer

Sukrut Parab 71,710 Moderator on at

Like (0)

Report

CSV import through custom batch job is not possible. You could utilize data management or data management packaged API to import your values and that can be used to run it in batch mode. Did you consider using these options?

Because you mentioned your batch job works but it takes time for large volume its also possible to add multiple task in your custom batch job through multi threading. You can search on the web how to multithread batch to get answer and see if it makes any improvement.

Was this reply helpful? Yes No
André Arnaud de Cal... 301,025 Super User 2025 Season 2 on at

Like (0)

Report

Hi Tushar,

What exact parameters will be filled by the user? What is the source of the file? Would there be an option to provide the parameters as columns before importing it? In that case, you can use Data management.

You can also consider using Microsoft Power Automate. Then you can create a Flow which has a "button" start. Then you can add parameters to be filled by the user as well.

Was this reply helpful? Yes No
Tushar G 35 on at

Like (0)

Report

Thanks Sukrut for the response.

Yeah, I think too that it's not possible as of now to run CSV import in SysOperation batch jobs.

My current solution is not a batch job instead it is a dialog form which after run() opens another upload popup to browse the file and then processes the file in the same screen with user blocked till the processing is complete.

I have thought of Data management as one of the options only if Batch job is not possible with my requirement.

Was this reply helpful? Yes No
Martin Dráb 237,880 Most Valuable Professional on at

Like (0)

Report

Why exactly is it impossible? Aren't data management and the recurring integration batch proofs that it's possible?

You can't grab files on client machines in a batch job, but when the file is uploaded to server, why couldn't you process it in batch? You could store the file somewhere and pick it in a batch, or you could quickly parse the file and store data in a table (thing about staging tables in data management) and use a batch to read data from staging and generate forcecast lines.

Also, non-blocking processing doesn't necessarily require a batch. Again, you can see it in data management (e.g. DMFExportImportData.importDataAsync()).

Was this reply helpful? Yes No
Tushar G 35 on at

Like (0)

Report

Hi Andre,

The parameters are below.

I could ask for the requirement to change and provide the parameters in the CSV file instead because the concern is only regarding performance.

But the purpose of this post was to learn if there is a way to provide browsing option in the same dialog with the other parameters and can also be run in batch.

As I said earlier, I have thought of Data Management as another option to execute this requirement.

I really like the power automate option to create flow for user input.

Thanks for the suggestion. I am sure these options will work. :)

Was this reply helpful? Yes No
Martin Dráb 237,880 Most Valuable Professional on at

Like (0)

Report

Regarding "if there is a way to provide browsing option in the same dialog with the other parameters": Yes, it's possible and you can again see it in data management. When creating an import job, you upload a sample file in the same dialog where you select the data entity, file format etc.

Was this reply helpful? Yes No
Tushar G 35 on at

Like (0)

Report

Thanks Martin for the insight.

I see there are ways to go across this problem.

You said - "You could store the file somewhere and pick it in a batch."

Is the batch executed separately or is it like explained below?

When file is uploaded, users clicks OK. I will store the file in a variable and pass to a class as a parameter. But I am unable to think of a way to execute the class as a batch through X++ code at the same moment.

In both of your scenarios, I need to execute class in batch through x++ code ( not running manually).

Could please provide insight into it.

Was this reply helpful? Yes No
Martin Dráb 237,880 Most Valuable Professional on at

Like (0)

Report

It all depends on what you mean by "the class". If you mean the class that you want to execute, then it's not possible. The batch will create a new instance without any value you set to some other instance.

If you mean a contract class used as a parameter of the operation executed via SysOperation, it's possible to put the file content there. But it's likely not a good idea to use for a large files. In such cases, you can put the file to another storage, such as blob storage (that's where it gets uploaded, so maybe you can simply leave it where it is) and pass just a link to the data contract.

By the way, you're saying that you need a batch job, but the functional requirement you mentioned was different. I suspect you're limiting yourself to batch processing because you didn't accept that there are other solutions.

Was this reply helpful? Yes No