Finance | Project Operations, Human Resources, ...

Unanswered

How to manually create a data package for the Data management package REST API?

(0) Share

Report

Posted on by MathewR

I read the API's documentation in https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/data-management-api , and from my understanding this is the recommended API for importing large amounts of data without making specific configurations in a Finance and Operations instance - compared to the Recurring integrations API where you need to configure 'data projects' (please correct me if I am wrong). I understand the import and export flow as a concept, but I have several questions that I would appreciate an answer. Can you generate a data package manually? Manifest, data files etc? Is there some specific format/template that someone can use to generate data packages?

To give a concrete example, if I am currently creating a 'People' entity by making a POST request to:

POST dynamics.com//data/People
{
"@odata.type": "#Microsoft.Dynamics.DataEntities.Person",
"FirstName": "First",
"LastName": "Last",
"NameSequenceDisplayAs": "FirstMiddleLast",
"MiddleName": "Middle"
}

How can I create a data package with multiple 'People' manually and import it? My problem is creating the data package specifically, I understand how to make the HTTP requests to the API.

All responses (11)

Answers (0)

MathewR 10 on at

Like (0)

Report

RE: How to manually create a data package for the Data management package REST API?

Thank you very much for the explanation.

I did not factor in OData API rate limitations (throttling) when comparing performances, so that would explain my experience with performance.

------------

I wanted to ask some last questions if possible.

In this case where we are importing Employee data, as mentioned previously, the Personnel Number is auto-generated when creating with OData, but with imports we need to specify all key fields (since it will default to empty with Data management API). I have created the data file with employee data, and want to generate the Personnel Numbers manually (through maybe random integer code generation or similar).

1) Is there any simple way to guarantee the uniqueness of the keys compared to existing keys in the database if for example I generate a random Personnel Number through code? From my understanding this entity has a composite primary key, so the composite key needs to be unique, and this seems like a complex issue since I need to check every existing record's composite key.
2) Or do I only need to do this unique key check for the records in the data file itself and not any existing employees in the platform?
3) Is this a typical use-case when importing large amounts of data - meaning do other clients generate all their data including primary keys before importing with this API? Or do they do some comparison by exporting the data firstly? I am not sure if I am doing something wrong here, generating the PersonnelNumbers seems like something clients should not need to do manually.
4) I assume the Recurring integrations API functions similarly with staging tables and would have the same key problem?
Martin Dráb 230,900 Most Valuable Professional on at

Like (0)

Report

RE: How to manually create a data package for the Data management package REST API?

The difference in behavior is that data management uses staging tables, the OData service is not.

Your experience about performance is strange; I don't know what's going on there.
MathewR 10 on at

Like (0)

Report

RE: How to manually create a data package for the Data management package REST API?

No problem, it is thanks to your guidance that I got this far.

I am still confused though, why does the Data management API require more fields than the OData API when creating an entity? Or is it because the Data management API is for "importing" existing data while with OData you are "creating" new data and some of the "new" data is auto-generated?

The execution job for importing 5000 records took about half a hour with the Data management API, while it takes less than a few minutes to create 5000 records with the OData batch endpoint. I was under the assumption that the Data management API was the recommended API for working with large amounts of data, but scheduling, running and monitoring the execution from an outside program for this API's jobs for hours seems pointless when you can send all the data with the OData batch endpoint. Or is there something I am missing about this and the Recurring integration APIs usefulness?
Martin Dráb 230,900 Most Valuable Professional on at

Like (0)

Report

RE: How to manually create a data package for the Data management package REST API?

Ah, sorry, it seems I was looking at a wrong entity (employment instead of employee).