Hello,
We have created approximately 180 custom data entities that we are exporting to a BYOD from a self service Dynamics F&O environment. The data entities are direct copies of underlying dynamics tables such as CustTable, EcoResProduct, etc. We're exporting the data entities as a batch job incrementally every 6 hours in a single export project across 14 companies in the self service environment.
Most of the time, the export executes completely, which takes about 30mins. However, occasionally the export batch job does not finish and seems to get stuck in 'Executing' status and the remaining data entities are in 'Not run' status. This doesn't happen to any specific data entities, but the exact moment when the execution stops is random. We have verified from the BYOD that the data entities that are in 'Not run' do not get updated in the BYOD, so it doesn't seem to be an UI error in the Data management > Job history form. We have increased the sql command timeout to 6000 seconds from the framework parameters settings, but it has had no effect on the export performance. Even though this issue doesn't occur each time, it causes a data sync problem in the follow up process when the data is accessed in the BYOD shortly after the export has finished.
So far, we haven't been able to find the technical reason behind this issue. It doesn't seem to generate any error message in system logs and when looking at the batch job task history, it shows a single batch job task that is stuck in 'Executing' status and none of the data entities after that have completed. We have discussed this issue with Microsoft support, but they haven't been able to find the cause. For some reason, they're not even able to find the data entity export project/Definition group in their logs. Microsoft support has suggested to split the export project that contains 180 data entities to smaller exports that contain 8-10 data entities. However, this has been avoided since it can lengthen the total execution time and the customer process needs to access the BYOD data within a short time after the export has run.
Due to this issue, we haven't yet created a similar setup in a production environment since we haven't been able to export the data entities consistently. Does anyone have an experience of a similar problem?
